evaluma.methods.rank_sensitivity

evaluma.methods.rank_sensitivity#

Functions#

_validate_aligned_axes(scores_a, scores_b)

Validate that model and dataset label sets match across conditions.

_ranks_from_scores(→ pandas.Series)

Aggregate per-model scores and convert them to ranks.

_delta_table(→ pandas.DataFrame)

Build the sorted per-model delta_rank table from two rank vectors.

compute_rank_sensitivity_from_ranks(...)

Point-estimate rank sensitivity from two precomputed rank vectors.

compute_rank_sensitivity(...)

Compute ranking sensitivity between two model×dataset score matrices.

Module Contents#

evaluma.methods.rank_sensitivity._validate_aligned_axes(scores_a: pandas.DataFrame, scores_b: pandas.DataFrame)#

Validate that model and dataset label sets match across conditions.

Parameters:

  • scores_a – Condition A normalized scores (model × dataset).

  • scores_b – Condition B normalized scores (model × dataset).

Raises:

  • ValueError – If model label sets differ.

  • ValueError – If dataset label sets differ.

evaluma.methods.rank_sensitivity._ranks_from_scores(scores: pandas.DataFrame, agg='trimmed_mean') → pandas.Series#

Aggregate per-model scores and convert them to ranks.

Parameters:

  • scores – Normalized score matrix (model × dataset).

  • agg – Aggregation mode passed to _aggregate_scores(); defaults to "trimmed_mean" to match aggregate_ranking.

Returns:

Average ranks with rank 1 as best (higher score is better).

Return type:

pd.Series

evaluma.methods.rank_sensitivity._delta_table(rank_a: pandas.Series, rank_b: pandas.Series, cond_a_str: str, cond_b_str: str) → pandas.DataFrame#

Build the sorted per-model delta_rank table from two rank vectors.

Parameters:

  • rank_a – Condition A per-model ranks (rank 1 = best), indexed by model.

  • rank_b – Condition B per-model ranks, aligned to rank_a’s index.

  • cond_a_str – Condition A label (used in the rank_{cond_a} column).

  • cond_b_str – Condition B label.

Returns:

pd.DataFrame with columns model, rank_{cond_a}, rank_{cond_b}, delta_rank, sorted by descending |delta_rank| then model.

evaluma.methods.rank_sensitivity.compute_rank_sensitivity_from_ranks(rank_a: pandas.Series, rank_b: pandas.Series, cond_a, cond_b, agg: str = 'ranker') → evaluma.results.RankSensitivityResult#

Point-estimate rank sensitivity from two precomputed rank vectors.

Decouples the ranking step from the tau computation so any ranking method (average rank, ELO, improvability, or a custom callable) can drive the sensitivity readout — not only the _aggregate_scores family. The inputs are assumed to be literal rank vectors, not arbitrary score-like keys. No bootstrap CI is produced (tau_ci is (nan, nan)).

Parameters:

  • rank_a – Condition A per-model ranks (rank 1 = best), indexed by model.

  • rank_b – Condition B per-model ranks; must cover rank_a’s roster.

  • cond_a – Label for condition A.

  • cond_b – Label for condition B.

  • agg – Label describing the ranking provenance (stored on the result for provenance / plot titles).

Returns:

Point estimates (tau, rho), the delta_rank table, and tau_ci=(nan, nan).

Return type:

RankSensitivityResult

evaluma.methods.rank_sensitivity.compute_rank_sensitivity(scores_a: pandas.DataFrame, scores_b: pandas.DataFrame, cond_a, cond_b, n_bootstrap=1000, random_state=None, agg='trimmed_mean') → evaluma.results.RankSensitivityResult#

Compute ranking sensitivity between two model×dataset score matrices.

Parameters:

  • scores_a – Condition A normalized scores (model × dataset).

  • scores_b – Condition B normalized scores (model × dataset).

  • cond_a – Label for condition A (used in output table/plot labels).

  • cond_b – Label for condition B (used in output table/plot labels).

  • n_bootstrap – Number of dataset-bootstrap samples for the 95% CI.

  • random_state – Seed for numpy.random.default_rng.

  • agg – Per-model aggregation defining the ranking — "trimmed_mean" (default, matching aggregate_ranking), "mean", or "median".

Returns:

Rank sensitivity point estimates, CI, and table.

Return type:

RankSensitivityResult

Raises:

  • ValueError – If n_bootstrap < 0.

  • ValueError – If agg is not a supported mode.

  • ValueError – If model or dataset labels are misaligned.