evaluma.methods.improvability

Functions

_compute_error_matrix(→ pandas.DataFrame)	Reconstruct raw error from a dense score matrix and metric metadata.
compute_improvability(...)	Compute mean improvability per model in raw error space.

Module Contents

evaluma.methods.improvability._compute_error_matrix(raw_matrix: pandas.DataFrame, direction_map: dict, optimum_map: dict) → pandas.DataFrame

Reconstruct raw error from a dense score matrix and metric metadata.

Per column, error is optimum - score for max metrics and score - optimum for min metrics, so higher error is always worse. Assumes a dense (complete) matrix.

Parameters:

• raw_matrix – Model × dataset raw score matrix.

• direction_map – Dataset → "min" / "max" metric direction.

• optimum_map – Dataset → theoretical error optimum (perfect score).

Returns:

Error matrix with the same shape, index, and columns.

Return type:

pd.DataFrame

evaluma.methods.improvability.compute_improvability(raw_matrix: pandas.DataFrame, direction_map: dict, optimum_map: dict) → evaluma.results.ImprovabilityResult

Compute mean improvability per model in raw error space.

Reconstructs raw error, takes the per-dataset best error over models, and computes per-cell (error - best_err) / error * 100 (percent error reduction needed to match the best method). A zero-error cell is already optimal, so its improvability is 0.0 (avoiding division by zero). Mean improvability averages over datasets. Faithful to the TabArena / BeyondArena definition.

Parameters:

• raw_matrix – Model × dataset raw score matrix (dense).

• direction_map – Dataset → "min" / "max" metric direction.

• optimum_map – Dataset → theoretical error optimum.

Returns:

.table (model, improvability; sorted

ascending) and .per_dataset diagnostic long table.

Return type:

ImprovabilityResult