evaluma.methods.profiles

evaluma.methods.profiles#

Functions#

compute_profiles(→ evaluma.results.ProfileResult)

Compute Dolan-Moré performance profiles.

Module Contents#

evaluma.methods.profiles.compute_profiles(scores_matrix: pandas.DataFrame, metric_direction: dict | None = None) → evaluma.results.ProfileResult#

Compute Dolan-Moré performance profiles.

Ratios are computed directly from raw scores without normalization:

  • max metrics (higher is better): r_ij = best_j / score_ij

  • min metrics (lower is better): r_ij = score_ij / best_j

where best_j is the best score across all models on dataset j. Both definitions yield r_ij ≥ 1, with r_ij = 1 when model i is the best on dataset j.

The tau grid uses the exact observed ratio values as breakpoints, giving the exact Dolan-Moré step function. The plot renders on a log₁₀(τ) axis, following ML-GYM (Batra et al., 2025) and the AutoML Decathlon (Roberts et al., 2022), which extended Dolan & Moré (2002). Use ProfileResult.aup for the scalar Area Under the Profile summary.

Parameters:

  • scores_matrix – Raw model × dataset score matrix. All values must be strictly positive.

  • metric_direction – Dict mapping dataset (column) names to "min" or "max". Datasets absent from the dict default to "max".

Returns:

tau, model, fraction_within_tau).

Return type:

ProfileResult with .table (long-format

Raises:

ValueError – If any value in scores_matrix is zero or negative.