Skip to main content
Ctrl+K
evaluma  documentation - Home evaluma  documentation - Home

Contents

  • Overview
  • Configuration Guide
  • API Reference
    • evaluma
      • evaluma._version
      • evaluma.benchmark
      • evaluma.cli
      • evaluma.methods
        • evaluma.methods.aggregate
        • evaluma.methods.bayesian
        • evaluma.methods.elo
        • evaluma.methods.frequentist
        • evaluma.methods.improvability
        • evaluma.methods.iqm
        • evaluma.methods.profiles
        • evaluma.methods.rank_sensitivity
      • evaluma.metric_registry
      • evaluma.normalize
      • evaluma.plot
      • evaluma.results
  • Tutorials
    • Ranking Models Across a Benchmark: From Mean to IQM
    • Ranking Models by Head-to-Head Dominance: ELO from TabArena
    • Distance from the Best: Improvability Ranking
    • Frequentist Model Comparison: Friedman + Nemenyi / Wilcoxon + Holm
    • Evaluating a Benchmark from a Bayesian Perspective
    • Frequentist vs Bayesian Model Comparison
    • Performance Profiles
    • Rank Sensitivity: Do Model Rankings Hold Across Conditions?
  • References
  • Contributing
  • Repository
  • Suggest edit
  • Open issue
  • .md

Tutorials

Tutorials#

  • Ranking Models Across a Benchmark: From Mean to IQM
  • Ranking Models by Head-to-Head Dominance: ELO from TabArena
  • Distance from the Best: Improvability Ranking
  • Frequentist Model Comparison: Friedman + Nemenyi / Wilcoxon + Holm
  • Evaluating a Benchmark from a Bayesian Perspective
  • Frequentist vs Bayesian Model Comparison
  • Performance Profiles
  • Rank Sensitivity: Do Model Rankings Hold Across Conditions?

previous

evaluma.results

next

Ranking Models Across a Benchmark: From Mean to IQM

By Nils Lehmann

© Copyright 2024, Nils Lehmann.