Benchopt: Reproducible, efficient and collaborative optimization benchmarks

Thomas Moreau, Mathurin Massias, Alexandre Gramfort, Pierre Ablin, Pierre Antoine Bannier, Benjamin Charlier, Mathieu Dagréou, Tom Dupré la Tour, Ghislain Durif, Cassio F. Dantas, Quentin Klopfenstein, Johan Larsson, En Lai, Tanguy Lefort, Benoit Malézieux, Badr Moufad, Binh T. Nguyen, Alain Rakotomamonjy, Zaccharie Ramzi, Joseph SalmonSamuel Vaiter

Forskningsoutput: Kapitel i bok/rapport/Conference proceedingKonferenspaper i proceedingPeer review

Sammanfattning

Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: ℓ2-regularized logistic regression, Lasso, and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.

Originalspråkengelska
Titel på värdpublikationAdvances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
RedaktörerS. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
FörlagCurran Associates, Inc
Sidor25404-25421
ISBN (elektroniskt)9781713871088
StatusPublished - 2022 dec. 6
Evenemang36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, USA
Varaktighet: 2022 nov. 282022 dec. 9

Publikationsserier

NamnAdvances in Neural Information Processing Systems
Volym35
ISSN (tryckt)1049-5258

Konferens

Konferens36th Conference on Neural Information Processing Systems, NeurIPS 2022
Land/TerritoriumUSA
OrtNew Orleans
Period2022/11/282022/12/09

Bibliografisk information

Publisher Copyright:
© 2022 Neural information processing systems foundation. All rights reserved.

Ämnesklassifikation (UKÄ)

  • Sannolikhetsteori och statistik

Fingeravtryck

Utforska forskningsämnen för ”Benchopt: Reproducible, efficient and collaborative optimization benchmarks”. Tillsammans bildar de ett unikt fingeravtryck.

Citera det här