论文标题
建模机器学习多元宇宙
Modeling the Machine Learning Multiverse
论文作者
论文摘要
在对机器学习研究的可靠性和信誉的越来越关注的越来越多的关注下，我们提出了一个有原则的框架，用于提出可靠和可推广的主张：多元宇宙分析。我们的框架建立在多元宇宙分析（Steegen等，2016）的基础上，该框架是为了应对心理学自身的可重复性危机而引入的。为了有效地探索高维且经常连续的ML搜索空间，我们用高斯工艺代理对多元宇宙进行建模，并采用贝叶斯实验设计。我们的框架旨在促进有关模型性能的强大科学结论，因此我们的方法着重于探索而不是常规优化。在两个案例研究中的第一个中，我们研究了关于自适应优化者相对优点的有争议的主张。其次，我们综合了关于学习率对大批次培训概括差距的影响的矛盾研究。对于机器学习社区而言，多元宇宙分析是一种简单有效的技术，用于识别稳定的主张，提高透明度以及提高可重复性的一步。
Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis (Steegen et al., 2016) introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, the multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.
