论文标题
在语言和远见中重新审视神经缩放法律
Revisiting Neural Scaling Laws in Language and Vision
论文作者
论文摘要
近年来，深度学习的显着进步主要是由于规模的改进而驱动，在较大的数据集上，较大的模型以较长的时间表进行了培训。为了从经验上预测比例的好处，我们主张基于外推损失的更严格的方法，而不是报告最合适的（插值）参数。然后，我们提出了一种从学习曲线可靠地估算缩放定律参数的配方。我们证明，除了大基础评估基准的任务外，除了大型域中，包括图像分类，神经机器翻译（NMT）和语言建模，包括图像分类，神经机器翻译（NMT）和语言建模，它比以前的方法更准确地推断出更准确的方法。最后，我们发布了一个由90个评估任务组成的基准数据集，以促进该领域的研究。
The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.
