论文标题
对比度图像学习的可再现缩放定律
Reproducible scaling laws for contrastive language-image learning
论文作者
论文摘要
扩大神经网络已导致各种任务的出色表现。此外，性能通常遵循可靠的缩放定律，这是训练集大小，型号大小和计算的函数，随着大规模实验变得越来越昂贵，它提供了宝贵的指导。但是，先前的缩放法律工作主要使用了私人数据\＆模型，或者专注于单态语言或视觉学习。为了解决这些局限性，我们研究了与公共Laion数据集和开源OpencCencellip repository的对比性语言图像预训练（剪辑）的比例定律。我们的大规模实验涉及对多达20亿图像文本对训练的模型，并为多个下游任务（包括零拍，检索，线性探测和端到端微调）确定功率定律缩放。我们发现，尽管模型体系结构和相似的培训配方，但训练分布在缩放定律中起着关键作用，因为OpenAI和OpenCLIP模型表现出不同的缩放行为。我们开放评估工作流程和所有模型，包括最大的公共剪辑模型，以确保可重复性并使缩放定律研究更容易访问。复制本研究的源代码和说明将在https://github.com/laion-ai/scalinglaws-openc-plip上获得。
Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip
