论文标题
旨在协作优化分布式数据流作业的群集配置
Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs
论文作者
论文摘要
使用分布式数据流系统分析大型数据集需要使用群集。公共云提供商提供了可用于此类集群的大量资源。但是，在类型和数字中选择适当的资源通常可能具有挑战性，因为所选配置需要匹配分布式数据流的资源需求和访问模式。良好的集群配置避免了硬件瓶颈并最大化资源利用率，避免了昂贵的过度配置。 我们提出了一种协作方法，以根据分布式数据流作业的历史运行时数据共享和学习来查找最佳集群配置。可以使用专门的回归模型来协作共享数据来预测未来作业执行的运行时间。但是，在不同用户和不同环境中生成的历史运行时数据的培训预测模型需要这些模型考虑这些上下文。
Analyzing large datasets with distributed dataflow systems requires the use of clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. However, picking the appropriate resources in both type and number can often be challenging, as the selected configuration needs to match a distributed dataflow job's resource demands and access patterns. A good cluster configuration avoids hardware bottlenecks and maximizes resource utilization, avoiding costly overprovisioning. We propose a collaborative approach for finding optimal cluster configurations based on sharing and learning from historical runtime data of distributed dataflow jobs. Collaboratively shared data can be utilized to predict runtimes of future job executions through the use of specialized regression models. However, training prediction models on historical runtime data that were produced by different users and in diverse contexts requires the models to take these contexts into account.
