论文标题

集装箱剖面:分析资源利用容器化的大数据管道

Container Profiler: Profiling Resource Utilization of Containerized Big Data Pipelines

论文作者

Hoang, Varik, Hung, Ling-Hong, Perez, David, Deng, Huazeng, Schooley, Raymond, Arumilli, Niharika, Yeung, Ka Yee, Lloyd, Wes

论文摘要

本文介绍了容器profiler,这是一种软件工具,可以测量和记录任何容器化任务的资源使用情况。我们的工具介绍了CPU,内存,磁盘和网络利用,这些任务在虚拟机,容器和过程级别上收集了超过五十个Linux操作系统指标的容器化任务。容器profiler支持以可配置的采样间隔进行时间序列分析,以启用容器化任务和管道消耗的资源的连续监视。为了研究容器探测器的效用,我们介绍了多阶段生物信息学分析管道(使用唯一分子标识符的RNA测序)的资源利用率要求。我们检查了分析指标,以评估管道不同阶段的CPU,磁盘和网络资源利用的模式。我们还量化了我们的容器剖面工具的谱图开销,以评估分析运行管道的影响,并具有不同级别的分析粒度验证的粒度验证影响可以忽略不计。容器profiler提供了一个有用的工具,可用于连续监视本地或云上运行的长且复杂的容器化应用程序的资源消耗。这可以帮助识别需要更多资源来提高性能的瓶颈。

This paper presents the Container Profiler, a software tool that measures and records the resource usage of any containerized task. Our tool profiles the CPU, memory, disk, and network utilization of containerized tasks collecting over fifty Linux operating system metrics at the virtual machine, container, and process levels. The Container Profiler supports performing time series profiling at a configurable sampling interval to enable continuous monitoring of the resources consumed by containerized tasks and pipelines. To investigate the utility of the Container Profiler, we profile the resource utilization requirements of a multi-stage bioinformatics analytical pipeline (RNA sequencing using unique molecular identifiers). We examine profiling metrics to assess patterns of CPU, disk, and network resource utilization across the different stages of the pipeline. We also quantify the profiling overhead of our Container Profiler tool to assess the impact of profiling a running pipeline with different levels of profiling granularity verifying that impacts are negligible. The Container Profiler provides a useful tool that can be used to continuously monitor the resource consumption of long and complex containerized applications that run locally or on the cloud. This can help identify bottlenecks where more resources are needed to improve performance.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源