abstract
- Scientific workflows are increasingly adopted for large-scale data analysis because of their reproducibility, scalability, and trust-enhancing documentation of analysis processes. These workflows are typically I/O-heavy, processing large volumes of data. This is especially the case in bioinformatics where the amount of data to store is very large. In genomics, a lot of biotools directly integrate gzip decompression of the input data and compression of the output data, making it the default choice in this field. However, gzip is a general-purpose tool, and many domain-specific compression methods outperform it both on compression ratio and speed. Implementing these domain-specific compression methods within existing workflows requires manual code adaptation for every task, which is time-consuming and error-prone. To relieve developers, we propose our COmpressionMEThod for workFLOW engines (COMET-FLOW). The idea behind this method is to shift the compression responsibility from individual tools to the workflow execution engine, allowing uniform handling of all tasks. This approach requires only minimal adjustments to existing workflows and facilitates future support for other data formats and beyond Bioinformatics. We implemented COMET-FLOW in the popular workflow system Nextflow, enabling the workflow engine to automatically manage compression and decompression of files at runtime. Experiments with three nf-core workflows and several datasets showed that our approach can reduce storage size by up to 31% and runtime up to 41% compared to standard implementations.