abstract
- Software systems enabling large-scale data analysis workflows (DAWs) are a key technology for modern science as they allow extracting new insights from experimental results. DAWs are pipelines composed of interdependent tasks that are executed in a distributed fashion on large compute clusters. Typically, the individual task implementations are developed by research groups all over the world and usually not tested outside a narrow scope of possible inputs, parameters, and infrastructures. As a result, the operations' correctness depends on many implicit assumptions, such as the completeness and suitability of input data, infrastructure properties such as available cores, etc. This makes quality assurance of DAWs a critical issue. We propose to address this problem by introducing a contract-driven approach to DAW design and implementation. Following the well-known principle of Design by Contract, DAW developers specify contracts in the form of requirements and promises for each task of a DAW. These contracts serve as guards to ensure that tasks run in a proper environment and produce correct results. The detection of contract violations allows to halt the execution of a DAW and to identify the culprit that caused the violation. Thus, the integration of contracts into DAW design provides opportunities for efficiency improvements by reducing computation and debugging time in case of errors.