abstract
- The paradigm shift towards data-driven science is massively transforming the scientific process. Scientists use exploratory data analysis to arrive at new insights. This requires them to specify complex data analysis workflows, which consist of compositions of data analysis functions. Said functions encapsulate information extraction, integration, and model building through operations specified in linear algebra, relational algebra, and iterative control flow among these. A key challenge in these complex workflows is to understand and act upon irregularities in these workflows, such as outliers in aggregations. Regardless whether irregularities stem from errors or point to new insights, they must be localized and rationalized, in order to ensure the correctness and overall trustworthiness of the workflow. We propose to automatically reduce a workflow’s input data while still observing some outcome of interest, thereby computing a minimal reproducible example to support workflow debugging. In essence, we reduce the problem to the determination of the input relevant to reproducing the irregularity. To that end, we present a portfolio of different strategies being tailored to data analysis workflows that operate on tabular data. We investigate their feasibility in terms of input reduction, and compare their effectiveness and efficiency within three characteristic cases. Access provided by Humboldt Universität zu Berlin. Download conference paper PDF Similar content being viewed by others Data Moves: Libraries and Data Science Workflows Chapter © 2020 A Dockerized String Analysis Workflow for Big Data Chapter © 2019 Conclusion Chapter © 2019 Explore related subjects Discover the latest articles, books and news in related subjects, suggested using machine learning. Automation Big Data Data Analytics Data Analysis and Big Data Data processing Information Processing Statistical Methods in Psychological Science