abstract
- Deep learning models are deployed in an increasing number of industrial domains, such as retail and automotive applications. An instance of a model typically performs one specific task, which is why larger software systems use multiple models in parallel. Given that all models in production software have to be managed, this leads to the problem of managing sets of related models, i.e., multi-model management. Existing approaches perform poorly on this task because they are optimized for saving single large models but not for simultaneously saving a set of related models. In this paper, we explore the space of multi-model management by presenting three optimized approaches: (1) A baseline approach that saves full model representations and minimizes the amount of saved metadata. (2) An update approach that reduces the storage consumption compared to the baseline by saving parameter updates instead of full models. (3) A provenance approach that saves model provenance data instead of model parameters. We evaluate the approaches for the multi-model management use cases of managing car battery cell models and image classification models. Our results show that the baseline outperforms existing approaches for save and recover times by more than an order of magnitude and that more sophisticated approaches reduce the storage consumption by up to 99%. 1 Figure 1: Multi-Model Management One DL model performs one well-defined task, for example, to predict the behavior of a specific real-world entity. In the case of multiple equivalent entities in one device or location, multiple models with the same architecture run on the same device. One practical example is the modeling of electric car battery systems. Electric car batteries can consist of thousands of individual cells, each possibly being associated with its own DL model that has thousands of parameters and predicts the cell’s behavior. DL models are used because they can be evaluated and continuously updated with a comparably low computational cost [4, 14]. It is beneficial to use one model per battery cell instead of one single model for the entire battery because individual models provide a spatial resolution regarding, for instance, temperature evolution, cell aging, or current distribution. Regardless of the phase and use case, all DL models need to be managed. We distinguish between single-model management, where we save single models at a time, and multi-model management, where we save sets of models. In the model development phase, we run individual experiments and manage the associated models, which requires single-model management. In the model deployment phase, in many real-life use cases, we manage sets of different and frequently updated models sharing the same architecture. Here we apply multi-model management. Multi-model management is a common and very relevant model management scenario because (1) multi-model management is critical during the deployment phase, which is the phase a model is primarily used in, and (2) multi-model management is resource intense because of the large number of models and the high update frequency. In this paper, we focus on multi-model management in the deployment phase, as presented in Figure 1. We have 𝑛 (𝑛 >> 1000) individual models sharing the same architecture. Over time the model performance decreases, and the models are partially or fully updated on locally collected data. We save every model ever generated for analytical and archival purposes but only recover a selected number of models, for example, after an accident. Existing model management approaches [6, 11, 13, 15] do not cover multi-model management or perform poorly in this