René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Saving trained models and their metadata for inference and reproducibility","articleBody":"Following discussion on Wednesday 2019-12-04 in FRNN group meeting in San Diego, we need to start systematically saving the best trained models for:\r\n1. Collaboration (no need for multiple users to waste GPU hours retraining the same models)\r\n2. Practical inference (@mdboyer wants a Python interface derived from `performance_analysis.py` that would allow a user to load a trained model and easily feed a set of shot(s) for inference, without using the bloated shot list and preprocessing pipeline that has been oriented towards training for the first phase of the project. Would enable exploratory studies about proximity to disruption, UQ, clustering, etc. This is an important intermediate step to setting up the C-based real-time inference tool in the PCS. ) \r\n3. Reproducibility\r\n\r\nAs a part of a broader effort towards improving reproducibility of our workflow, these models should be stored with:\r\n- `.h5` file containing the tunable parameters (can be directly loaded by Keras or C-translated inference software)\r\n- Input configuration `conf.yaml` and/or dumped final configuration used in specifying and training the model\r\n- Output performance metrics of the trained model (train/validate/test ROC)\r\n- Normalization `.npz` pickled class. For `VarNormalizer`, this would only consist of the standard deviations of each channel of each signal from the set of shots used to train the normalizer. However, it is serialized and saved as a \"fat\" class object that requires the entire `plasma` module to load. Might want to dump a simple non-pickled array, or even `.txt`, alongside the pickle, so that we have a simple file to load with the Keras-C wrapper.\r\n- Some metadata about the layout of a preprocessed shot in `processed_shots/signal_group_*/*.npz` (order of channels and signals, sampling rates, thresholding? etc.), so that any real-time inference wrapper could apply a similar preprocessing to the incoming data.\r\n- **Exact** individual shot numbers used in the training, validation, and testing sets, so that anyone using the model for inference will know if the shot being supplied to the model has already been used to train the model. \r\n- SHA1 of Git commit \r\n- Conda environment; versions of dependencies such as TensorFlow, Keras, PyTorch, scikit-learn\r\n- Computer used for training, MPI library, CuDNN library, etc.\r\n-  Number of devices and MPI ranks used in training (least important)\r\n\r\nGiven the binary `.h5` and `.npz` files, we probably don't want to use VCS to store everything. But we might want to version control the plain-text metadata about the trained models. Store in this repository alongside the code? Or a new repository under our GitHub organization?\r\n\r\nAlso, should we consider ONNX?\r\n\r\n\r\n","author":{"url":"https://github.com/felker","@type":"Person","name":"felker"},"datePublished":"2019-12-05T19:00:54.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/41/plasma-python/issues/41"}