René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Speed up multiple inference steps at the end of each epoch","articleBody":"Presently, at the end of every epoch, the trained weights are reloaded via a call to `Keras.Models.load_weights()` 3x separate times in order to evaluate the accuracy on the shots in the training, validation, and testing sets:\r\n\r\nhttps://github.com/PPPLDeepLearning/plasma-python/blob/c82ba61e339882a5af10b1052edc0348e16119f4/plasma/models/mpi_runner.py#L932-L965\r\n\r\nDepending on the size of the datasets (number of shots, pulse length, number of signals per shot), network architecture, and hardware, this process might take a significant amount of time. This is especially noticeable if the epoch walltimes are relatively short due to small batch sizes, etc. \r\n\r\nFor example, for a recent test with `d3d_0D` on Traverse 4x V100s:\r\n```\r\nFinished training epoch 3.01 during this session (1.00 epochs passed) in 87.65 seconds\r\nFinished training of epoch 6.01/1000\r\nBegin evaluation of epoch 6.01/1000\r\n[2] loading from epoch 6\r\n[1] loading from epoch 6\r\n[0] loading from epoch 6\r\n[3] loading from epoch 6\r\n\r\n128/894 [===\u003e..........................] - ETA: 1:53\r\n640/894 [====================\u003e.........] - ETA: 13s\r\n896/894 [==============================] - 35s 39ms/step\r\n[0] loading from epoch 6\r\n[3] loading from epoch 6\r\n[1] loading from epoch 6\r\n[2] loading from epoch 6\r\n\r\n128/894 [===\u003e..........................] - ETA: 1:53\r\n640/894 [====================\u003e.........] - ETA: 13s\r\n896/894 [==============================] - 35s 39ms/step\r\nepoch 6, val_roc_30 = 0.85346611872694 val_roc_70 = 0.8345022047574768 val_roc_200 = 0.7913309535951044 val_roc_500 = 0.6638869724330323 va\\l_roc_1000 = 0.5480697123316435\r\n[3] loading from epoch 6                                                                                                                    [2] loading from epoch 6\r\n[0] loading from epoch 6                                                                                                                    [1] loading from epoch 6\r\n                                                                                                                                            128/894 [===\u003e..........................] - ETA: 1:53\r\n640/894 [====================\u003e.........] - ETA: 12s                                                                                         896/894 [==============================] - 35s 39ms/step\r\nepoch 6, test_roc_30 = 0.8400389140546622 test_roc_70 = 0.8236098866020126 test_roc_200 = 0.7792357036451524 test_roc_500 = 0.6798285349466\\453 test_roc_1000 = 0.5699692943787431\r\n```\r\n\r\nIt seems straightforward to deduplicate the 3x 1:53 load times via a new combined function instead of 2x calls to `mpi_make_predictions_and_evaluate_multiple_times()` + 1x call to `mpi_make_predictions_and_evaluate()`. \r\n","author":{"url":"https://github.com/felker","@type":"Person","name":"felker"},"datePublished":"2020-01-07T21:00:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/61/plasma-python/issues/61"}