René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"ETA calculation is inaccurate","articleBody":"Example of the current per-step (iteration) diagnostic output provided by FRNN around epoch 22 of the D3D 0D model (run on 4 V100 GPUs of Traverse):\r\n```\r\n[0] step: 0 [ETA: 468568011.02s] [0.00/1789], loss: 1.05701 [1.05701] | walltime: 5.7374 | 8.47E+02 Examples/sec | 6.04E-01 sec/batch [92.3% calc., 7.7% sync.][batch = 512 = 128*4] [lr = 7.30E-05 = 1.83E-05*4]\r\n```\r\nThe ETA provided in this example is clearly inaccurate (each epoch takes around 60s). Specifically, there are two types of issues:\r\n1. The ETA computed in the first step of any epoch is always inaccurate. \r\n2. For later epochs within a session, the ETA increases nearly monotonically for many steps before starting to decrease nearly monotonically. \r\n\r\n### First step\r\n\r\nFor the first epoch in a given session, it gives a huge ETA since `MPI_Model.num_so_far` is zero, resulting in `work_so_far` of 0 being passed to:\r\nhttps://github.com/PPPLDeepLearning/plasma-python/blob/c82ba61e339882a5af10b1052edc0348e16119f4/plasma/models/mpi_runner.py#L613-L616\r\ncausing `total_time` to explode. \r\n- [ ] Probably should just refuse to give an ETA for the first step (or steps) of the first epoch\r\n\r\nFor later epochs within a session, it gives a minuscule ETA:\r\n```\r\nstep: 0 [ETA: 0.55s] [1819.00/1789], loss: 0.98688 [0.98688] | walltime: 174.4240 | 8.93E+02 Examples/sec | 5.73E-01 sec/batch [96.1% calc., 3.9% sync.][batch = 512 = 128*4] [lr = 7.08E-05 = 1.77E-05*4]\r\n``` \r\n- [ ] I think an error was introduced when I changed the 0-based indexing of the epochs 1-2 months ago.\r\n\r\n### Later steps in later epochs\r\n\r\nE.g. here are the ETAs for some later epoch:\r\n```\r\n\r\nETA: 0.55s\r\nETA: 22.14\r\nETA: 27.98\r\nETA: 31.63\r\nETA: 35.88\r\nETA: 38.45\r\nETA: 34.89\r\nETA: 36.21\r\nETA: 35.35\r\nETA: 35.56\r\nETA: 36.04\r\nETA: 35.88\r\nETA: 35.33\r\nETA: 34.49\r\nETA: 34.73\r\nETA: 34.29\r\nETA: 34.13\r\nETA: 33.51\r\nETA: 33.16\r\n…\r\nETA: 1.35s\r\nETA: 1.06s\r\nETA: 0.67s\r\nETA: 0.11s\r\nETA: -0.45\r\n```\r\n- [ ] Consider using the measured runtimes of the previous epochs within this session to inform the ETA in later epochs. ","author":{"url":"https://github.com/felker","@type":"Person","name":"felker"},"datePublished":"2020-01-07T02:15:58.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/55/plasma-python/issues/55"}