René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Improve performance on V100s","articleBody":"Mostly repeating private email and in-person communication on this topic for reference notes and posterity. \r\n\r\nFRNN performance on V100s on the 2x IBM AC922 systems, OLCF Summit and Princeton's Traverse cluster, is **about 3x slower** than on the P100s on Princeton's TigerGPU cluster. See the below table, which tests the performance for `d3d_0D` training on both machines as a function of batch size (as suggested by @jnkh). I have run these tests with 1, 2, 8 GPUs as well, and several datasets. \r\n\r\n\u003ctable border=\"2\" cellspacing=\"0\" cellpadding=\"6\" rules=\"groups\" frame=\"hsides\"\u003e\r\n\r\n\r\n\u003ccolgroup\u003e\r\n\u003ccol  class=\"org-left\" /\u003e\r\n\r\n\u003ccol  class=\"org-right\" /\u003e\r\n\r\n\u003ccol  class=\"org-right\" /\u003e\r\n\r\n\u003ccol  class=\"org-right\" /\u003e\r\n\r\n\u003ccol  class=\"org-right\" /\u003e\r\n\r\n\u003ccol  class=\"org-right\" /\u003e\r\n\u003c/colgroup\u003e\r\n\u003cthead\u003e\r\n\u003ctr\u003e\r\n\u003cth scope=\"col\" class=\"org-left\"\u003eMachine (GPU Model)\u003c/th\u003e\r\n\u003cth scope=\"col\" class=\"org-right\"\u003eN_node\u003c/th\u003e\r\n\u003cth scope=\"col\" class=\"org-right\"\u003eN_{GPU}\u003c/th\u003e\r\n\u003cth scope=\"col\" class=\"org-right\"\u003eExamples/sec\u003c/th\u003e\r\n\u003cth scope=\"col\" class=\"org-right\"\u003eSec/batch\u003c/th\u003e\r\n\u003cth scope=\"col\" class=\"org-right\"\u003eBatch size\u003c/th\u003e\r\n\u003c/tr\u003e\r\n\u003c/thead\u003e\r\n\r\n\u003ctbody\u003e\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003eTraverse (V100)\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e4\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1.35e3\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.75\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1024\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\r\n\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e2.53e3\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.80\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e2048\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\r\n\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e5.20e3\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.80\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e4096\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003c/tbody\u003e\r\n\r\n\u003ctbody\u003e\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003eTigerGPU (P100)\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e4\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e4.30e3\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.24\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1024\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\r\n\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e7.70e3\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.26\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e2048\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\r\n\r\n\u003ctr\u003e\r\n\u003ctd class=\"org-left\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e\u0026#xa0;\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e1.38e4\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e0.30\u003c/td\u003e\r\n\u003ctd class=\"org-right\"\u003e4096\u003c/td\u003e\r\n\u003c/tr\u003e\r\n\u003c/tbody\u003e\r\n\u003c/table\u003e\r\n\r\n\r\nAt first, I suspected some issue with my Conda / MPI environment on the Power 9 architecture. However, @ge-dong and I compared figures, and we confirmed that we are both independently observing this behavior. In fact, the original modules on Traverse produced about even slower performance (20%). \r\n\r\n@ASvyatkovskiy identified the primary issue being that the TensorFlow backend for`tf.keras` or external Keras does not run the cuDNN autotuner unlike vanilla TensorFlow architecture definitions. See my notes about the autotuner in #51. The default implementations of our layers might be slower on V100 than on P100.\r\n\r\nHe opened issues about this when he first ran on Summit over 1.5 years ago:\r\nhttps://github.com/tensorflow/tensorflow/issues/18913,  https://github.com/keras-team/keras/issues/9825. Related: https://github.com/keras-team/keras/issues/9321\r\n\r\nAnd proposed the following optimizations especially for V100s:\r\n- Use https://github.com/NVIDIA/nccl library to perform all-reduce directly on the GPU\r\n- Use https://github.com/NVIDIA/apex mixed precision optimizers\r\n\r\n\u003e All these things are easier to enable/add in PyTorch, which now also support distributed training natively and through Horovod.\r\n\r\nAlso, I am systematically benchmarking the `LSTM`  Keras layer definition vs. `CuDNNLSTM`, which seems to be at least an order of magnitude faster. \r\n\r\n\r\n**IBM AC922 \"Traverse\" architecture details:**\r\n- Processor is 16-core Power 9 running at 2.7 GHz\r\n- Host memory 256 GB DDR4\r\n- 4 X V100 with 32 GB HBM2\r\n\r\n ","author":{"url":"https://github.com/felker","@type":"Person","name":"felker"},"datePublished":"2019-12-17T23:03:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/52/plasma-python/issues/52"}