René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Request for Feedback on Parallel I/O Performance with PnetCDF-Python on HPC Cluster","articleBody":"\u003e On Apr 8, 2025, at 3:59 AM, Sumanth Gopalagowda wrote:\n\u003e \n\u003e Dear Professor Liao,\n\u003e \n\u003e I hope this message finds you well.\n\u003e \n\u003e My name is Sumanth Gopalagowda, ...\n\u003e As part of my academic project, I am working on computing various deformation measures from snapshots of atomistic simulations.\n\u003e \n\u003e After computing these measures, I am attempting to write the resulting data to disk in parallel  using the PnetCDF-Python interface. However, I am facing performance issues during the I/O  stage on our HPC cluster. For instance, writing approximately 72 GB of data takes around 269 seconds on a single node, but the time increases to about 337 seconds when using two nodes. This I/O bottleneck is negating the computational speed-up achieved through parallelization.\n\u003e \n\u003e I have attached the relevant portion of my code that performs the file writing. I would be extremely grateful if you could take a moment to review it and provide any suggestions or insights on how I might improve its scalability and performance.\n\u003e \n\u003e Thank you very much for your time and consideration.\n\u003e \n\u003e Best regards,\n\u003e Sumanth Gopalagowda\n\n[write_out.py.txt](https://github.com/user-attachments/files/19653837/write_out.py.txt)\n\nPlease let us know what file system you are using (is it a parallel file system?).\nIf you can edit your short program to add the main function so we can reproduce\nthe performance number, that will be helpful.\n","author":{"url":"https://github.com/wkliao","@type":"Person","name":"wkliao"},"datePublished":"2025-04-08T17:20:24.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":13},"url":"https://github.com/63/PnetCDF-Python/issues/63"}