René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add platform-dependent Conda YAML and environments/module dependencies","articleBody":"The current `requirements-travis.txt` in the root directory of the repository should be converted to a YAML format that complies with the [Conda environment format](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually). Especially note the version range formatting: https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#package-match-specifications\r\n\r\n`requirements.yaml` should contain the Conda dependencies for a generic (GPU?) platform.\r\n\r\nWe will probably want to have custom Conda environments files for each of the following computers, mostly to handle the non-default Conda channels that may be necessary (e.g. IBM Watson AI for the 2x Power9 architectures with V100s). But we also want to set strict channel priority, be more specific about compatible dependency version ranges, etc. \r\n\r\n- Princeton Research Computing\r\n  - Tiger 2/TigerGPU P100s\r\n  - Traverse V100s\r\n- ALCF\r\n  - Theta KNLs\r\n  - Cooley K80s\r\n- OLCF\r\n   - Summit V100s\r\n\r\nAlso, I am in favor of storing files such as `traverse-env.cmd` containing the following lines, e.g.:\r\n```\r\n#!/usr/bin/env bash\r\n\r\nmodule load anaconda3\r\nconda activate frnn  # must activate conda env before module loads\r\nexport OMPI_MCA_btl=\"tcp,self,vader\"\r\n\r\nmodule load cudatoolkit\r\nmodule load cudnn/cuda-10.1/7.6.1  \r\nmodule load openmpi/gcc/3.1.4/64\r\nmodule load hdf5/gcc/openmpi-3.1.4/1.10.5\r\n```\r\n\r\nThis will make it easier for the user to build FRNN on each platform after creating the Conda environment, e.g. from the new directory named `envs/` or `environments/`:\r\n```\r\nconda env create --file envs/requirements-traverse.yaml\r\n# alternative: \"conda create --name frnn --file traverse.yaml\"\r\nsource traverse-env.cmd\r\npython setup.py install\r\n```\r\nSee [Sample Installation on TigerGPU](https://github.com/PPPLDeepLearning/plasma-python/blob/master/docs/PrincetonUTutorial.md#sample-installation-on-tigergpu), for example.\r\n\r\nAlso, `examples/slurm.cmd` can source the exact same modules in the `.cmd` file for consistency.\r\n\r\nHowever, this will require frequent updates to the `*.cmd` files as system admins upgrade the modules and libraries on the various platforms. \r\n\r\n- [ ] Should the `.yaml` and `.cmd` pairs of files per platform be all stored in a top-level subdirectory, e.g. `environments/`, `env/`, or `envs/`? How do other projects handle this problem and organize such files? Or do they typically only define a single Conda YAML?\r\n- [ ] Have `examples/slurm.cmd` check the hostname in order to automatically `source` the correct `.cmd`?\r\n- [ ] Should the `.cmd` environment files assume that the Conda environment is named `frnn`? Or force the user to define an environment variable `FRNN_CONDA_ENV`?\r\n- [ ] Update [PrincetonUTutorial.md](https://github.com/PPPLDeepLearning/plasma-python/blob/master/docs/PrincetonUTutorial.md) after these changes are made \r\n- [x] Remove all version info from list of dependencies in `setup.py`. \r\n- [ ] Add optional feature dependencies to `setup.py`\r\n\r\n**Current limitations of Conda YAML format**: \r\n\r\n- [ ] Cannot set Conda (strict) channel priority setting within these files. See 336135be2c4d495106726f3f645874cbd186ae24.\r\nFollow conda/conda#8675 for updates regarding this requested feature extension. Until then, it is recommended that each user modifies their `.condarc` local configuration with\r\n`conda config --set channel_priority strict`\r\n\r\n- [ ] Should not install `mpi4py` via pip within the Conda YAML file, since we need to source the modules to get the system MPI libraries before `pip install mpi4py`. There is currently no way to execute arbitrary shell commands, modify environment variables, etc. between the `conda install pkg` and `pip install pkg` stages of the environment creation process from a YAML file, as far as I am aware.\r\nSee similar requests in https://github.com/conda/conda/issues?q=is%3Aopen+is%3Aissue+label%3Atag-environment_spec.\r\nSee https://github.com/PPPLDeepLearning/plasma-python/pull/46/commits/f68f2f0c342656939b170311ca2984c73deceafa.","author":{"url":"https://github.com/felker","@type":"Person","name":"felker"},"datePublished":"2019-12-05T22:45:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/47/plasma-python/issues/47"}