René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Experiments, Benchmark and Hyperoptimizer","articleBody":"I've come to a point where I'm doing something that I though should be fairly simple, but which is a lot more complicated than expected with our current code. I need your thoughts on the matter.\n\nI have several models I want to benchmark against one another. To do so, I have several benchmarks I want a few models to compete on. To do so, I need to: \n- Iterate through the benchmarks;\n- Iterate through the models;\n- Find the best hyper-parameters;\n- Test from a few seeds and see the results;\n- Keep track of how each model performed; \n- Output appropriate plots.\n\nAs it is, the library is set up to facilitate the training of a single model on a single dataset with a single batch of hyperparameters. I want something higher-level. We could even think about integrating the long awaited spearmint.\n\nSome ideas, all mixed together: \n- Create three more levels of training: Experiment, Benchmark and Hyperoptimizer (I'm certainly open to other names).\n- The `Experiment` is a collection of Benchmark. It runs them, then aggregate data about them.\n- The `Benchmark` is used to pit models against each other and find the best-performing one. To do so, it trains and then collects appropriate data about each of them.\n- The `Hyperoptimizer` is used to find the best hyperparameters for a model to perform a specific task.\n- Some way to add test hyperparameters. You know, just to make sure the code can run through the whole script without crashing.\n- Deal with the issue of the RNG. As of now, nothing is really reproducible. I suggest a hard reset of numpy/theano/blocks RNGs at some level (and it's not clear which one... all of them?)\n\nFor now, a grid/random search of the `Hyperoptimizer` would be great. The `Benchmark` and `Experiment` would need more definite roles and type of data to collect. This brings me to the _how_ part.\n\nWe will need a way to configure an experiment. The more I work on problems, the more I'm tempted to make a more general class to set and access those parameters. However, this would be such a pain to generalize enough.\n\nWe also need a standard way to collect data. I can easily see an `Experiment` telling a `Benchmark` what data to collect, which would then cascade all the way to the `Trainer` and its `Tasks`. I _don't_ want to have to hardcode everything every time, even though for now it's the easiest solution.\n\nI think those classes have the potential, if done right, to make SL a much greater help in scientific reproducibility and it would accelerate a lot the process of getting experiments done faster.\n","author":{"url":"https://github.com/ASalvail","@type":"Person","name":"ASalvail"},"datePublished":"2015-09-30T05:28:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/54/smartlearner/issues/54"}