René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`pacemaker` crashes when early stopping and maximum iterations flags are raised at the same time","articleBody":"I have been using the `min_relative_train_loss_per_iter` and `min_relative_test_loss_per_iter` in my fittings but during the ladder scheme fitting when the early stopping is triggered at the same time the maximum number of iterations have been reached (I set this to 1000), it crashes the run.\n\nThis may happen when `min_relative_test_loss_per_iter` is set to 5e-5 or 1e-4\nThose are the other parameters I used in my fittings,\n\n```\nkappa = 0.08\nnrad_max = [15, 7, 3, 2, 1, 1], l_max = [0, 4, 3, 2, 1, 1] =\u003e 374 functions\nladder_type = 'power_order'\nladder_step = [20, 0.1]\nbatch_size = 1000\nearly_stopping_patience = 200\nmax_iter = 1000\n```\n\nThis is the error message I get from the log file after the 1000th iteration,\n\n```\n --------------------------------------------TEST STATS--------------------------------------------\n Iteration:  #1000Loss:    Total:  2.6584e-05 (100%) \n                          Energy:  1.2040e-05 ( 45%) \n                           Force:  1.3858e-05 ( 52%) \n                              L1:  4.0020e-07 (  2%) \n                              L2:  2.8561e-07 (  1%) \n Number of params./funcs:    585/100                                  Avg. time:       0.00 mcs/at\n -------------------------------------------------------------------------------------------------\n            Energy/at, meV/at   Energy_low/at, meV/at      Force, meV/A        Force_low, meV/A   \n    RMSE:            7.70                 3.36                31.63                   13.22\n     MAE:            3.90                 2.27                10.31                    6.08\n  MAX_AE:          154.11                28.21              1023.20                  208.15\n -------------------------------------------------------------------------------------------------\n 2025/02/11 13:15:36 I - Last relative TEST loss change -4.72e-05/iter (averaged over last 50 step(s))\n /cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/scipy/optimize/_minimize.py:726: OptimizeWarning: Maximum number of iterations has been exceeded.\n   res = _minimize_bfgs(fun, x0, args, jac, callback, **options)\n 2025/02/11 13:15:40 I - EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05  / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping\n          Current function value: 0.000229\n          Iterations: 1000\n          Function evaluations: 1028\n          Gradient evaluations: 1028\n Fitting took 1767.17 seconds\n          Current function value: 0.000065\n          Iterations: 1000\n          Function evaluations: 1023\n          Gradient evaluations: 1023\n Fitting took 1998.61 seconds\n          Current function value: 0.000044\n          Iterations: 1000\n          Function evaluations: 1024\n          Gradient evaluations: 1024\n Fitting took 3360.17 seconds\n          Current function value: 0.000031\n          Iterations: 1000\n          Function evaluations: 1010\n          Gradient evaluations: 1010\n Fitting took 2765.05 seconds\n          Current function value: 0.000024\n          Iterations: 1000\n          Function evaluations: 1013\n          Gradient evaluations: 1013\n Traceback (most recent call last):\n   File \"/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker\", line 401, in \u003cmodule\u003e\n     main(sys.argv[1:])\n   File \"/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker\", line 248, in main\n     general_fit.fit()\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py\", line 481, in fit\n     self.target_bbasisconfig = self.ladder_fitting(self.initial_bbasisconfig, self.target_bbasisconfig)\n                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py\", line 509, in ladder_fitting\n     current_bbasisconfig = self.cycle_fitting(current_bbasisconfig)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py\", line 564, in cycle_fitting\n     current_bbasisconfig = self.fit_backend.fit(\n                            ^^^^^^^^^^^^^^^^^^^^^\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py\", line 129, in fit\n     raise e\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py\", line 96, in fit\n     fit_res = self.run_tensorpot_fit(bbasisconfig, dataframe, loss_spec, fit_config,\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py\", line 236, in run_tensorpot_fit\n     self.fitter.fit(dataframe, test_df=test_dataframe, niter=fit_config[FIT_NITER_KW],\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py\", line 125, in fit\n     self.process_test_metric()\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py\", line 318, in process_test_metric\n     self.test_metric_callback(curr_test_metrics_data)\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py\", line 410, in test_metric_callback\n     self.detect_early_stopping(mode='test')\n   File \"/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py\", line 465, in detect_early_stopping\n     raise TestLossChangeTooSmallException(msg)\n pyace.generalfit.TestLossChangeTooSmallException: EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05  / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping\n Exception: Potential file output_potential.yaml doesn't existsLoading B-basis from 'output_potential.yaml'\n Traceback (most recent call last):\n   File \"/cmmc/ptmp/hgaafer/mambaforge/bin/pace_yaml2yace\", line 28, in \u003cmodule\u003e\n     bbasis = ACEBBasisSet(input_yaml_filename)\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ValueError: Potential file output_potential.yaml doesn't exists\n```\n","author":{"url":"https://github.com/HaithamGaafer","@type":"Person","name":"HaithamGaafer"},"datePublished":"2025-02-13T16:42:54.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/86/python-ace/issues/86"}