René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Function outlining errors in mirgecom","articleBody":"Trying out the outlining capability in #221 with mirgecom:\r\n- with pytato from (inducer/pytato#431)\r\n- with mirgecom from (illinois-ceesd/mirgecom#880)\r\n\r\nTo reproduce this, install mirgecom:\r\n`emirge/install.sh --branch=production-outlining`\r\n\r\nThen run any example with CNS operator (cd examples):\r\n`python -m mpi4py  combozzle-mpi.py`\r\n\r\nThe error is as follows:\r\n```\r\nfrozen_inv_metric_deriv_vol: check array access within bounds: started 3s ago\r\nfrozen_inv_metric_deriv_vol: check array access within bounds: completed (3.94s wall 1.00x CPU)\r\nfrozen_inv_metric_deriv_vol: generate code: completed (2.18s wall 1.00x CPU)\r\nbuild program: kernel 'frozen_inv_metric_deriv_vol' was part of a lengthy source build resulting from a binary cache miss (2.60 s)\r\n/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/miniforge3/envs/prediction.env/lib/python3.11/site-packages/pyopencl/invoker.py:366: UserWarning: Kernel 'frozen_inv_metric_deriv_vol_0' has 468 arguments with a total size of 3744 bytes, which approaches the limit of 4352 bytes on \u003cpyopencl.Device 'Tesla V100-SXM2-16GB' on 'Portable Computing Language' at 0x101f47d38\u003e. This might lead to compilation errors, especially on GPU devices.\r\n  warn(f\"Kernel '{function_name}' has {num_cl_args} arguments with \"\r\nTraceback (most recent call last):\r\n  File \"\u003cfrozen runpy\u003e\", line 198, in _run_module_as_main\r\n  File \"\u003cfrozen runpy\u003e\", line 88, in _run_code\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/miniforge3/envs/prediction.env/lib/python3.11/site-packages/mpi4py/__main__.py\", line 7, in \u003cmodule\u003e\r\n    main()\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/miniforge3/envs/prediction.env/lib/python3.11/site-packages/mpi4py/run.py\", line 198, in main\r\n    run_command_line(args)\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/miniforge3/envs/prediction.env/lib/python3.11/site-packages/mpi4py/run.py\", line 47, in run_command_line\r\n    run_path(sys.argv[0], run_name='__main__')\r\n  File \"\u003cfrozen runpy\u003e\", line 291, in run_path\r\n  File \"\u003cfrozen runpy\u003e\", line 98, in _run_module_code\r\n  File \"\u003cfrozen runpy\u003e\", line 88, in _run_code\r\n  File \"combozzle-mpi.py\", line 1309, in \u003cmodule\u003e\r\n    main(use_logmgr=args.log, use_leap=args.leap, input_file=input_file,\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/mpi.py\", line 200, in wrapped_func\r\n    func(*args, **kwargs)\r\n  File \"combozzle-mpi.py\", line 1211, in main\r\n    advance_state(rhs=my_rhs, timestepper=timestepper,\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/steppers.py\", line 439, in advance_state\r\n    _advance_state_stepper_func(\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/steppers.py\", line 164, in _advance_state_stepper_func\r\n    state = timestepper(state=state, t=t, dt=dt, rhs=maybe_compiled_rhs)\r\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/integrators/lsrk.py\", line 66, in euler_step\r\n    return lsrk_step(EulerCoefs, state, t, dt, rhs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/integrators/lsrk.py\", line 53, in lsrk_step\r\n    k = coefs.A[i]*k + dt*rhs(t + coefs.C[i]*dt, state)\r\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/arraycontext/arraycontext/impl/pytato/compile.py\", line 316, in __call__\r\n    output_template = self.f(\r\n                      ^^^^^^^\r\n  File \"combozzle-mpi.py\", line 1154, in cfd_rhs\r\n    ns_operator(\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/mirgecom/mirgecom/navierstokes.py\", line 462, in ns_operator\r\n    grad_cv = get_grad_cv(state)\r\n              ^^^^^^^^^^^^^^^^^^\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/arraycontext/arraycontext/impl/pytato/outline.py\", line 159, in __call__\r\n    call_site_output = func_def(**call_parameters)\r\n                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/p/gpfs1/mtcampbe/CEESD/AutomatedTesting/MIRGE-Timing/timing/y3-prediction-testing/emirge/pytato/pytato/function.py\", line 172, in __call__\r\n    if expected_arg.dtype != kwargs[argname].dtype:\r\n                             ~~~~~~^^^^^^^^^\r\nKeyError: '_actx_in_1_0_mass_0'\r\n--------------------------------------------------------------------------\r\nMPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD\r\nwith errorcode 1.\r\n\r\nNOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.\r\nYou may or may not see output from other processes, depending on\r\nexactly when Open MPI kills them.\r\n--------------------------------------------------------------------------\r\n```\r\n\r\nAs an initial guess, it looks like maybe the outlining infrastructure picks apart the array containers down to DOFArrays, but then looks for the function arguments to be the DOFArrays instead of the containers?","author":{"url":"https://github.com/MTCam","@type":"Person","name":"MTCam"},"datePublished":"2023-04-20T13:47:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/224/arraycontext/issues/224"}