René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Assignment of Expression to Tensor of Incorrect Dimensions - Strange Error - Numpy Unable to Allocate very Large Array","articleBody":"Version:\r\n```\r\nname: SomeEnvironment\r\nchannels:\r\n  - https://conda.anaconda.org/conda-forge\r\n\r\n# Originally created on Ubuntu Jammy:\r\ndependencies:\r\n  - numpy=1.24\r\n  - python=3.9\r\n  - xtensor-python=0.26.1\r\n  - xtensor-blas: Need to check, the machine in question is down.\r\n```\r\n\r\n\r\n\r\nI spent a good few hours lost trying to dig out a bug that was leading to a segfault, and then an obscure message coming back from numpy about not being able to allocate enough space for an enormous array.\r\n\r\nIt turns out that I was simply assigning an expression that should have been a (1D) vector to a 2-dimensional pytensor. This one verifies my constant love of constraining dimension at compile-time! \"Let's just keep things simple\" they say, and then I waste time trying to debug code that never could have run correctly and without safety measures it's not so simple :sweat_smile: \r\n\r\nIn any case, we are at runtime and here is a MRE:\r\n```C++\r\nvoid debugEntryPoint(){\r\n    \r\n    \r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 2\u003e matrix{\r\n            {3, 0, 0, 0},\r\n            {0, 4, 0, 0},\r\n            {0, 0, 5, 0},\r\n            {0, 0, 0, 6}\r\n    };\r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 1\u003e vector{{0, 0, 0, 1}};\r\n\r\n\r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 1\u003e   correctResult = xt::linalg::dot(matrix, vector);\r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 2\u003e incorrectResult = xt::linalg::dot(matrix, vector);\r\n    // We never get here\r\n\r\n    cout \u003c\u003c correctResult \u003c\u003c endl;\r\n\r\n\r\n    // Or equally:\r\n    const auto \u0026view = xt::linalg::dot(matrix, vector);\r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 1\u003e   correctResult = view;\r\n    xt::pytensor\u003cstd::complex\u003cfloat\u003e, 2\u003e incorrectResult = view;\r\n    \r\n}\r\n```\r\n\r\nI would expect a message explaining that the size of the view as the result of the expression created cannot be assigned to a tensor of this shape. I.e. note that the error comes at the moment we assign, not when we try to compute the result.\r\nAnd the problem is that this is the error that results:\r\n\r\n```Python\r\n\u003e\u003e\u003e debugEntryPoint()\r\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.98 PiB for an array with shape (4, 140095169944816) and data type complex64\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\r\nRuntimeError: NumPy: unable to create ndarray\r\n\r\n```\r\n\r\nNot sure if this is down to the BLAS or Python package, but opening here as it specifically relates to the message from Numpy and at the point we perform the copy.\r\n\r\nCheers","author":{"url":"https://github.com/stellarpower","@type":"Person","name":"stellarpower"},"datePublished":"2023-03-02T01:07:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/287/xtensor-python/issues/287"}