René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"High-level outline of the first draft","articleBody":"@njsmith @ambv feel free to edit this message.\r\n\r\n----------\r\n\r\n# Introduction\r\n\r\nThis should be a relatively high-level and fun to read section.\r\n\r\n- [ ] Talk about exceptions in an abstract way. I.e. why it is important to have them in Python, why one of the key Python design principle is that errors should never happen silently.\r\n\r\n- [ ] Why isn't one exception enough? Sometimes you have multiple errors that are all equally important:\r\n\r\n  * Main reason: Concurrency (async/await, threads, multiprocessing, ...): If you're doing multiple things at once, then multiple things can fail at once. Only options are to catch them, throw some away, or have some way to propagate multiple exceptions at once. We can't force users to catch them, so right now we have to throw them away, which breaks \"Errors should never pass silently\".\r\n\r\n  * Hypothesis runs a single test lots of times with random input. Sometimes it detects multiple distinct failures, on different inputs. Currently it has no good way to report that.\r\n\r\n  * A unit test: the test failed and also failed its tear down code.\r\n\r\n- [ ] Why aren't `__context__` and `__cause__` good enough? (They give more details about a single exception, but in these cases there are multiple independent errors, that need to be handled or propagated independently. Maybe a concrete example, e.g., you have both a network error + an assertion error, some routine retry code handles the network error, that shouldn't cause the assertion error to be lost?)\r\n\r\n# Motivation\r\n\r\nHere we should talk about exception groups in more detail explaining \"why\" and \"why now\".\r\n\r\n- [ ] Why we are looking at this problem now?  The answer is async/await.  Here we need to talk about `asyncio.gather()` and Trio/nurseries.\r\n\r\n  * `asyncio.gather(t1, t2, t3)` is the primitive one uses in asyncio to wait until multiple concurrent tasks are finished. It is flawed.  If `t1` is failed then we'd propagate its error and try to shutdown both `t2` and `t3`, but shutting them down can also result in errors. Those errors are currently lost.\r\n\r\n  * There's a `return_exceptions=True` flag for `asyncio.gather` that changes its behavior entirely. Exceptions are returned along with the results. This is a slightly better approach, but: it's an opt-in; handling exceptions this way is cumbersome -- you suddenly can't use `try..except` block.\r\n\r\n  * Controlling how things run concurrently is the key thing in asyncio. Current APIs are bad and we need a replacement ASAP.\r\n\r\n  * Trio has a concept called nurseries and it's great. This is what users want in asyncio—literally the most requested feature. Nurseries are context managers that let you structure your async code in a very visual and obvious way.\r\n\r\n  * The problem is that a nursery `with` block needs to propagate more than one exception. Boom, we need exception groups (**EG**) to implement them for asyncio.\r\n\r\n- [ ] To sum up: there were always use cases that required a concept like EGs. It's just that they were not as pronounced as they are now with async/await.\r\n\r\n- [ ] Why does it need to be in the language/stdlib? Can it be a third party library?\r\n\r\n  1. asyncio is in the stdlib, so everything asyncio uses has to be in the stdlib\r\n  2. Also, Trio tried that, and it kind of works but there's a lot of unfixable limitations:\r\n     - lots of basic exception stuff lives in stdlib, e.g. sys.excepthook needs to know about these, the traceback and logging modules need to know about these, etc.\r\n     - exceptions are often raised and caught in different pieces of code, e.g. pytest/ipython/sentry/ubuntu's [apport](https://wiki.ubuntu.com/Apport)/web frameworks/... all want to handle arbitrary exceptions and do something useful with them, so they and the async libraries all need to agree on how to represent these new objects. Ecosystem-wide coordination is what PEPs are for.\r\n     - Currently trio handles this by monkeypatching all the libraries it knows about...\r\n\r\n# What does an EG look like?\r\n\r\n- [ ] Summarize our answer: a concrete (final?) class `ExceptionGroup` that inherits from `BaseException`, and holds a list of exceptions + for each one an string tag giving some human-readable info about where this exception came from to enter this group. Show examples. Show nested examples.\r\n\r\nThen walk through the rationale for these choices:\r\n\r\n- [ ] The EG type must be an exception itself.  It cannot be a tuple or list.  Why: we want `try..finally` to work correctly when an exception group is propagated. Also, making `sys.exc_info()` / `PyThreadState-\u003eexc_info` start holding non-exception objects would be super invasive and probably break lots of things.\r\n\r\n- [ ] The EG type must be a `BaseException` as it can potentially contain multiple `BaseExceptions`.\r\n\r\n- [ ] To give useful tracebacks, we want to preserve the path the exception took, so these need to be nested (show an example of a Trio nested traceback to illustrate)\r\n\r\n- [ ] We want to attach tags / titles to exceptions within EGs. Tasks in asyncio/trio and threads in Python have names -- we want to attach a bit of information to every exception within a EG saying what's its origin.\r\n\r\n- [ ] But semantically, they represent an unstructured set of exceptions; we just use the tree structure to hold traceback info and to get a single object representing the set\r\n\r\n- [ ] Discuss why we allow single-element `ExceptionGroup`s\r\n\r\n  - [ ] Gives opportunity to attach tags to show which tasks an exception passed through as it propagated\r\n\r\n  - [ ] In current prototypes, catching exceptions inside an `ExceptionGroup` requires special ceremony. If this ceremony is needed *sometimes*, then we want to make it needed *always*, so that users don't accidentally use regular `try`/`except` and have it seem to work until they get unlucky and multiple exceptions happen at the same time. Therefore, exceptions that pass through a Trio nursery/asyncio `TaskGroup` should be unconditionally wrapped in an `ExceptionGroup`. But, this rationale may or may not apply to a \"native\" version of `ExceptionGroup`s, depending on what design we end up with for catching them.\r\n\r\n# Working with exception groups\r\n\r\n- [ ] Core primitives: `split` and `leaves`\r\n\r\n- [ ] Semantics\r\n\r\n- [ ] Rationale\r\n\r\n# Catching exceptions in `ExceptionGroups`\r\n\r\n- [ ] Explain basic desired semantics: can have multiple handler \"arms\", multiple arms can match the same group, they're executed in sequence, then any unhandled exceptions + new exceptions are bundled up into a new `ExceptionGroup`\r\n\r\n- [ ] How should this be spelled? We're not sure. Trade-offs are *extremely* messy; we're not even going to try doing a full discussion in this first draft. Some options we see:\r\n\r\n  * Modifying the behavior of `try..except` to have these semantics. (Downside: major change to the language!)\r\n\r\n  * Leave `try`/`except` alone, add new syntax (`grouptry`/`groupexcept` or whatever). (Downside: you always want `grouptry`/`groupexcept`, never `try`/`except`!)\r\n\r\n  * No new syntax, use awkward circumlocutions instead of `try`/`except`. (Downside: they're extremely awkward!)\r\n\r\nSince the design space and trade-offs are *super* complex, we're leaving a full discussion for a later draft / follow-up PEP.","author":{"url":"https://github.com/1st1","@type":"Person","name":"1st1"},"datePublished":"2019-12-16T20:34:54.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/2/exceptiongroups/issues/2"}