René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"What should reload reload?","articleBody":"**Background:**\r\nAmos created the soft reload branch which is now merged to master. It's a large amount of work.\r\n\r\nWe found #1250 as an issue in that code. Now the question is: disable some functionality or fix it?\r\n\r\nThe setting is as follows: I have code in Python. It stores references to objects in C#. That means there are proxies in Python pointing to objects in C#.\r\n\r\nWhen C# unloads the domain, those proxies are now pointing to garbage. In the setting where this happens (Unity domain reload), C# will soon load a new domain. The new domain will be a lot like the old one.\r\n\r\nAmos implemented a mechanism in `runtime_data.cs` to reconnect the proxies. The idea is we know all the proxies that were ever build (and not yet deleted). Serialize that C# data using `System.Runtime.Serialization`. When the new domain arrives, deserialize.\r\n\r\n**Problem:**\r\nDeserialization may fail. See #1250 for one example.\r\n\r\nThis happens in practice when the Unity user renames a field, a class, a namespace, or an assembly -- and on the Python side we had a reference to one of those. E.g. if we handed a C# static function to a python function that expects a callable, we'd have a FieldObject proxy in the code. If the user renames or deletes the C# static function in their code, when Unity reloads the domain, that FieldObject won't be able to reload.\r\n\r\nThis will cause an exception when trying to reinitialize Python for .NET on the C# side, and therefore the system is now in an undefined state.\r\n\r\nThe `System.Runtime.Serialization` is not fault tolerant. It's not possible to provide a callback if deserialization fails for a certain object. If you control a class you can specify custom deserialization (and serialization) for that class, and you can specifically code your stuff to be fault tolerant, but if you recursively serialize any other classes, upon deserialization everything that got serialized gets deserialized as a flat list; any unhandled exceptions there will stop the deserialization.\r\n\r\n**What are the user requirements anyway?**\r\n\r\nFor a Unity user, the requirement is that when I change my C# code and Unity automatically reloads, the Python side doesn't throw a fit no matter how mean I've been to it.\r\n\r\nThis is mainly two things:\r\n1. If in Python I `import MyModule` where MyModule is in C# (this includes `UnityEngine`), I need some way to be able to use that even after a domain reload. Ideally the module reloads itself.\r\n2. If in Python I somehow store a reference to a proxy to C#, I need that the system is able to survive domain reload: it does not crash and it does not leave the Python for .NET data structures in an unstable state.\r\n\r\n**Possible solution 1: fix the bugs**\r\nIn #1250 I prototyped a possible solution. The reason deserialization fails in that case is that FieldObject (in `runtime/fieldobject.cs`) stores a System.Reflection.FieldInfo. Since it can't find that field after reloading, deserialization throws an exception.\r\n\r\nSolution: \r\n1. Add custom serialization to FieldObject.\r\n2. Serialize the FieldInfo into its own serialization context and store the byte stream.\r\n3. When we deserialize the FieldObject, deserialize the FieldInfo from that byte stream.\r\n4. If that fails, catch the exception and mark the FieldObject invalid. When someone wants to access the FieldInfo in the FieldObject, throw an exception.\r\n\r\nEffectively this delays the exception so that we can deserialize everything that works and we delay exceptions until the point of use.\r\n\r\nThere are 13 subclasses of ManagedType that store a pointer to some other object. We can write custom serializations for all of them.\r\n\r\nThere probably are some ManagedType subclasses that can't reasonably be reloaded, e.g. CLRObject. Those need to simply not serialize, and then fail gracefully just like the others.\r\n\r\n**Possible solution 2: disable reconnecting the proxies**\r\nAn alternate plan that Mohammed brought up: what if we just don't reload at all.\r\n\r\nWe'd basically just add a `DisableProxy()` function to ManagedType. Most functions on every ManagedType would need to check if the ManagedType is disabled, and if so raise an exception on use. Probably keep ToString working, definitely keep the finalizer/Dispose working.\r\n\r\n**Possible solution 3: hybrid**\r\nFigure out the most important things to reload, and reload those. Probably this is modules (namespaces) and assemblies.\r\n\r\nFor everything else, disable the proxy.","author":{"url":"https://github.com/benoithudson","@type":"Person","name":"benoithudson"},"datePublished":"2020-10-22T17:17:19.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/1268/pythonnet/issues/1268"}