Plugin discovery: Store imported modules to sys.modules

[iLLiCiTiT]

Changelog Description

Overal improvement of the dynamic import logic. It is possible to use dataclasses in case annotations are imported and it is possible to use relative imports within the plugin directory (if a directory is registered).

Additional info

Both modules_from_path and import_filepath can expect Path. The modules_from_path supports to pass in path to a python file.

I might overdue this changes so maybe I'll make it smaller.

1. First change is that combination of dataclass and from __future__ import annotations does not cause crashes. That was fixed by always storing the module into sys modules. Default implementation is using parent directory as hash of parent module.
2. Directory discover does support relative imports. That requires more maintanance of the modules but it works if there is an __init__.py file in the directory (the file itself is probably imported or used).

NOTE: It does not mean we can use dataclasses in publish plugins, there are still places using pyblish default discover logic.

Testing notes:

1. All hosts should be able to discovery any kind of plugins.

Dataclasses fix

1. Use dataclasses and __future__.annoations in a plugin file.
2. The plugin should be discovered.

Directory import

1. Add an __init__.py file to a plugins directory.
2. Add helper file e.g. _structures.py into the directory and define a class there. NOTE: Files starting with _ are not used for plugin discovery.
3. Import the class using relative import from ._structures import MyClass in other plugin.
4. It should work.

[BigRoy]

Wait a second - what's this solving? 🤔

This was only ever broken inside pyblish plug-ins, right? Or did we just get luck and so far never tried/faced it with other AYON plug-ins?

Also, where in AYON have we used "Directory discovery"? 🤔 Is that for AYON addons themselvs?

[iLLiCiTiT]

This was only ever broken inside pyblish plug-ins, right? Or did we just get luck and so far never tried/faced it with other AYON plug-ins?

No, in pyblish, you just can't use dataclasses. In AYON (before this PR) you can't use from __future__ import annotations and dataclasses at the same file.

Also, where in AYON have we used "Directory discovery"?

Everywhere? We don't return path to each individual plugin file, we do register directories with plugins (ayon_maya/plugins/create is a directory).

[BigRoy]

Everywhere? We don't return path to each individual plugin file, we do register directories with plugins (ayon_maya/plugins/create is a directory).

Sure - but at some point we discover the files in those folders, and it's those files that we import that need to be in sys.modules to avoid the bug, because only those files are short-lived and loaded dynamically.

We load file pluginA.py live - only for that session, but any import it does inside of it, will end up in sys.modules just fine - we don't need to manage that. As such, those 'folders' and everything else does not need patching. We're only here to deal with the live loaded file itself. And for that the only fix we need is to add it to sys.modules before we exec the file, instead of after (which is what Pyblish did originally and got patched here).

Also, why need the hashing - doesn't pyblish use abspath and it's absolutely fine. Why do we even need to hash it? 🤔

Shall we call about this?

[BigRoy]

Let me know when you needed that other review pass for this @iLLiCiTiT

[iLLiCiTiT]

Let me know when you needed that other review pass for this

@BigRoy should be ready. Removed the package logic as the relative imports were not stable.