Integrator: Refactor to `ContextPlugin` and add type hints by BigRoy · Pull Request #1992 · ynput/ayon-core

BigRoy · 2026-08-07T19:05:53Z

Changelog Description

Integrator: Refactor to ContextPlugin and add type hints

Additional info

Turning this into an integrator allows us to optimize some queries, like the prefetching of products, version data, etc. and also join the file transactions into one threaded bulk and the OperationsSession into fewer bulks. This should make big ingests with many instances faster. But timings will still need to be made.

Draft for now, because it needs more performance tests but also general discussion

Testing notes:

Publishing should work from all DCCs

BigRoy · 2026-08-10T08:05:14Z

@iLLiCiTiT can you give me your thoughts on this?

Copilot

Pull request overview

Refactors the publish integrator to operate as a pyblish.api.ContextPlugin with more upfront data prefetching (products/versions/representations) and introduces stronger typing via dataclasses and type hints to support faster large ingests.

Changes:

Converted IntegrateAsset from InstancePlugin to ContextPlugin and integrated all eligible instances in one pass.
Added _PreparedInstance / _PreparedRepresentation dataclasses and expanded type hints across the integrator code.
Added operation chunking (operations_chunk_size) and prefetch logic via get_products / get_versions / get_representations.

Suppressed comments (2)

client/ayon_core/plugins/publish/integrate.py:915

Using assert here is unsafe because asserts can be stripped with -O, and then repre_context["frame"] = ... could crash if repre_context is unexpectedly None. Prefer an explicit guard (and keep the final if repre_context is None: raise as the authoritative error).

            if not is_udim:
                assert repre_context is not None
                repre_context["frame"] = first_index_padded

client/ayon_core/plugins/publish/integrate.py:920

This assert can be stripped with -O, so the next line could crash if repre_context is None. Use a guard instead of an assert for safety.

            if instance.data.get("renderlayer"):
                assert repre_context is not None
                repre_context["renderlayer"] = instance.data["renderlayer"]

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…erations for a single instance.

…com/ynput/ayon-core into enhancement/integrate_ContextPlugin

BigRoy · 2026-08-21T11:47:37Z

May be good to do some performance testing with this. Maybe @antirotor loves to do that, or @jakubjezek001 comparing it with latest develop on a large CSV ingest or alike? :)

moonyuet · 2026-09-14T05:38:53Z

I agree that we need to do some performance testing and my concern is whether prefetching data in this PR rapidly increasing memory usage.

Multiple instances
Single very large dataset (CSV ingest etc.)
Extremely large instance count

iLLiCiTiT · 2026-09-15T09:33:05Z

 import logging
 import sys
 import copy
+from dataclasses import dataclass


Dataclasses can't be used. Same issue as here #2050

They can, if you don't have from __future__ import annotations in the file. ;)
As such, do we still care?

Also, given how painful this is it makes me wonder if we should have a dataclass decorator wrapper that avoids the bug, just so we can use dataclasses. Because this stuff gets painful easily otherwise.

I have it in my TODO list to fix the issue.

It's fixed in pyblish pyblish/pyblish-base#403 btw ;) there's just no new pyblish release since.

BTW I think from __future__ import annotations is issue with core's import logic, but with pyblish import logic it fails completelly event without the import.

Pretty sure the pyblish one also ONLY happened if that from future import was there. @antirotor worked on that fix at the time.

Added PR #2062 to fix the dataclasses issue.

Pretty sure the pyblish one also ONLY happened if that from future import was there. @antirotor worked on that fix at the time.

Oh, yeah, but that is not released, and 100% is not in any dependency package we have. So in practice we can't rely on it.

iLLiCiTiT · 2026-09-15T09:47:09Z

                    template_data["udim"] = index
                else:
                    template_data["frame"] = index
                template_filled = path_template_obj.format_strict(


NOTE: In ideal case scenario we should collect file info now. The template_filled object has rootless path with root that was 100% used instead of guessing it later on.

iLLiCiTiT · 2026-09-15T09:50:52Z

-            "anatomy_data": template_data,
-            "transfers": transfers,
+        return _PreparedRepresentation(
+            representation=repre_doc,


Why the hell is the variable still named repre_doc? 😆

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Turn Integrator into a ContextPlugin and add type hints.

6ae6ef3

BigRoy requested review from antirotor and iLLiCiTiT August 7, 2026 19:05

BigRoy self-assigned this Aug 7, 2026

BigRoy added the type: enhancement Improvement of existing functionality or minor addition label Aug 7, 2026

ynbot added the size/M label Aug 7, 2026

BigRoy requested a lite review from Copilot August 10, 2026 08:05

Merge branch 'develop' into enhancement/integrate_ContextPlugin

71e6f2f

Copilot started reviewing on behalf of BigRoy August 10, 2026 08:05 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

Comment thread client/ayon_core/plugins/publish/integrate.py

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

iLLiCiTiT reviewed Aug 10, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

ynbot added this to PR reviewing Aug 10, 2026

ynbot moved this to Review In Progress in PR reviewing Aug 10, 2026

iLLiCiTiT reviewed Aug 10, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

BigRoy and others added 4 commits August 10, 2026 14:11

Potential fix for pull request finding

78aced7

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Chunk operations at minimum per istance so we never commit half of op…

6c889b8

…erations for a single instance.

Merge branch 'enhancement/integrate_ContextPlugin' of https://github.…

cb4fc76

…com/ynput/ayon-core into enhancement/integrate_ContextPlugin

Replace assert with enforce Type Hints

2d748f5

BigRoy requested a review from iLLiCiTiT August 13, 2026 18:55

BigRoy added 2 commits August 14, 2026 00:22

Merge branch 'develop' into enhancement/integrate_ContextPlugin

9a36096

Merge branch 'develop' into enhancement/integrate_ContextPlugin

0e2ce0d

BigRoy marked this pull request as ready for review August 21, 2026 11:46

BigRoy requested a review from jakubjezek001 August 21, 2026 11:47

BigRoy requested review from LiborBatek and moonyuet September 13, 2026 21:36

Merge branch 'develop' into enhancement/integrate_ContextPlugin

fb08f24

iLLiCiTiT reviewed Sep 15, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py

iLLiCiTiT reviewed Sep 15, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

iLLiCiTiT reviewed Sep 15, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

iLLiCiTiT reviewed Sep 15, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

iLLiCiTiT reviewed Sep 15, 2026

View reviewed changes

Comment thread client/ayon_core/plugins/publish/integrate.py Outdated

BigRoy and others added 4 commits September 15, 2026 14:01

Update client/ayon_core/plugins/publish/integrate.py

ac77bad

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Update client/ayon_core/plugins/publish/integrate.py

afc47ad

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Update client/ayon_core/plugins/publish/integrate.py

fa03d31

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Update client/ayon_core/plugins/publish/integrate.py

c7e9925

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Uh oh!

Conversation

BigRoy commented Aug 7, 2026

Changelog Description

Additional info

Testing notes:

Uh oh!

BigRoy commented Aug 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BigRoy commented Aug 21, 2026

Uh oh!

moonyuet commented Sep 14, 2026

Uh oh!

iLLiCiTiT Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BigRoy Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iLLiCiTiT Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

BigRoy Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

iLLiCiTiT Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

BigRoy Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

iLLiCiTiT Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iLLiCiTiT Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iLLiCiTiT Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

iLLiCiTiT Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iLLiCiTiT Sep 15, 2026 •

edited

Loading

BigRoy Sep 15, 2026 •

edited

Loading

iLLiCiTiT Sep 15, 2026 •

edited

Loading

iLLiCiTiT Sep 15, 2026 •

edited

Loading