You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Clone the folders of a recursive asset clone before its assets - #2160
Targets 2025.4 and reaches 2026.3 and 2026.x through the forward merges. Follow-up to #2112, which fixed the clone target lookup for names with regex characters. Recursive asset clones still end incomplete with plain names as soon as more than one worker consumes pimcore_generic_execution_engine.
Additional info
Root cause
CloneService::cloneAssetRecursively() clones the selected folder and then starts a job with one FOR_EACH step for all descendants. The execution engine dispatches one AssetCloneMessage per descendant at once. Every message looks up its target folder by path (getNewCloneTarget()), and that folder is the clone of its parent, created by another message of the same job. With several workers the messages run in parallel, so an asset or sub folder is regularly processed before its target exists:
Asset with path: /…/01_January_copy_1/Production Planning not found
The transport has max_retries: 0 and the job runs with CONTEXT_STOP_ON_ERROR, so the job stops and the clone stays incomplete. Direct children are never affected (their target is created synchronously in the request), deeper levels are.
Data object and document clones are not affected: they create one job step per element, and the execution engine runs steps strictly one after another.
Fix
The job now has two steps (same pattern as the folder patching in PatchService):
FolderCloneMessage, ONCE: one message clones all folders, parents first.
AssetCloneMessage, FOR_EACH: the assets, one message each, in parallel. Every target folder exists at this point.
A step is only added if it has something to do (an empty FOR_EACH step would still be dispatched once, without an element).
The descendants are now read from the database (CloneDescendantsRepository, an Asset\Listing with escapeLike() on the path prefix, folders ordered by LENGTH(path) so parents come first), the same query the Classic UI uses for its copy. Previously they came from the search index, so elements with a pending index update were silently left out. They are loaded before the source is cloned, so a clone into a sub folder of the source does not clone its own new copy again.
The job building moved into CloneJobFactory so it can be tested without a database.
Tests
New unit tests: CloneJobFactoryTest (two steps, step order and modes, folders passed in the given order, only the assets selected for the parallel step, folder-only / asset-only / empty trees) and FolderCloneHandlerTest (every folder is cloned into its target before the next one is resolved; the step reports its progress over the folders). Both fail on the previous code.
Full Unit suite: 660 tests green on 2025.4 (PHP 8.3, core 12.3.12.1 = lowest); the same change on 2026.3: 1608 green (PHP 8.4). No pre-existing failures. PHPStan (level 6) and php-cs-fixer clean on the changed files.
Reproduction on a running instance (2026.x demo, real clone endpoint, 4 workers on pimcore_generic_execution_engine), tree with 245 elements, 3 levels below the cloned folder, plain names:
before: 10/10 runs failed within ~1 s, 1–55 of 245 elements cloned, error as above;
after: 10/10 runs complete (245/245, ~9 s each), source untouched.
Edge cases after the fix, 4 workers, all complete and exact: sub folder older than its new parent (moved), 01_January next to 01XJanuary (_ is escaped in the LIKE), folder with only assets, tree with only folders, clone into a sub folder of the source, clone of a single asset (no job).
Performance (before vs. after, same instance, 2 runs each)
Scenario
Before (request / total)
After (request / total)
50 empty sub folders + 1,000 assets in one folder, 4 workers (the case that also works before)
21.1 s / 126.0 s, 23.8 s / 132.0 s
3.2 s / 129.0 s, 3.3 s / 131.3 s
10 folders with 1,000 assets, 1 worker
21.5 s / 507.7 s, 23.8 s / 497.5 s
2.4 s / 527.4 s, 2.3 s / 519.9 s
With several workers the total time is unchanged, the request returns ~8× faster: the asset messages are no longer dispatched inside the HTTP request but by the worker once the folder step is done.
With a single worker the dispatch (~20 ms per message, execution engine middleware) can no longer overlap with the processing, so the total is ~4 % higher. The work itself is the same (asset cloning ~493 s before vs. ~495 s after).
Large trees stay slow, as before: every element message loads and writes the whole job run, which contains all selected elements (~1 MB for 20,000 assets, ~560 ms to load). That is an execution engine issue, unchanged by this PR, and tracked separately.
Backward compatibility
CloneServiceInterface is unchanged. CloneService (@internal, final readonly) gets other constructor dependencies.
New @internal classes only: message, handler, repository, factory, value object.
StepConfig and JobSteps each get one new case (additive).
Observable change: a recursive asset clone job now has up to two steps (studio_ee_job_step_asset_folder_cloning, studio_ee_job_step_asset_cloning), English translation added.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Active issues allow restricted folders to block permitted sibling copies, prevent cancellation from stopping subsequent folder clones, and report incorrect progress for the ONCE step.
This follow-up to #2112 addresses recursive asset-clone races by creating parent folders before dispatching parallel asset copies. The fix belongs in the backend job-building and execution layers; CloneServiceInterface remains unchanged.
Changes:
Splits cloning into a sequential folder step followed by parallel asset processing.
Reads descendants from the database before cloning, with escaped path matching and parent-first folder ordering.
Adds job-factory and handler tests, service wiring, message routing, and a translated step label.
Tests were not run during this review. The permission-sensitive database selection warrants maintainer review.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
SQL-backed descendant selection and cross-step sequencing under multiple workers require maintainer integration and security validation because unavailable Pimcore and execution-engine dependencies prevented independent runtime verification.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Add regression test for parent-first query ordering
The parent-first query ordering is not covered by the new tests: tests/Unit/Asset/Service/ExecutionEngine/CloneJobFactoryTest.php:41–56 and tests/Unit/Asset/ExecutionEngine/FolderCloneHandlerTest.php:84–93 only verify preservation of supplied IDs. A regression to ID ordering would leave those tests green while causing missing-target failures for an older folder moved beneath a newer parent. Add a repository-level regression test asserting that the parent is returned before its child regardless of their IDs.
A recursive asset clone dispatched one message per descendant, all at once. Every message looks up
its target folder by path, and that folder is created by another message of the same job. With
several workers on the pimcore_generic_execution_engine transport, an asset or sub folder was
processed before its target folder existed: "Asset with path ... not found", the job stopped and
the clone stayed incomplete.
The job now has two steps: one message clones all folders, parents first, then the assets are
cloned one message each, in parallel. The descendants are read from the database instead of the
search index, so elements with a pending index update are cloned as well.
Co-Authored-By: Claude <noreply@anthropic.com>
The folder step is a single message, so the regular progress counted it as one of the selected
assets of the next step: no event for larger trees, wrong percentages for small ones. It now
publishes its own progress over the cloned folders.
FolderCloneMessage / FolderCloneHandler: the imports of the previous names exceeded 120 characters.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes in this pull request
Related: pimcore/service-operations#1606
Targets 2025.4 and reaches 2026.3 and 2026.x through the forward merges. Follow-up to #2112, which fixed the clone target lookup for names with regex characters. Recursive asset clones still end incomplete with plain names as soon as more than one worker consumes
pimcore_generic_execution_engine.Additional info
Root cause
CloneService::cloneAssetRecursively()clones the selected folder and then starts a job with one FOR_EACH step for all descendants. The execution engine dispatches oneAssetCloneMessageper descendant at once. Every message looks up its target folder by path (getNewCloneTarget()), and that folder is the clone of its parent, created by another message of the same job. With several workers the messages run in parallel, so an asset or sub folder is regularly processed before its target exists:The transport has
max_retries: 0and the job runs withCONTEXT_STOP_ON_ERROR, so the job stops and the clone stays incomplete. Direct children are never affected (their target is created synchronously in the request), deeper levels are.Data object and document clones are not affected: they create one job step per element, and the execution engine runs steps strictly one after another.
Fix
The job now has two steps (same pattern as the folder patching in
PatchService):FolderCloneMessage, ONCE: one message clones all folders, parents first.AssetCloneMessage, FOR_EACH: the assets, one message each, in parallel. Every target folder exists at this point.A step is only added if it has something to do (an empty FOR_EACH step would still be dispatched once, without an element).
The descendants are now read from the database (
CloneDescendantsRepository, anAsset\ListingwithescapeLike()on the path prefix, folders ordered byLENGTH(path)so parents come first), the same query the Classic UI uses for its copy. Previously they came from the search index, so elements with a pending index update were silently left out. They are loaded before the source is cloned, so a clone into a sub folder of the source does not clone its own new copy again.The job building moved into
CloneJobFactoryso it can be tested without a database.Tests
CloneJobFactoryTest(two steps, step order and modes, folders passed in the given order, only the assets selected for the parallel step, folder-only / asset-only / empty trees) andFolderCloneHandlerTest(every folder is cloned into its target before the next one is resolved; the step reports its progress over the folders). Both fail on the previous code.pimcore_generic_execution_engine), tree with 245 elements, 3 levels below the cloned folder, plain names:01_Januarynext to01XJanuary(_is escaped in theLIKE), folder with only assets, tree with only folders, clone into a sub folder of the source, clone of a single asset (no job).Performance (before vs. after, same instance, 2 runs each)
Backward compatibility
CloneServiceInterfaceis unchanged.CloneService(@internal,final readonly) gets other constructor dependencies.@internalclasses only: message, handler, repository, factory, value object.StepConfigandJobStepseach get one new case (additive).studio_ee_job_step_asset_folder_cloning,studio_ee_job_step_asset_cloning), English translation added.🤖 Generated with Claude Code