Skip to content

refactor(resharding): migrate copy_data to the tasks api - #1526

Open
meskill wants to merge 3 commits into
meskill-2026-09-04-test-resharding---ruklfrom
meskill-2026-09-10-fix-api---olm
Open

refactor(resharding): migrate copy_data to the tasks api#1526
meskill wants to merge 3 commits into
meskill-2026-09-04-test-resharding---ruklfrom
meskill-2026-09-10-fix-api---olm

Conversation

@meskill

@meskill meskill commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Refactor table copies logic to the api/tasks.

  • move out logic for copying from orchestrator/publisher/table to separate data_sync entities
  • extend the info types in pgdog-stats for data copy
  • migrate to tasks context that is sent to ee
  • add WorkerPool entity to distribute load between resharding replicas more evenly - use work stealing with cap configured with resharding_parallel_copies for single replica. Previously the distribution was not fair and some of the replicas could acquire significantly more work
  • add more tests coverage for the process.

I verified the copy_data benchmarks and didn't notice any performance changes after the migration. (the benchmarks doesn't cover multiple replicas though, but I think that should improve performance)

@meskill
meskill added this pull request to stack #1527 September 11, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant