Repository navigation
fix(cloud): preserve existing stream sync settings in set_selected_streams - #1461
Closed
devin-ai-integration[bot] wants to merge 1 commit into
Closed
devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
…reams Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1794330309-preserve-stream-config-on-select' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1794330309-preserve-stream-config-on-select'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Contributor
Code coverage overviewLanguages: Python Python / code-coverage/pytest-no-creds
View coverage of the 10 most impacted files.
Python / code-coverage/pytest
View coverage of the 10 most impacted files.
Python / code-coverage/pytest-unit
Line coverage data for the main branch is not yet available. View coverage of the 10 most covered files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
Aaron ("AJ") Steers (aaronsteers)
marked this pull request as ready for review
November 11, 2026 18:24
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the change preserving existing sync settings when stream selections are updated.
No open issues 🎉
What changed in this PR
Fixes Cloud stream selection updates so streams that remain selected retain their existing sync settings instead of resetting to Full Refresh | Overwrite.
Changes:
- Fetches fresh connection settings and reuses existing stream configurations; newly selected streams retain API defaults.
- Documents the preservation behavior in the SDK and Cloud MCP tool.
- Adds regression tests for serialized PATCH payloads, stale-cache bypass, empty configurations, and GET failures.
| File | Description |
|---|---|
tests/unit_tests/test_cloud_stream_selection.py |
Adds six regression cases covering configuration preservation and update behavior. |
airbyte/mcp/cloud.py |
Clarifies stream sync settings behavior in the tool docstring. |
airbyte/cloud/connections.py |
Fetches fresh configurations before updating stream selection. |
airbyte/_util/api_util.py |
Adds optional reuse of existing stream configurations. |
Review effort: Balanced
Contributor
Author
|
Closing in favor of #1460, which carries the same stream-preservation fix on Posted by Devin, on behalf of Aaron ("AJ") Steers (@aaronsteers). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
👉 TL;DR: Changing which streams a Cloud connection syncs (via the SDK or the Cloud MCP tool) silently reset every stream that stayed selected to Full Refresh | Overwrite. Streams that stay selected now keep their existing sync settings; only newly added streams get the API defaults.
Specifically,
CloudConnection.set_selected_streams()now fetches fresh connection info and passes itsconfigurations.streamstoapi_util.build_stream_configurations(..., existing_streams=...), which resends each keptStreamConfigurationunchanged (sync_mode,cursor_field,primary_key,selected_fields,mappers,namespace, ...).Resolves https://github.com/airbytehq/oncall/issues/13947:
Close on merge: no — customer-filed; support confirms the fix with the customer before closing.
Changes
api_util.build_stream_configurations: new keyword-onlyexisting_streams; matching streams are reused as-is, unmatched names getStreamConfiguration(name=...)(unchanged behavior forcreate_connection, which doesn't pass it).CloudConnection.set_selected_streams: fetches connection info withforce_refresh=Truebefore building the PATCH body; docstring documents the behavior.set_cloud_connection_selected_streamsMCP tool docstring: says existing streams keep their sync settings and new streams use the API default.tests/unit_tests/test_cloud_stream_selection.pypinning the serialized PATCH body.Root cause
The public API (
PATCH /v1/connections/{id}) maps a stream configuration with nosyncModetoFULL_REFRESH+OVERWRITE. Previously the PATCH body contained only stream names, so every kept stream was rewritten as Full Refresh | Overwrite.New-stream defaults
Newly added streams still send only
name, i.e. the API default (Full Refresh | Overwrite). Picking an incremental default was left out to keep scope minimal.Test Coverage
tests/unit_tests/test_cloud_stream_selection.py(6 tests, mocked API): kept streams retainsyncMode/cursorField/primaryKeyand other fields, deselected streams are dropped, new streams are added name-only, stale cache is bypassed, empty configurations work, and GET failures propagate. 4 of the 6 fail with the production change removed (the PATCH body contained[{'name': 'kept'}, {'name': 'new'}]); all pass with it. Full unit suite:1302 passed, 1 skipped.Follow-ups
syncModeis omitted, and record catalog changes in the connection timeline.exec: "sandbox": executable file not found). It needs one re-run once the image is fixed.Link to Devin session: https://app.devin.ai/sessions/e2115dbec37c49dd8f35b0763070f74b
Open in Devin Desktop: https://app.devin.ai/desktop/session/e2115dbec37c49dd8f35b0763070f74b?variant=devin