Repository navigation
feat: add a nightly copy pipe for repo commit contributors (CM-1824) #4886
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
25 changes: 25 additions & 0 deletions
25
services/libs/tinybird/datasources/repo_commit_contributors_copy_ds.datasource
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| DESCRIPTION > | ||
| Precomputed all-time commit authorship per (channel, memberId, platform, username) over | ||
| authored-commit and co-authored-commit activities. Bots, team members and organization | ||
| profiles are excluded by the cleaned source (members_sorted). Rebuilt nightly by | ||
| repo_commit_contributors_copy.pipe. Read in sorting-key order by the packages worker | ||
| git-activity contributors sync (CM-1824), which maps `channel` to a packages-db repo and | ||
| `username` to an email or GitHub login identity, and only pulls rows whose `lastUpdatedAt` | ||
| passed its stored watermark. | ||
|
|
||
| TAGS "Member contributors", "Repository URLs" | ||
|
|
||
| SCHEMA > | ||
| `channel` String, | ||
| `memberId` String, | ||
| `platform` LowCardinality(String), | ||
| `username` String, | ||
| `commitCount` UInt64, | ||
| `firstCommitAt` DateTime64(3), | ||
| `lastCommitAt` DateTime64(3), | ||
| `lastUpdatedAt` DateTime64(3), | ||
| `computedAt` DateTime | ||
|
|
||
| ENGINE ReplacingMergeTree | ||
| ENGINE_SORTING_KEY channel, memberId, platform, username | ||
| ENGINE_VER computedAt |
36 changes: 36 additions & 0 deletions
36
services/libs/tinybird/pipes/repo_commit_contributors_copy.pipe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| DESCRIPTION > | ||
| Nightly copy pipe that precomputes all-time commit authorship per | ||
| (channel, memberId, platform, username) for the packages worker git-activity | ||
| contributors sync (CM-1824). Authored and co-authored commits are summed into one | ||
| count. The cleaned source already restricts members to `members_sorted`, so bots, team | ||
| members and organization profiles are excluded upstream. Pre-1971 sentinel timestamps | ||
| are dropped like in repositories_populated_copy.pipe. `lastUpdatedAt` is the latest | ||
| activityRelations.updatedAt per row, which moves on ingestion and member merges; the | ||
| consumer uses it as an incremental watermark. | ||
|
|
||
| TAGS "Member contributors", "Repository URLs" | ||
|
|
||
| NODE repo_commit_contributors_copy_data | ||
| SQL > | ||
| SELECT | ||
| channel, | ||
| memberId, | ||
| platform, | ||
| username, | ||
| count() AS commitCount, | ||
| min(timestamp) AS firstCommitAt, | ||
| max(timestamp) AS lastCommitAt, | ||
| max(updatedAt) AS lastUpdatedAt, | ||
| now() AS computedAt | ||
| FROM activityRelations_deduplicated_cleaned_bucket_union | ||
|
Copilot marked this conversation as resolved.
|
||
| WHERE | ||
| type IN ('authored-commit', 'co-authored-commit') | ||
| AND channel != '' | ||
| AND username != '' | ||
| AND timestamp > toDateTime('1971-01-01') | ||
| GROUP BY channel, memberId, platform, username | ||
|
|
||
| TYPE COPY | ||
| TARGET_DATASOURCE repo_commit_contributors_copy_ds | ||
| COPY_MODE replace | ||
| COPY_SCHEDULE 30 3 * * * | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed that updatedAt alone does not see members_sorted or repos_to_channels changes. The consumer (worker PR, same ticket) handles it on its side: the daily run is incremental on lastUpdatedAt, and every two weeks it ignores the watermark, reads the full datasource and reconciles with the same touch-and-delete pattern the governance sync uses, which catches both newly eligible historical contributors and rows that left the source. No pipe change needed; a replace-mode copy cannot cheaply version those dependency changes.