Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
DESCRIPTION >
Precomputed all-time commit authorship per (channel, memberId, platform, username) over
authored-commit and co-authored-commit activities. Bots, team members and organization
profiles are excluded by the cleaned source (members_sorted). Rebuilt nightly by
repo_commit_contributors_copy.pipe. Read in sorting-key order by the packages worker
git-activity contributors sync (CM-1824), which maps `channel` to a packages-db repo and
`username` to an email or GitHub login identity, and only pulls rows whose `lastUpdatedAt`
passed its stored watermark.

TAGS "Member contributors", "Repository URLs"

SCHEMA >
`channel` String,
`memberId` String,
`platform` LowCardinality(String),
`username` String,
`commitCount` UInt64,
`firstCommitAt` DateTime64(3),
`lastCommitAt` DateTime64(3),
`lastUpdatedAt` DateTime64(3),
`computedAt` DateTime

ENGINE ReplacingMergeTree
ENGINE_SORTING_KEY channel, memberId, platform, username
ENGINE_VER computedAt
36 changes: 36 additions & 0 deletions services/libs/tinybird/pipes/repo_commit_contributors_copy.pipe
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
DESCRIPTION >
Nightly copy pipe that precomputes all-time commit authorship per
(channel, memberId, platform, username) for the packages worker git-activity
contributors sync (CM-1824). Authored and co-authored commits are summed into one
count. The cleaned source already restricts members to `members_sorted`, so bots, team
members and organization profiles are excluded upstream. Pre-1971 sentinel timestamps
are dropped like in repositories_populated_copy.pipe. `lastUpdatedAt` is the latest
activityRelations.updatedAt per row, which moves on ingestion and member merges; the
consumer uses it as an incremental watermark.

TAGS "Member contributors", "Repository URLs"

NODE repo_commit_contributors_copy_data
SQL >
SELECT
channel,
memberId,
platform,
username,
count() AS commitCount,
min(timestamp) AS firstCommitAt,
max(timestamp) AS lastCommitAt,
max(updatedAt) AS lastUpdatedAt,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that updatedAt alone does not see members_sorted or repos_to_channels changes. The consumer (worker PR, same ticket) handles it on its side: the daily run is incremental on lastUpdatedAt, and every two weeks it ignores the watermark, reads the full datasource and reconciles with the same touch-and-delete pattern the governance sync uses, which catches both newly eligible historical contributors and rows that left the source. No pipe change needed; a replace-mode copy cannot cheaply version those dependency changes.

now() AS computedAt
FROM activityRelations_deduplicated_cleaned_bucket_union
Comment thread
Copilot marked this conversation as resolved.
WHERE
type IN ('authored-commit', 'co-authored-commit')
AND channel != ''
AND username != ''
AND timestamp > toDateTime('1971-01-01')
GROUP BY channel, memberId, platform, username

TYPE COPY
TARGET_DATASOURCE repo_commit_contributors_copy_ds
COPY_MODE replace
COPY_SCHEDULE 30 3 * * *
Loading