From 3a0dd5df3c71cb40a4b79201bec57bf74acc0658 Mon Sep 17 00:00:00 2001 From: Tyler Erickson <92862+tylere@users.noreply.github.com> Date: Fri, 23 Oct 2026 16:28:26 -0700 Subject: [PATCH 1/4] docs: add data connection concept to core-concepts page Adds a Data Connections section to the Core Concepts page, defining what a data connection is, its key configuration fields, and linking to the Bring Your Own Bucket guide for setup instructions. Closes #41 Co-Authored-By: Claude Sonnet 4.6 --- docs/about-source/core-concepts.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/about-source/core-concepts.md b/docs/about-source/core-concepts.md index 03cd2b0..ee174ba 100644 --- a/docs/about-source/core-concepts.md +++ b/docs/about-source/core-concepts.md @@ -74,6 +74,20 @@ Source uses object storage rather than traditional file storage. This has import - **No version control**: Unlike Git-based systems, Source doesn't provide granular version control on individual objects. Source is designed for publishing "fully baked" data products - **Flat namespace**: While you can organize objects with path prefixes, the underlying storage is flat rather than hierarchical +### Data Connections + +A data connection is a configuration that describes an external S3 bucket you own and how Source accesses it. Data connections are used with the [Bring Your Own Bucket (BYOB)](/bring-your-own-bucket) feature, which allows data products to be served from storage you control rather than Source-managed storage. + +Each data connection specifies: + +- **Bucket**: The external S3 bucket that holds the data +- **Authentication Type**: Whether the bucket is public (no credentials needed) or private (accessed via OIDC federation — no credentials stored) +- **Base Prefix**: An optional root folder within the bucket; individual products can read from a child path beneath it +- **Allowed Visibilities**: Which data product visibility levels (public, unlisted, restricted) may use this connection +- **Read Only**: Whether the connection allows browse/download only (required for public, unsigned connections) + +Data connections belong to an account and can be shared across multiple data products. For setup instructions, see [Bring Your Own Bucket](/bring-your-own-bucket). + ### Data Viewers Source provides built-in preview functionality for common data formats directly in the browser. This allows users to visualize and explore data before downloading. When you navigate to an individual file in Source, you'll see a preview along with metadata rather than immediately downloading the file. From 459ec15e69538612e520994df1892ffa6495bc3e Mon Sep 17 00:00:00 2001 From: Tyler Erickson <92862+tylere@users.noreply.github.com> Date: Fri, 23 Oct 2026 17:39:29 -0700 Subject: [PATCH 2/4] docs: correct the data connection concept against the code A data connection can point at S3 (or an S3-compatible store such as R2), Azure Blob or Google Cloud Storage, not only S3. Source's own storage is also configured as data connections, so the section no longer presents them as BYOB-only. The authentication bullet now describes what the data proxy serves today: no authentication, or federated access via an assumed IAM role. The read-only rule is tied to having no authentication, not to public visibility, and the section says creating connections must be enabled for an account. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/about-source/core-concepts.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/about-source/core-concepts.md b/docs/about-source/core-concepts.md index ee174ba..6ed9415 100644 --- a/docs/about-source/core-concepts.md +++ b/docs/about-source/core-concepts.md @@ -76,17 +76,17 @@ Source uses object storage rather than traditional file storage. This has import ### Data Connections -A data connection is a configuration that describes an external S3 bucket you own and how Source accesses it. Data connections are used with the [Bring Your Own Bucket (BYOB)](/bring-your-own-bucket) feature, which allows data products to be served from storage you control rather than Source-managed storage. +A data connection describes a storage location — an Amazon S3 bucket (or S3-compatible store such as Cloudflare R2), an Azure Blob container, or a Google Cloud Storage bucket — and how Source accesses it. Source's own storage is configured as data connections, and with the [Bring Your Own Bucket (BYOB)](/bring-your-own-bucket) feature an account can add connections to storage it controls. Each data connection specifies: -- **Bucket**: The external S3 bucket that holds the data -- **Authentication Type**: Whether the bucket is public (no credentials needed) or private (accessed via OIDC federation — no credentials stored) -- **Base Prefix**: An optional root folder within the bucket; individual products can read from a child path beneath it +- **Location**: The bucket (S3, GCS) or storage account and container (Azure) that holds the data +- **Authentication**: None, for a public bucket read anonymously, or federated access in which the data proxy assumes an IAM role you grant it, so no long-lived credentials are stored +- **Base Prefix**: An optional root folder within the bucket; each product's data sits beneath it, at a path set by the connection's prefix template - **Allowed Visibilities**: Which data product visibility levels (public, unlisted, restricted) may use this connection -- **Read Only**: Whether the connection allows browse/download only (required for public, unsigned connections) +- **Read Only**: Whether products can only browse and download; required for connections without authentication -Data connections belong to an account and can be shared across multiple data products. For setup instructions, see [Bring Your Own Bucket](/bring-your-own-bucket). +Data connections belong to an account and can be shared across that account's data products. Creating them is in beta and must be enabled for your account by Source staff. For setup instructions, see [Bring Your Own Bucket](/bring-your-own-bucket). ### Data Viewers From b96fed5004d51e4ea1bcf63b0f23231f8134ea4e Mon Sep 17 00:00:00 2001 From: Tyler Erickson <92862+tylere@users.noreply.github.com> Date: Fri, 23 Oct 2026 17:42:37 -0700 Subject: [PATCH 3/4] docs: base prefix applies to Azure containers too The data proxy prepends a connection's base prefix to every object key whatever the provider, so for Azure it is a folder inside the container. The Base Prefix bullet now says bucket or container, matching the Location bullet. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/about-source/core-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-source/core-concepts.md b/docs/about-source/core-concepts.md index 6ed9415..f5733d2 100644 --- a/docs/about-source/core-concepts.md +++ b/docs/about-source/core-concepts.md @@ -82,7 +82,7 @@ Each data connection specifies: - **Location**: The bucket (S3, GCS) or storage account and container (Azure) that holds the data - **Authentication**: None, for a public bucket read anonymously, or federated access in which the data proxy assumes an IAM role you grant it, so no long-lived credentials are stored -- **Base Prefix**: An optional root folder within the bucket; each product's data sits beneath it, at a path set by the connection's prefix template +- **Base Prefix**: An optional root folder within the bucket or container; each product's data sits beneath it, at a path set by the connection's prefix template - **Allowed Visibilities**: Which data product visibility levels (public, unlisted, restricted) may use this connection - **Read Only**: Whether products can only browse and download; required for connections without authentication From 5c3d493dd9860df2396c35b40ccf0e7a54013eee Mon Sep 17 00:00:00 2001 From: Tyler Erickson <92862+tylere@users.noreply.github.com> Date: Fri, 23 Oct 2026 17:44:06 -0700 Subject: [PATCH 4/4] docs: link to Bring Your Own Bucket once in the section The section's closing line already points to the BYOB page for setup, matching the single see-also link that ends the Data Products section, so the first mention is now plain text. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/about-source/core-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about-source/core-concepts.md b/docs/about-source/core-concepts.md index f5733d2..d11c159 100644 --- a/docs/about-source/core-concepts.md +++ b/docs/about-source/core-concepts.md @@ -76,7 +76,7 @@ Source uses object storage rather than traditional file storage. This has import ### Data Connections -A data connection describes a storage location — an Amazon S3 bucket (or S3-compatible store such as Cloudflare R2), an Azure Blob container, or a Google Cloud Storage bucket — and how Source accesses it. Source's own storage is configured as data connections, and with the [Bring Your Own Bucket (BYOB)](/bring-your-own-bucket) feature an account can add connections to storage it controls. +A data connection describes a storage location — an Amazon S3 bucket (or S3-compatible store such as Cloudflare R2), an Azure Blob container, or a Google Cloud Storage bucket — and how Source accesses it. Source's own storage is configured as data connections, and with the Bring Your Own Bucket (BYOB) feature an account can add connections to storage it controls. Each data connection specifies: