Kostis-S-Z · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -27,7 +27,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: '3.11'
+          python-version: '3.11.4'
 
       - name: Install uv
         uses: astral-sh/setup-uv@v4
@@ -57,7 +57,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: '3.11'
+          python-version: '3.11.4'
 
       - name: Install uv
         uses: astral-sh/setup-uv@v4
@@ -69,4 +69,7 @@ jobs:
         run: uv pip install --system -e . --group dev
 
       - name: Run e2e tests
+        env:
+          MDC_TEST_API_KEY: ${{ secrets.MDC_TEST_API_KEY }}
+          MDC_TEST_API_URL: ${{ secrets.MDC_TEST_API_URL }}
         run: pytest -q tests/e2e
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -10,4 +10,4 @@ repos:
     rev: v1.18.2
     hooks:
       - id: mypy
-        additional_dependencies: [types-requests, types-PyYAML]
+        additional_dependencies: [types-requests, types-PyYAML, pandas-stubs~=2.3.3, pydantic>=2.7.0]
diff --git a/README.md b/README.md
@@ -28,7 +28,7 @@ The official Python SDK for accessing and contributing to the [Mozilla Data Coll
 
 ## Features
 
-- **Download datasets** to local storage with automatic resume after an interruption.
+- **Download datasets** to local storage with automatic resume after an interruption, or just a dataset's **sample** to evaluate it first.
 - **Load datasets into pandas DataFrames or HuggingFace `Dataset` objects**, with optional lazy audio decoding.
 - **Browse and search the public catalog** from Python, no API key needed.
 - **Upload datasets programmatically**: create a submission, set its metadata, upload the archive, and send it for review in a single call.
@@ -198,6 +198,9 @@ Parts of the dataset are uploaded concurrently and uploads are resumable. Tune t
 > [!TIP]
 > To also attach an optional sample of your dataset, pass `sample_file_path="/path/to/dataset-sample.tar.gz"` to `create_submission_with_upload`, or upload it separately with `upload_sample_file(file_path=..., submission_id=...)`.
 
+> [!TIP]
+> A submission that has not been published yet can be discarded, together with its uploaded files, with `delete_submission(submission_id)`. This cannot be undone.
+
 > [!TIP]
 > To upload a new `.tar.gz` version to an already approved dataset, call `upload_dataset_file(file_path=..., submission_id=...)` directly. Find the submission under **Profile → Uploads**, open the approved dataset, and copy the value after `/profile/submissions/` in the URL. Note that this value is the submission ID, which is different from the public dataset ID.
 

diff --git a/docs/api.md b/docs/api.md
@@ -15,6 +15,12 @@
 
 ::: datacollective.models
 
+::: datacollective.errors
+
+::: datacollective.archive_utils
+
+::: datacollective.hf_utils
+
 ::: datacollective.schema
 
 ::: datacollective.schema_loaders.base

diff --git a/docs/index.md b/docs/index.md
@@ -47,7 +47,7 @@ Optional:
 
 - `MDC_DOWNLOAD_PATH` - Local directory where datasets will be downloaded
   (defaults to `~/.mozdata/datasets`).
-- `MDC_LOG_PATH` - File path for logs (defaults to a `.log` file in the download directory).
+- `MDC_LOG_PATH` - Directory or file path for the log file (defaults to `~/.mozdata/datacollective.log`).
 
 Example using environment variables directly:
 
@@ -59,8 +59,8 @@ export MDC_LOG_PATH=~/.mozdata
 
 ### `.env` file
 
-The client will automatically load configuration from a `.env` file in your
-project root or present working directory.
+The client will automatically load configuration from a `.env` file in the
+directory you run your code from, or in any of its parent directories.
 
 Create a file named `.env`:
 
@@ -79,7 +79,7 @@ MDC_LOG_PATH=~/.mozdata
 The library has the ability to emit detailed logs for debugging and monitoring purposes, 
 on both the console and a local log file. This is disabled by default.
 To enable logging, pass `enable_logging=True` to the main functions.
-When enabled, logs are stored in a `.log` file in the download directory.
+When enabled, logs are written to `~/.mozdata/datacollective.log` (rotated at 10 MB, 5 backups kept).
 You can change the log file location by setting the `MDC_LOG_PATH` environment variable to your desired path,
 either through the `.env` file or by executing on your terminal:
 
@@ -347,12 +347,12 @@ for dataset in page:
     print(f"{dataset.id}  {dataset.name}  ({dataset.task}, {dataset.locale}, {size_gb:.1f} GB)")
 ```
 
-Other filters: `format` (e.g. `"WAV"`), `upload_date` (`"today"`, `"thisWeek"`,
+Other filters: `file_format` (e.g. `"WAV"`), `upload_date` (`"today"`, `"thisWeek"`,
 `"thisMonth"`, `"thisYear"`), `has_sample=True` for datasets with a sample file,
 `pricing` (`"free"` or `"compensated"`) and `sort_direction` (`"asc"`/`"desc"`).
-`limit` accepts 1 to 100 results per page (the platform defaults to 24).
+`results_per_page` accepts 1 to 100 results per page (the platform defaults to 24).
 
-To walk through every page, increase `page` until you have collected `total` items:
+To walk through every page, increase `page_number` until you have collected `total` items:
 
 ```python
 from datacollective import list_datasets

diff --git a/docs/schema_documentation.md b/docs/schema_documentation.md
@@ -16,7 +16,7 @@ print(df.head())
 
 Under the hood, `load_dataset()` performs the following steps automatically:
 
-1. **Resolve the schema**: check local cache or the schema registry for `schema.yaml`. If the dataset is not registered this step raises a warning, so we never download an unsupported archive.
+1. **Resolve the schema**: check the schema registry for `schema.yaml`. If the dataset is not registered this step raises a `RuntimeError`, so we never download an unsupported archive.
 2. **Download** the archive (with resume support). The schema we fetched in step 1 tells the loader how the files are structured.
 3. **Extract** the `.tar.gz` / `.tgz` archive to a local directory.
 4. **Parse** the YAML into a validated `DatasetSchema` (Pydantic model) and dispatch to the loader for the schema's `root_strategy` (index, glob, …), which returns the final **DataFrame**. When the schema declares a `task` with a known contract (ASR, TTS, LLM), the loaded DataFrame is checked for the task's required logical columns; a `TaskValidationWarning` is emitted if any are missing (the DataFrame is still returned).
@@ -389,15 +389,15 @@ at parse time.
 
 ## Complete examples
 
-For full examples for each strategy, visit the respective strategy documentation pages under [docs/loaders/](./loaders/).
+For full examples for each strategy, visit the respective strategy documentation pages under [Loading Strategies](loaders/index.md).
 
 ## Schema caching
 
 The SDK caches the `schema.yaml` inside the extracted dataset directory to
 minimise API calls:
 
 1. When `load_dataset()` runs, it obtains the **archive checksum** from the
-   download plan.
+   dataset details returned by the API.
 2. If a local `schema.yaml` exists and its stored `checksum` matches the
    archive checksum, the cached copy is used and no network request needed.
 3. If the checksums differ (dataset was updated) or no cache exists, the

diff --git a/docs/upload.md b/docs/upload.md
@@ -368,7 +368,7 @@ print(f"Upload complete! File Upload ID: {upload_state.fileUploadId}")
 ```
 
 !!! tip
-    You can also find your submission ID by going to your [Uploads](https://mozilladatacollective.com/profile/uploads) in your profile, click on the dataset submission of your choice, and the URL will contain the submission ID (e.g., `https://mozilladatacollective.com/submissions/cmmjpewijXXXXXXXXX`).
+    You can also find your submission ID by going to your [Uploads](https://mozilladatacollective.com/profile/uploads) in your profile, click on the dataset submission of your choice, and the URL will contain the submission ID (e.g., `https://mozilladatacollective.com/profile/submissions/cmmjpewijXXXXXXXXX`).
 
 ### Step 4 (Optional): Upload a Sample File
 

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -1,6 +1,8 @@
 site_name: Mozilla Data Collective Python SDK
 repo_url: https://github.com/Mozilla-Data-Collective/datacollective-python
 repo_name: datacollective-python
+exclude_docs: |
+  downloads/
 
 nav:
   - Home: index.md
@@ -36,10 +38,8 @@ theme:
       toggle:
         icon: material/lightbulb-outline
         name: Switch to light mode
-  logo: docs/mdc_logo.png
-  favicon: docs/mdc_ccc.png
-  extra_css:
-    - assets/custom.css
+  logo: mdc_logo.png
+  favicon: mdc_ccc.png
   features:
     - content.code.copy
     - content.tabs.link

diff --git a/pyproject.toml b/pyproject.toml
@@ -18,7 +18,7 @@ authors = [
 ]
 description = "A Python package for interfacing with the Mozilla Data Collective's API"
 readme = "README.md"
-requires-python = ">=3.11"
+requires-python = ">=3.11.4"
 classifiers = [
     "Programming Language :: Python :: 3",
     "Operating System :: OS Independent",
@@ -39,6 +39,10 @@ dev = [
     "bump-my-version>=1.2.6",
     "datasets>=5.0.0",
     "pandas-stubs~=2.3.3",
+    "ruff==0.14.6",
+    "mypy==1.18.2",
+    "types-requests",
+    "types-PyYAML",
 ]
 
 docs = [
@@ -52,13 +56,16 @@ Homepage = "https://github.com/Mozilla-Data-Collective/datacollective-python"
 Documentation = "https://Mozilla-Data-Collective.github.io/datacollective-python/"
 Issues = "https://github.com/Mozilla-Data-Collective/datacollective-python/issues"
 
-[project.scripts]
-lint = "ruff:__main__"
-typecheck = "mypy:__main__"
-
 [tool.uv]
 preview = true
 
+[tool.mypy]
+plugins = ["pydantic.mypy"]
+
+[[tool.mypy.overrides]]
+module = ["fox_progress_bar", "fox_progress_bar.*", "datasets", "datasets.*"]
+ignore_missing_imports = true
+
 [[tool.uv.index]]
 name = "testpypi"
 url = "https://test.pypi.org/simple/"
@@ -76,7 +83,7 @@ regex = false
 ignore_missing_version = false
 tag = false
 sign_tags = false
-tag_name = "v{new_version}"
+tag_name = "{new_version}"
 tag_message = "Bump version: {current_version} → {new_version}"
 allow_dirty = false
 commit = false

diff --git a/src/datacollective/api_utils.py b/src/datacollective/api_utils.py
@@ -31,7 +31,7 @@
 GENERIC_ERROR_STATUSES = frozenset({400, 402, 409, 422})
 
 
-load_dotenv(find_dotenv())
+load_dotenv(find_dotenv(usecwd=True))
 
 
 def _send_api_request(
@@ -80,7 +80,7 @@ def _send_api_request(
 
     logger.debug(
         f"API request: {method.upper()} {url} (stream={stream})\n"
-        f"json_body: {_redact_sensitive(json_body)}"
+        f"json_body: {_redact_sensitive(json_body)}\n"
         f"params: {_redact_sensitive(params)}"
     )
 

diff --git a/src/datacollective/schema_loaders/base.py b/src/datacollective/schema_loaders/base.py
@@ -364,7 +364,7 @@ def _normalize_column_key(self, column: str) -> str:
 
     def _resolve_file_path(
         self,
-        value: object,
+        value: Any,
         col_map: ColumnMapping,
         row: pd.Series | None = None,
         misses: list[str] | None = None,
@@ -426,7 +426,7 @@ def _resolve_file_path(
 
     def _load_file_content(
         self,
-        value: object,
+        value: Any,
         col_map: ColumnMapping,
         row: pd.Series | None = None,
         misses: list[str] | None = None,

diff --git a/tests/conftest.py b/tests/conftest.py
@@ -1,3 +1,3 @@
 from dotenv import find_dotenv, load_dotenv
 
-load_dotenv(find_dotenv(), override=False)
+load_dotenv(find_dotenv(usecwd=True), override=False)
diff --git a/tests/schema_loaders/strategies/test_multi_sections_loader.py b/tests/schema_loaders/strategies/test_multi_sections_loader.py
@@ -158,7 +158,7 @@ def test_file_paths_resolve_relative_to_section_directory(
         df = MultiSectionsLoader(schema, tmp_path).load()
 
         assert not [w for w in recwarn if issubclass(w.category, DataLoadWarning)]
-        resolved = {row.section: Path(row.audio_path) for row in df.itertuples()}
+        resolved = {row.section: Path(str(row.audio_path)) for row in df.itertuples()}
         assert (
             resolved["General"]
             == (tmp_path / "dataset" / "General" / "clips" / "general.wav").resolve()

diff --git a/tests/test_models.py b/tests/test_models.py
@@ -189,7 +189,7 @@ def test_currency_is_set_only_when_paid() -> None:
 
 def test_currency_is_not_a_settable_field() -> None:
     with pytest.raises(ValidationError):
-        DatasetSubmission(isPaid=True, basePriceCents=25_000, currency="eur")
+        DatasetSubmission(isPaid=True, basePriceCents=25_000, currency="eur")  # type: ignore[call-arg]  # extra="forbid" at runtime
 
 
 def test_dataset_details_requires_id() -> None:
@@ -217,7 +217,7 @@ def test_dataset_details_accepts_unknown_enum_values() -> None:
     details = DatasetDetails.model_validate(
         {"id": "abc", "visibility": "hidden", "task": "BRAND_NEW_TASK"}
     )
-    assert details.visibility == "hidden"
+    assert details.visibility == "hidden"  # type: ignore[attr-defined]  # extra="allow" field
     assert details.task == "BRAND_NEW_TASK"
-Original file line number
+Diff line change
@@ Expand Up @@
     ```
     !!! tip
-        You can also find your submission ID by going to your [Uploads](https://mozilladatacollective.com/profile/uploads) in your profile, click on the dataset submission of your choice, and the URL will contain the submission ID (e.g., `https://mozilladatacollective.com/submissions/cmmjpewijXXXXXXXXX`).
+        You can also find your submission ID by going to your [Uploads](https://mozilladatacollective.com/profile/uploads) in your profile, click on the dataset submission of your choice, and the URL will contain the submission ID (e.g., `https://mozilladatacollective.com/profile/submissions/cmmjpewijXXXXXXXXX`).
     ### Step 4 (Optional): Upload a Sample File
@@ Expand Down @@