Release 2.3.3

A patch release on the 2.3 line, focused on storage credentials, diagram rendering, and type validation.

S3 stores no longer require static credentials. access_key and secret_key are now optional in an s3 store spec — omit both and the AWS credential chain resolves an ambient identity (EC2 instance profile, IRSA, ECS task role, SSO), matching how the gcs and azure protocols already behaved. This is what lets a pipeline run under an assumed role with no long-lived keys in its configuration. Setting exactly one of the two is now rejected at validation with a clear message instead of failing later inside the AWS client. See Configure Storage.

Diagrams were redesigned. dj.Diagram output moves to the DataJoint brand palette — Manual green, Lookup grey, Imported blue, Computed orange, with tier shapes unchanged — and renamed foreign keys are now drawn in amber, distinct from the Computed tier. Line weight now encodes cardinality and only cardinality: thick for one-to-one, thin for one-to-many. It previously also tried to signal the master-part relationship, which made the two unreadable together. A new display.diagram_theme setting takes auto (default), light, or dark; auto emits a single SVG that adapts to the viewer's light or dark mode. See Read Diagrams and Configuration.

Attribute types are validated at declaration and insert. Four related gaps closed, the most serious being that inserting a NumPy array into a native blob attribute silently stored the array's text representation, with no error on insert and none on fetch. decimal(M,D) unsigned is also accepted again after being rejected in 2.x, and misspelled native types are now caught at declaration rather than producing invalid DDL at the server.

Also: braces in a table or attribute comment no longer crash declaration, alter() no longer loses an added attribute's declared type, file-protocol stores are safe on Windows, and non-unique foreign-key columns are indexed on PostgreSQL.

Upgrading from 2.3.2? No API breaks. Two of the fixes above do reject input that previously passed silently — a misspelled native type at declaration, and a NumPy array inserted into a native blob. Both cases were already broken rather than working: the first produced invalid DDL at the server, the second corrupted the stored value. Anything affected was losing data.

Full details in What's New in DataJoint 2.3.

---

What's Changed

• docs: fix missing pixi -e flags in CONTRIBUTING.md(#1541)@gtouloumes
• ci: add Windows unit-test job to catch OS-specific path bugs(#1522)@dimitri-yatsenko

⚡️ Enhancements

• feat(#1543): adopt the DataJoint brand tier colors in diagram themes(#1544)@dimitri-yatsenko
• feat(#1532): modernize dj.Diagram rendering style(#1534)@dimitri-yatsenko
• fix(#1537): allow s3 stores without static credentials(#1538)@dimitri-yatsenko
• fix(#1512): index foreign-key columns on PostgreSQL (coverage-aware)(#1526)@dimitri-yatsenko

🐛 Bug Fixes

• fix: validate attribute types at declaration and insert (#1527, #1528, #1529, #1530)(#1531)@dimitri-yatsenko
• fix: braces in comments crash declaration, and alter() drops type metadata(#1548)@MilagrosMarin
• fix(diagram): collapsed-to-collapsed edges carry no per-foreign-key style(#1545)@dimitri-yatsenko
• fix(#1535): render diagrams with unresolved (raw table-name) nodes on PostgreSQL(#1536)@dimitri-yatsenko
• fix(#1532): diagram edge weight encodes cardinality, not master-part(#1533)@dimitri-yatsenko
• Use POSIX separators in file-protocol storage URLs and paths(#1521)@lochhh

📝 Documentation

• docs(codecs): thread the connection config in the SchemaCodec example(#1549)@MilagrosMarin
• docs(settings): correct the config-precedence docstring(#1542)@dimitri-yatsenko

Full Changelog: v2.3.2...v2.3.3

Release 2.3.2

Overview

v2.3.2 is a patch release — internal hardening and bug fixes on top of v2.3.1, with no breaking changes to the public API. The main items: the dependency graph moved to networkx.MultiDiGraph (correctly modeling multiple foreign keys between the same pair of tables), a ~jobs-table fix for UUID foreign-key-derived primary keys, removal of the process-wide sys.excepthook (uncaught exceptions now use Python's default handler, with tracebacks rendered by the log formatter), and a garbage-collection fix that avoids acting on partial storage listings. The package version is now derived from the git tag via hatch-vcs.

---

• test: cover parallel foreign-key edges to the same parent (#1492)(#1514)@dimitri-yatsenko
• tests: add coverage for deferred audit findings + isolate test_settings from ambient env(#1513)@MilagrosMarin
• chore: normalize stdlib copy imports to import copy(#1509)@dimitri-yatsenko
• docs(autopopulate): fix stale finally-comment; document upstream in class Attributes(#1510)@MilagrosMarin
• docs: post-2.3.1 stale-text sweep (3 small drift fixes)(#1511)@MilagrosMarin
• chore(release-flow): stale doc pointers, RELEASE_MEMO cleanup, and dead badge step(#1504)@MilagrosMarin
• docs(codecs): fix stale storage-path and hash-algorithm docstrings(#1505)@MilagrosMarin
• [github-actions]Update version.py to Release 2.3.1(#1502)@github-actions[bot]

⚡️ Enhancements

• refactor(graph): migrate Dependencies & Diagram to nx.MultiDiGraph (#1492)(#1508)@dimitri-yatsenko

🐛 Bug Fixes

• Fix jobs table generation for uuid FK-derived primary keys (#1515)(#1518)@dimitri-yatsenko
• Remove process-wide sys.excepthook; harden log formatter (#1516)(#1517)@dimitri-yatsenko
• fix(release): derive version from git tag via hatch-vcs (#1503)(#1507)@dimitri-yatsenko
• fix(gc): don't act on partial storage listings(#1506)@MilagrosMarin

Full Changelog: v2.3.1...v2.3.2

Release 2.3.1

Highlights

• populate() performance — self.upstream is now built lazily and dependencies are no longer reloaded per key, removing redundant work from populate() loops (#1499, #1493).
• Cascade / part-integrity fix — a cascaded delete now restricts the master from every restricted part and validates part integrity, fixing multi-part-master cascades (#1484).
• Garbage-collection correctness — codec-driven reference discovery with file-level, schema-addressed orphan matching (#1479, #1469).
• strict_provenance runtime guardrail retired — the best-effort runtime guard is removed; provenance remains an intrinsic structural property of the pipeline (#1489, #1488).
• Python 3.14 support — supported range extended through 3.14 with min/max exercised in CI; MySQL 8.4 LTS added to the test matrix (#1490, #1498).

⚡️ Enhancements

• fix(#1493): build self.upstream lazily; stop reloading deps per populate key(#1499)@dimitri-yatsenko
• ci: test MySQL 8.0 and 8.4 (LTS) in the matrix; parameterize MySQL image(#1498)@dimitri-yatsenko
• test(cascade): pin part-integrity paths (rollback, empty-materialization, U3 arm)(#1485)@dimitri-yatsenko
• Retire strict_provenance runtime guardrail (#1488)(#1489)@dimitri-yatsenko
• Extend Python support to 3.14; test min/max in CI(#1490)@dimitri-yatsenko

🐛 Bug Fixes

• fix(cascade): restrict master from every restricted Part; validate part_integrity(#1484)@dimitri-yatsenko
• ci(pre-commit): pin numpy<2.2 for mypy stub compatibility(#1491)@dimitri-yatsenko
• fix(#1469): codec-driven GC reference discovery + file-level orphan matching(#1479)@dimitri-yatsenko

📝 Documentation

• docs: clarify _propagate_part_to_master materialization is delete-order, not just MySQL 1093(#1495)@dimitri-yatsenko
• docs(#1484): document pair-keyed part-to-master invariant in _propagate_restrictions(#1494)@dimitri-yatsenko

Full Changelog: v2.3.0...v2.3.1

Release 2.3.0

Highlights

DataJoint 2.3 builds on the 2.2.x line with a set of features centered on provenance — knowing, and being able to trust, exactly which upstream data a computed result was derived from.

The headline is the provenance trinity. Diagram.trace constructs the upstream view of any result; self.upstream makes those declared ancestors ergonomic to read inside make(); and the opt-in strict_provenance flag turns the long-standing "read only from declared dependencies, write only to self" convention into something the framework actively checks. Together they move DataJoint's core provenance promise from a convention people hope to follow to one the framework helps construct and — when enabled — enforce.

Around that:

• SparkAdapter Codec Protocol — typed codecs can expose their decoded values as Spark-native types, opening lakehouse / Delta-Sharing consumers (e.g. Databricks) to columns that were previously opaque blobs.
• dj.deploy.set_replica_identity — a new dj.deploy module configures PostgreSQL REPLICA IDENTITY for change-data-capture pipelines.
• Cascade fix — part_integrity="cascade" is now correct across Part-of-Part and renamed-foreign-key chains (the same upward-propagation machinery Diagram.trace builds on).

No breaking changes. strict_provenance defaults off and everything else is additive, so existing pipelines are unaffected. See What's New in 2.3 for the full narrative.

🚀 Features

• feat(#1425): strict_provenance config flag for runtime enforcement(#1474)@dimitri-yatsenko
• feat(#1424): self.upstream property for pre-restricted ancestor access(#1473)@dimitri-yatsenko
• feat(#1423): Diagram.trace() for upstream restriction propagation(#1471)@dimitri-yatsenko
• feat(#1458): SparkAdapter Codec Protocol(#1472)@dimitri-yatsenko
• feat(deploy): set_replica_identity for PostgreSQL CDC (#1447)(#1466)@dimitri-yatsenko

🐛 Bug Fixes

• fix(#1429): cascade through FK chain for part_integrity="cascade"(#1468)@dimitri-yatsenko
• fix(#1454): refresh ~lineage on every @schema decoration(#1467)@dimitri-yatsenko
• fix(staged_insert): converge metadata shape with ObjectCodec.encode(#1465)@dimitri-yatsenko

Full Changelog: v2.2.4...v2.3.0

Release 2.2.4

Highlights

A small follow-up patch on 2.2.3 cleaning up rough edges discovered during the docs review:

• dj.StorageAdapter and dj.get_storage_adapter() exposed at top level. The Storage Adapter plugin contract documented in the spec used these names as if they were re-exported from the top-level datajoint namespace, but they were not. Plugin authors following the spec literally hit AttributeError. Both symbols are now in datajoint.__all__ and importable as dj.StorageAdapter / dj.get_storage_adapter.
• packaging declared as an explicit dependency. src/datajoint/migrate.py imports packaging.version.Version, but the package was never declared in pyproject.toml. With Python 3.12 venvs no longer shipping setuptools (which used to pull packaging transitively), pip install datajoint==2.2.3 into a clean Py3.12 venv hit ModuleNotFoundError: No module named 'packaging' on import datajoint. Fixed.
• Source-side version markers refreshed in settings.py docstrings and comments to match the public docs.

⚡️ Enhancements

• feat: export StorageAdapter and get_storage_adapter at top level (#1463)@dimitri-yatsenko

🐛 Bug Fixes

• fix: declare packaging as an explicit dependency (#1462)@dimitri-yatsenko

📝 Documentation

• docs: refresh settings.py version markers (#1461)@dimitri-yatsenko

Companion docs: datajoint/datajoint-docs#172 — env-var configuration of stores + Storage Adapter API spec.

Full Changelog: v2.2.3...v2.2.4

Release 2.2.3

Highlights

This patch release adds env-var configuration for object stores (closes the gap for env-var-only deployments such as the DataJoint platform, Kubernetes, and Lambda) and ships several minor fixes and documentation improvements.

• DJ_STORES — JSON-encoded copy of the stores dict, same shape as in datajoint.json. Replaces the file's stores block when set, with arbitrary adapter-defined attribute names (token, api_key, workspace_url…).
• DJ_IGNORE_CONFIG_FILE (default false) — when true, skips datajoint.json, the project .secrets/, and /run/secrets/datajoint/ entirely. Hard guarantee that no file on disk leaks into config.
• .secrets/stores.<name>.<attr> now accepts any attribute, not only access_key / secret_key — supports plugin-registered adapters (e.g. Databricks Unity Catalog Volumes) with non-S3-style credential fields.

Precedence (high → low): programmatic dj.config["stores"] > DJ_STORES > datajoint.json > .secrets/stores.<name>.<attr> (fills missing attrs only).

⚡️ Enhancements

• feat(config): DJ_STORES env var + DJ_IGNORE_CONFIG_FILE flag + arbitrary .secrets/ attrs (#1452)@dimitri-yatsenko
• chore: remove .vscode and .devcontainer (#1459)@dimitri-yatsenko

📝 Documentation

• docs: extend datajoint.migrate timeline to 2.4 or 2.5 (#1456)@dimitri-yatsenko
• docs: fix code formatting on rendered codecs API page (#1455)@dimitri-yatsenko
• docs: use relative path for README pipeline image (#1450)@MilagrosMarin
• docs: refresh README pipeline illustration (#1448)@dimitri-yatsenko
• docs: cite DataJoint 2.0 manuscript and add RRID (#1449)@dimitri-yatsenko

Companion docs: datajoint/datajoint-docs#172

Full Changelog: v2.2.2...v2.2.3

Release 2.2.2

⚡️ Enhancements

• feat: add StorageAdapter plugin system for third-party storage protocols (#1432) @kushalbakshi
• refactor: simplify index query, warn on MariaDB connections (#1439) @dimitri-yatsenko
• feat: export AutoPopulate and Job in public API (#1431) @dimitri-yatsenko

🐛 Bug Fixes

• fix(#1442): scan_*_references reads raw JSON metadata instead of decoded codec output (#1444) @kushalbakshi
• fix(#1438): preserve json flag for MariaDB longtext-aliased columns (#1443) @dimitri-yatsenko
• fix(#1433): clear error for leading-underscore attribute names (#1441) @dimitri-yatsenko
• fix: remove EXPRESSION column dependency in get_indexes_sql() (#1437) @dimitri-yatsenko
• fix: preserve NOT NULL constraint in migrate_columns() (#1435) @dimitri-yatsenko

Full Changelog: v2.2.1...v2.2.2

Release 2.2.1

Released: 2026-04-09

⚡️ Enhancements

• feat: add database.name setting for PostgreSQL connections (#1426) @kushalbakshi
  • Configurable database name via dj.config["database.name"], DJ_DATABASE_NAME env var, or database_name kwarg on Connection
  • Extracts _build_connect_kwargs() helper to eliminate duplicated connection parameter construction
  • __repr__ and logger show database name when set (e.g., user@host:5432/my_db)
  • Warns when database.name is set with MySQL backend
  • Deprecates database_prefix (warning emitted at Schema.activate())

📝 Documentation

• Fix CONTRIBUTING.md link for PyPI (#1422) @dimitri-yatsenko
• Add PostgreSQL configuration to CONTRIBUTING.md
• Document database.name setting and database_prefix deprecation (datajoint-docs#159)
• Fix dead links in llms.txt — now auto-generated from mkdocs nav (datajoint-docs#160)

Full Changelog: v2.2.0...v2.2.1

v2.2.0

What's Changed

For a comprehensive overview of all new features, see What's New in DataJoint 2.2.

Added

• Graph-driven cascade delete and restrict on Diagram (#1407, fixes #865, #1110): New Diagram.cascade(), Diagram.restrict(), Diagram.prune(), and Diagram.counts() methods replace the error-driven cascade approach. Delete and drop operations now use the pipeline DAG to determine affected tables before executing, with full dry-run support via safemode=True.
• Thread-safe mode with dj.Instance (#1404): New dj.Instance() class provides independent database connections with connection-scoped configuration. Enables safe concurrent access from multiple threads (e.g., web servers, parallel workers).
• Directory references in <filepath@store> (#1415, fixes #1410): Filepath storage now supports directory references. is_dir is detected dynamically; existence checks and storage operations handle directories correctly.

Fixed

• populate() with reserve_jobs=True ignores restrictions (#1414, fixes #1413): Restrictions are now correctly applied when fetching pending keys in distributed mode, matching the behavior of direct (non-distributed) populate.
• Populate antijoin uses .proj() for correct pending key computation (#1405): Fixes cases where overlapping secondary attributes caused incorrect pending key calculations.
• Allow attribute names starting with 'index' in declarations (#1412, fixes #1411): Table definitions with attribute names like index_value no longer raise parse errors.
• Cascade delete failures on MySQL 8 (fixes #1110): The graph-driven cascade in #1407 eliminates the error code mismatch (1217 vs 1451) that caused cascade delete failures on MySQL 8.

Changed

• Backend-agnostic quoting and adapter abstractions (#1419): Refactored identifier quoting, table name construction, and schema queries into adapter methods for cleaner multi-backend (MySQL + PostgreSQL) support.
• skip_duplicates=True behavior documented for PostgreSQL (#1417, fixes #1049): PostgreSQL already enforces secondary unique constraints when skip_duplicates=True (raises DuplicateError on secondary unique conflicts, unlike MySQL which skips silently). This asymmetry is now documented and tested.

Full Changelog: v2.1.1...v2.2.0

v2.1.1

What's Changed

Bug Fixes

• Atomic job reservation to prevent race condition (#1399, fixes #1398): Job.reserve() now uses a single atomic UPDATE ... WHERE status='pending' instead of a non-atomic SELECT→UPDATE pattern, preventing multiple workers from reserving the same key.
• Hide comments from table preview display (#1393): SQL comments in table definitions are no longer shown in .preview() output.
• Correct Part table names in diagrams (#1392): Part tables now display correctly in diagrams by properly stripping the module prefix.

Removals

• Remove size_on_disk (#1395): Removed size_on_disk property from Table and Schema classes. Use database-native tools for storage metrics.

Full Changelog: v2.1.0...v2.1.1