sdk/server: Implement missing serialization modifier by hpoeche · Pull Request #660 · eclipse-basyx/basyx-python-sdk

hpoeche · 2026-10-10T09:12:12Z

Motivation

Previously, our SDK missed the JSON serialization formats "Metadata", "ValueOnly" and "Path",
that are defined in the IDTA-01001 specification. The StrippedAASToJsonEncoder implemented
a different serialization format, but followed none of the described ones (see #651).
Therefore, our server implementation responded with 501 Not Implemented to GET requests
on /$value and /$path routes, ignored ?extent and wrongly interpreted the /$metadata and
?level=core modifiers (see #650, #651).

Additionally, the order of objects in JSON arrays, which are not required to be ordered by the specification,
was not explicitly defined during serialization, which lead to different JSON strings for the same objects
(see #573).

Finally, during implementation we discovered a bug in the current serialization of SubmodelElementList
and their children. Because internally each child gets assigned a generated id_short this value was never
serialized, even when manually set.

Changes

We need to support all serialization formats, defined in IDTA-01001, as well as all serialization modifiers
from IDTA-01002 which are defined on top of these formats. We therefore chose to keep all
serialization logic in the SDK and also implement level and extent as parts of the encoder classes.

SDK changes - `basyx.aas.adapter.json`

With this change, the SDK offers a JSON encoder for every serialization format of IDTA-01001,
each one a subclass of AASToJsonEncoder in the new subpackage adapter.json.formats:
MetadataJsonEncoder, ValueOnlyJsonEncoder, ReferenceJsonEncoder and PathJsonEncoder.
To support the level and extent modifier introduced in IDTA-01002, AASToJsonEncoder itself
gets the init argument deep for level=core and the class attribute with_blob_value for
extent=WithoutBLOBValue.
All formats and modifier are tested in sdk/test/adapter/json/formats using unittest.

Collections that are unordered according to the specification are now serialized in a
canonical order, defined in adapter._ordering. For Referableobjects,modelType`
is now always the first key in JSON serialization.

Elements of SubmodelElementList now output their idShort, if it is a non generated one. This
holds for JSON and XML serialization and uses a central serializable_id_short() function that hides
generated idShort values of elements in SML as None and lets all other values pass.

Server changes

The server parses the modifiers centrally in app.adapter.modifiers and selects the matching
JSON encoder, so all content modifiers as well as level and extent are supported on the
Submodel Repository GET routes. Combinations that IDTA-01002 does not allow are rejected with
400 Bad Request.
Requests that expect an XML response fail fast with 406 Not Acceptable before object fetching, when
a serialization different from format Normal with level=deep and extent=WithBlobValue is present.
This changes existing behavior, which however was never defined in the specification.

All endpoints are tested for correct behavior under all modifier values.

Decisions we made

Where the specification is ambiguous or contradicts itself, the following decisions were made:

Operations and Capabilities have neither a ValueOnly nor a Metadata representation. They
are omitted from listings, and requesting them directly results in 400 Bad Request.
In ValueOnly, elements without a value are omitted, except within a SubmodelElementList,
where they are kept as null to preserve the indices. Further assumptions are documented
in the docstring of ValueOnlyJsonEncoder.
The idShortPaths of $path start with the idShort of the requested element. Therefore, an
element of a SubmodelElementList without an idShort has no Path representation
(400 Bad Request).
GET .../submodel-elements/$value returns a single object with the idShorts as names, so that
each value stays associated with its element.
The API spec states the response to /submodels/$value as one SubmodelValue object, which
can only hold values for one submodel, because the idShorts can duplicate across submodels.
Because this is a known error of the spec submodels-value-issue, that is corrected in a slightly
later version, we chose to implement this correct version directly.

Changed behavior

The implementation changes result in changes in the existing behavior as follows:

Normal JSON responses of the Submodel Repository omit Blob values unless
?extent=WithBLOBValue is given, as the specification defines WithoutBLOBValue as the default.
XML only supports $reference and the Normal serialization with level=deep and
extent=WithBLOBValue. Any other combination, including a request without ?extent=WithBLOBValue,
is rejected with 406 Not Acceptable. This breaks existing clients.
?limit=0 is rejected with 400 Bad Request.

AI Usage

This contribution was written with the help of Claude Opus 5.5.

Fixes #446
Fixes #573
Fixes #643
Fixes #644
From #650 and #651, GET requests are covered while PATCH requests are still wrong

Closes #649
Closes #623
Closes #621

All `$value` routes previously responded with `501 Not Implemented`. The specification leaves some points open, which are resolved as follows: - Property values are mapped to the JSON type of their `valueType`, so booleans and numbers are not quoted. Values without a JSON literal are returned as their XSD representation. `xs:integer` is unbounded in both Python and JSON and is therefore passed on as it is. - Elements without a value are omitted from Submodels, collections and Entity statements, but kept as `null` within a SubmodelElementList, as omitting them would shift the indices of the remaining elements. - Operations and Capabilities have no ValueOnly representation. They are skipped in containers and result in `400 Bad Request` when requested directly, which is how the `$metadata` route handles them. - `level=core` returns the direct children of the requested resource and their nested containers as empty, matching the examples of the specification. - The entries of `GET /submodels/{id}/submodel-elements/$value` are mapped by idShort, as a bare `SubmodelElementValue` would lose the association between a value and its element. Submodels are returned as-is, since their entries are already named. - The languages of a MultiLanguageProperty are sorted, to keep the order of the response deterministic. The ValueOnly serialization is only defined for JSON, so the routes respond with `406 Not Acceptable` if XML is requested. `extent` remains unimplemented, as on all other routes. `PATCH …/$value`, the `$path` routes and the operation invocation routes are not part of this change. Fixes eclipse-basyx#446

…ormats

Previously, the ValueOnly serialization was method based in the server package. These changes shift the code from serve package to the sdk, as ValueOnly serialiaztion is defined in Part 1 of the spec. Additionally, I restructured the code to live inside a `JSONEncoder` subclass, so it can be used the same way as `AASToJsonEncoder`. Still TODO: - Implement `level=core` serialization - Integrate this serialiazer into the server

Functionality follows eclipse-basyx#649

Showcases collapse of rules and handler methods using converter and modifiers

hpoeche

I really like the idea of the ContentModifierConverter for the endpoint definition! I found some small things we should discuss, so I can mirror them for the ValueOnly encoder.

Previously, `PathJsonEncoder` carried `deep` as a class variable and the `?level=core` variant was expressed by a dedicated `ShallowPathJsonEncoder` subclass that flipped it to `False`. Composing this axis with other content modes required another `Shallow*` sibling per encoder, so the class count scaled with the Cartesian product of axes. With this change, `deep` is passed to `PathJsonEncoder.__init__` as a keyword argument and stored on the instance. The encoder uses `default()` directly instead of extending the classmethod- based `_get_aas_class_serializers` dispatch, which avoids `cls`/`self` mismatches along the MRO when carrying per-request state. On the server side, `pick_encoder` now returns `(cls, kwargs)` and `JsonResponse.serialize` forwards the kwargs through `json.dumps` into the encoder constructor; `ResultShallowPathJsonEncoder` is removed since the single `ResultPathJsonEncoder` now covers both levels via the kwarg.

These changes implement a core serialization for the `AASToJsonEncoder` that is backward compatible to custom subclasses of this class. For all five types, that are affected by core serialization, addtional methods were introduced that call the standard serialization method and afterwards serialize all children, omitting the grandchildren. This solution allows us to pass the `deep` parameter via the `__init__()` method, which helps to avoid addtional subclass constructs. The `ValueOnlyJsonEncoder` overwrites these methods to also implement core serialization. Test are written for both classes in core serialization mode.

Fixes __init__ ordering by using local imports

Following API Specification Part 2

…iat/basyx-python-sdk into feat/serialization-formats

Previously, the order of elements in JSON arrays was non-deterministic for the same AAS. This overcomplicated easy change detection of serialized objects. With these changes, this behavior is changed. All JSON arrays are now ordered, either because the metamodel requires them to be sorted (e.g. SubmodelElementList) or sorting is done during serialization (e.g. elements of SubmodelElementCollections are now ordered by their `idShort`). The order of keys in the JSON objects is already defined by their insertion order in the Python `dict`. From now on, objects with equal content will result in identical JSON representations. Fixes eclipse-basyx#573

Cut down test_submodels.py to improve maintainability

Address review comments on the ported test_metadata.py: - Drop XML support, expect 406 for XML requests - Check responses are actually serialized as Metadata - Use FormatClient for all requests - Derive element counts from the example submodel - Add extent and 404 tests, check all level values

Address review comments on the ported test_value.py General overhaul following metadata testing and encoder redesign - Drop tests and helpers duplicating SDK Encoder tests - Replace 501 extent test expecting functional blob toggle - Add a shared value endpoint test base to enforce proper modifier combination handling - Flag list endpoint shape with TODO for now

Replace the separate handlers for $metadata, $value, $reference and $path with single handlers driven by the parsed serialization modifiers, which now select the response encoder. Extract the result serialization into a mixin, so each format can combine it directly with its SDK encoder. Listings of submodel-elements are wrapped so that each format controls the shape of its result. Filtering and pagination of $path listings is still to be resolved.

Previously, in path serialization the pagination was still applied to the SubmodelElements. Because each SubmodelElement is mapped to multiple idShortPath strings, the paginated results exceeded the limit. Now the pagination is done on the already encoded idShortPaths. Therefore the `ResultPathJsonEncoder` is extended with a `iter_paths()` method, that returns an iterator for the paths, that is then paginated using the existing `_get_slice()`.

Elements added to a SubmodelElementList without an idShort get an internal `generated_submodel_list_hack_...` idShort. Previously, this could not leak because elements in SubmodelElementList were not allowed to have idShorts. When this contstrain (AASd-120) was dropped, this lead to the following problems: - Normal + Metadata Serialization (JSON + XML): Previously the idShort of an element inside a SML was never serialized. This wrongly hid existing idShorts. To fix this, we introduce a central `serializable_id_short()` function that only hides generated idShorts of elements in SML as `None` and lets all other values pass. - Path Serialization (JSON): When elements of a SML are serialized directly, their idShort leads the idShortPaths that are generated. At this point the generated isShorts could leak, so we use the `serializable_id_short()` helper and raise a `ValueError` on generated or missing idShorts. - Server `POST .../submodel-elements/<idShortPath>`: When a new SubmodelElement is created the response header `Location` contians the address that should return this element. This was manually contructed from the new idShort, leaking generated values, which lead to a wrong address. We now use the `Referable.get_id_short_path_as_list()` method, which handles children of SML correctly. Test cases are added, that ensure no generated idShorts are leaking and all manually set idShorts are serialized.

- `limit=0` returned an empty page whose cursor points to the same start. `_get_slice` now rejects `limit < 1` with 400 for all routes. - By-path `$path` on an element without idShort (child of a SubmodelElementList) raised a ValueError (500). It now responds with 400. List children with an idShort are still supported. - Use a format client in all `$path` tests instead of the bare client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

thammel and others added 5 commits August 31, 2026 17:58

Merge branch 'feat/446-get-value-endpoints' into feat/serialization-f…

f9f9f88

…ormats

Integrate /$value endpoint tests with existing tests

a1480a0

Rewire path serialization into JSON Encoder subclass

00016d9

Functionality follows eclipse-basyx#649

JAB1305 mentioned this pull request Oct 10, 2026

Implement path serialization #649

Draft

JAB1305 added 2 commits October 11, 2026 21:55

Draft central modifier and encoder handling

2aec044

Exmplatory wire up for get submodel endpoint

54a49f4

Showcases collapse of rules and handler methods using converter and modifiers

hpoeche commented Oct 12, 2026

View reviewed changes

Comment thread server/app/adapter/modifiers.py Resolved

Comment thread server/app/adapter/modifiers.py Resolved

Comment thread sdk/basyx/aas/adapter/json/path_serialization.py Resolved

Comment thread sdk/basyx/aas/adapter/json/path_serialization.py Resolved

hpoeche and others added 5 commits October 14, 2026 12:31

Implement Reference-Serialization as AASToJsonEnocoder subclass

fde20f9

Implement Metadata-Serialization as AASToJsonEnocoder subclass

39d4685

Fix import errors and missing copyright

bd0dbcd

Add testing for path serialization

2fe15bc

Refactor bool related names for better clarity

ca6a363

hpoeche mentioned this pull request Oct 19, 2026

server: Implement the read-only $value routes #623

Draft

JAB1305 and others added 13 commits October 19, 2026 20:36

Refine PathJsonEncoder testing

d6dbf01

Refactor adapter structure

1d10056

Restructure imports

08fc93b

Fixes __init__ ordering by using local imports

Restrurcture tests following serializers

c57c2cb

Create metadata serialization test

b92fd60

Implement modifier extent=WithoutBLOBValue

05f8eb4

Realign path serialization and tests with supported types

7321c0c

Following API Specification Part 2

Merge branch 'feat/serialization-formats' of https://github.com/rwth-…

93b98cf

…iat/basyx-python-sdk into feat/serialization-formats

Fail Metadata serialization on unsupported types

873c00b

Fix docstrings

4bb8e0f

hpoeche mentioned this pull request Oct 29, 2026

CI: server-docker stage installs SDK from PyPI #680

Closed

Make modelType first key in serialization of model.Referables

3c43a08

JAB1305 added 2 commits November 2, 2026 23:00

Restructure existing serialization related tests

37e01c2

Cut down test_submodels.py to improve maintainability

Draft $path endpoint tests

61a3c8a

hpoeche commented Nov 5, 2026

View reviewed changes

JAB1305 and others added 19 commits November 5, 2026 22:16

Apply to all /$ routes + fix XML reference

21141c2

Fix XML serialization and GET /shells/$reference

6adcce2

Implement /$reference tests for all routes

c02a83c

Finalize /$path tests

6d9c099

Make /$path return flat list

3f01a9a

Omit Operation and Capability for /$metadata

08172d2

Test extent and level handling of Normal Submodel routes

f677518

Fix tests for ..submodel-elements/$value

cb31ca9

Add docstrings and comments

f997641

Update server README.md

2ac676f

Remove unsupported @overwrite decorator

831452e

Reject ?level=core on /shells and /concept-description

4397bad

Merge branch 'develop' into feat/serialization-formats

df2fd84

hpoeche marked this pull request as ready for review November 12, 2026 22:11

Navigation Menu

Repository navigation

sdk/server: Implement missing serialization modifier - #660

sdk/server: Implement missing serialization modifier#660
hpoeche wants to merge 47 commits into
eclipse-basyx:developfrom
rwth-iat:feat/serialization-formats

hpoeche commented Oct 10, 2026 •

edited

Loading

Uh oh!

hpoeche left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hpoeche commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

SDK changes - basyx.aas.adapter.json

Server changes

Decisions we made

Changed behavior

AI Usage

Uh oh!

hpoeche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hpoeche commented Oct 10, 2026 •

edited

Loading

SDK changes - `basyx.aas.adapter.json`