Carry the terms a 51Did was created under in the identifier

[jwrosewell]

Draft. The specification this implements has merged as 51Degrees/specifications#27, and this stays in draft until the cloud has been updated to write both fields. pipeline-dotnet#409 goes first, because the cloud is .NET and needs that package published before it can write anything.

What this is

Two fields of the 51Did payload, read by pipeline.did.

The Terms is one byte after the match key, saying which terms document the identifier was created under. A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, so the identifier has to say which terms those are. In OpenRTB there is somewhere to put the answer, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own and any hop can drop an answer carried beside it without the identifier looking any different, so the terms travel inside the identifier.

The payload version is bits 4 and 5 of the Flags byte, saying which layout the payload follows.

The Terms

The byte is an index into a table in the specification and is not a version number, so that a later document can live at any address rather than only at one a number could compose. An index is never reused and never repointed once published, because repointing one would rewrite what an identifier already issued says it agreed to.

getTerms() answers with the address of the document. The package turns the index into the address, so a caller never handles the byte.

Index	Document	getTerms()
0	Not stated in the identifier	null
1	Model Terms for Marketing, version 2	https://m4ow.uk/mtm/2.txt
any other	One this package cannot name	null

That one member is the whole of the addition to the public surface. The package answers with the address and never fetches it, and the receiver decides what to do with the document.

The payload version

This package reads version 0 and refuses every other version with FodIdParseStatus.UNSUPPORTED_PAYLOAD_VERSION, with the throwing readers naming the version they found in the message. FodIdParseResult carries the version so the message can name it, and that field is package-private, since a caller has nothing to decide with it.

No field is read under the layout this package knows once the version says otherwise. A later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent, which is worse than refusing. A version that nothing checks protects nothing.

The version is not exposed. Either this package read the layout, in which case the accessors are the answer, or it did not, in which case there is no identifier to read fields from.

The parts that are easy to get wrong

No address is ever built from an index this package cannot name. That would name a document nobody wrote, and a receiver would record having accepted terms that do not exist. An index of zero and an index added after this release therefore give a caller the same answer, which is deliberate, since both say the identifier does not give the terms and the answer has to come from somewhere else.

The Terms byte is read at the offset the match key ends at. Bits 6-7 of the Flags byte select the type and the type sets the match key length, so the offset follows the type, and a reader with one fixed offset reads a match key byte on a RANDOM identifier.

A payload with no byte after the match key reads as index 0. That is the same answer a byte holding zero gives, so absence and zero mean the same thing, no reader has to tell them apart and no presence flag or format version is needed. The byte is not counted in the minimum payload length for any identifier type.

A RESERVED identifier reads as index 0. That type is not yet assigned and this package reads it best effort, taking every byte after the header as the match key, so no byte follows the match key. The specification asks for exactly that, since it says a payload with no byte after the match key is a Terms of zero, and reading the last context byte as a terms index instead would be worse. It is still worth stating, because a RESERVED identifier that did carry a Terms would be reported as carrying none, and nothing can be done about that until the type is assigned and the length of its match key is known.

No address does not mean the identifier is unrestricted. It says only that the identifier does not carry the answer, so the answer has to come from the surrounding protocol. Where an identifier may go is a separate question getUsage() answers, which still bars a non-marketing identifier from a demand source.

The Terms sits before the creator context section. An issuer writing a context section writes the Terms byte before it. The lengths of that section belong to the cloud and this package holds no upper bound of its own, so a longer payload is accepted and the same four fields are read.

The version is read on its own bits. A reader masking the wrong bits would refuse a version 0 identifier of some usages, or let a later version through, so there is a test over every combination of the usage and type bits.

Files changed

• pipeline.did/src/main/java/fiftyone/pipeline/did/Terms.java, the named values the specification gives for each index and fromIndex(int). It is package-private, since the package turns the index into an address and a caller never handles the byte.
• pipeline.did/src/main/java/fiftyone/pipeline/did/FodId.java, reads the version before any field, refuses a version it does not know, reads the Terms byte wherever the type puts the end of the match key, and adds getTerms() after getMatchKey() so it sits in payload order.
• pipeline.did/src/main/java/fiftyone/pipeline/did/FodIdParseStatus.java, UNSUPPORTED_PAYLOAD_VERSION.
• pipeline.did/src/main/java/fiftyone/pipeline/did/FodIdParseResult.java, carries the refused version for the message.
• pipeline.did/src/main/java/fiftyone/pipeline/did/package-info.java, the payload fields a 51Did exposes.
• pipeline.did/src/test/java/fiftyone/pipeline/did/FodIdTestFactory.java, the creating side, so it writes both new fields.
• pipeline.did/src/test/java/fiftyone/pipeline/did/FodIdParseTests.java, the tests below.
• pipeline.did/README.md, the payload layout, a payload version section and a section on which terms a 51Did was created under.

The change stays inside pipeline.did. The developer example module is outside the package and printing the member there is a separate change.

The test fixtures write what an issuer writes

canonicalPayload() and canonicalRandomPayload() are the creating side, so they now carry the payload version 0 in the Flags byte and the Terms byte their usage calls for, being index 1 for the personalized marketing canonical payload and index 0 for the non-marketing Random one. payloadEndingAtMatchKey() and randomPayloadEndingAtMatchKey() are the fixtures for a payload that carries no Terms byte, which a reader takes as index 0 and which no issuer would write.

Tests

In FodIdParseTests, over the Terms:

1. getTerms_NoByteAfterTheMatchKey_HasNoAddress, a payload ending at the match key answers with no address, for both the 32 byte and the 16 byte type.
2. getTerms_ZeroByte_AnswersAsAbsenceDoes, a byte holding zero answers exactly as no byte at all does.
3. getTerms_IndexOne_ModelTermsAddressForBothKeyLengths, index 1 returns https://m4ow.uk/mtm/2.txt exactly, for both match key lengths, and each match key is compared against the same payload without the byte so neither loses a byte to the Terms.
4. getTerms_IndexThisPackageDoesNotKnow_HasNoAddress, indexes 2, 127, 200 and 255 answer with no address. 255 is there because the byte is signed in Java and an unmasked read would answer -1.
5. terms_EveryIndexOutsideTheTable_IsUnknownWithNoUrl, every one of the 254 indexes the table does not carry has no address.
6. terms_EveryMemberAgreesWithTheTable, every member reads back from its own index and one that names a document has an https address.
7. getTerms_ByteThenContextSection_ReadAtTheRightOffset, a payload carrying a Terms byte and then a 512 byte creator context section still reads the match key, the Terms and the whole payload, and still verifies.

And over the payload version:

7. version_Zero_ReadsEveryField, version 0 reads the type, usage, match key and terms.
8. version_NotZero_IsRefused, versions 1, 2 and 3 answer UNSUPPORTED_PAYLOAD_VERSION and hand back nothing.
9. version_NotZero_MessageNamesTheVersion, the message names the version it found.
10. version_IsReadApartFromTheUsageAndTypeBits, every combination of the usage and type bits reads at version 0 and is refused at 1, 2 and 3.

The terms table

The address, the index and the name are one table, so adding a terms document is one row and not a search for every place a number was written down. That is the point of the byte being an index rather than a version number, and the first version of this change did not honour it, spreading the same fact over two, being the enumeration and a switch that repeated the index places that could disagree.

Each member now carries its index alongside its address, and the index to member map is built from the members in a static block rather than written out again, so fromIndex is one lookup and a new document is one new member.

A test walks every member and fails if one does not read back from its own index, or names a document with no address, which is what would happen if a member and its address were added apart.

What was run and what it reported

Apache Maven 3.9.16 on OpenJDK 21, from the repository root.

mvn -pl pipeline.did -am test

FodIdParseTests 32 run, 0 failures. FodIdTests 40 run, 0 failures. DidClientTests 68 run, 0 failures. DidClientLiveTests 2 run, 2 skipped, being the tests that need the cloud. Module total 142 run, 0 failures, 0 errors, 2 skipped.

mvn -pl pipeline.did javadoc:javadoc

BUILD SUCCESS, no warnings, so every link added resolves.

The value names

The specification fixes the names every package uses for the three values, which in the Java form are NOT_STATED, MODEL_TERMS_FOR_MARKETING_2 and UNKNOWN, so the constants here follow the table in 51Degrees/specifications#27 rather than a choice made in this pull request. The enum is package-private and the names are for a reader of the code, since the public surface is the address alone.