Skip to content

Resource Listing Distillation

Jump to bottom
jvandegriff edited this page Nov 12, 2026 · 19 revisions

Resource Listings in HAPI

See this ticket: https://github.com/hapi-server/data-specification/issues/218

See also dataset schema issue tag.

See also https://github.com/hapi-server/data-specification/wiki/availability This is a distillation of the previous wiki page on file listings, which we now want to call Resource Listing.

Here is what is left to do in order to add this into the spec:

  1. Bob: add PR to the spec: statement that HAPI times in the primary time column must never decrease (and that duplicates are OK in some cases, like resource listings?)
  2. Jon: add PR to the spec: the new checksum type as one of the valid stringType options:
    • https://github.com/hapi-server/data-specification/issues/291
    • make branch; use content from here to describe enumerated algorithm types
    • see below for link to existing algorithm list online that we should reference
    • are we absolutely sure there is not already a JSON spec for this somewhere (maybe we check Andriy's stack of standards?)
  3. for datasetSchema, we will use a list of objects
  4. Bob: draft a document that will end up in the spec to describe the datasetSchema for a ResourceListing (see coord schema for capitalization)
  5. notes on how accurate should the startDate and stopDate should be? Are the day boundary times good enough, or should the server open up the file to get actual times (and thus have to save those out somewhere!!)?
    • Recommendations:
    • Use case: "I want all the files that contain data in this time range."
    • Response of server: It is OK to report more files (that have data outside the requested range, so that the data in the listed files needs to be trimmed if needed), but it is not OK to report fewer files so that some of the data in the requested time range comes from files that did not get returned in the list.
    • The best solution is to have the file listing report exact start and stop times for the content of each file.
    • Ok solutions are to report nominal times for files, as long as the main use case is met (all files with data in given time range are returned).
    • Example: Burst data - it is ok to report whole day; better to report just the burst hour;
  6. make sure we are OK with the interaction of HAPI queries with the inherent time ranges presented in a Resource Listing (returned dataset rows included only if start time is within request time - some users might not expect this, but it's standard HAPI behavior which ends up being somewhat non-intuitive for these listing datasets)
  7. possibly revisit: why are mod date and size optional?

The previous wiki had a lot of outdated and inconsistent info, and this page has just what I think we have decided to pursue, and is also designed to explain it in terms digestible by new contributors, and also by AI.

It would be useful and would enhance interoperability across data systems if certain types of responses from a HAPI server could be standardized. A schema that gives specific meaning to the contents of a HAPI dataset provides both semantic and syntactic surety to a client, enabling automated treatment of the HAPI content. Low hanging fruit for this kind of standardized expression are simple data types that are not measurements, but commonly experienced entities within a data system. This includes File Listings, Event Lists, and Data Availability Information, all of which are themselves a kind of meta-information about the content at a HAPI server (or other data providers, in the case of event lists).

This page summarizes our discussion and conclusions about how to implement a file or uri listing schema for HAPI that would allow a server to emit a standardized expression describing file names and other file properties. Think of it as a web-based, standardized expression for the Unix ls -l command. Many web servers provide file listings, and many programmers have written HTML scraping routines to grab these listings, but what if there could be a semantically solid, syntactically rigid way of expressing this? Any HAPI client could then get a file listing and know what it was dealing with. This would save so much time and hassle. Even for AI tools, the ability to use a standardized schema for common information will help automate routine tasks.

HAPI datasets can already represent a file listing. The stringType for a parameter will identify that parameter as a retrievable resource if the stringType is set to uri. An informal file listing is then just a dataset with at least a time and a URI. But many file listings include other details, and this document describes a nearly complete proposal to create a HAPI datasetSchema that will describe and constrain what can be in a HAPI dataset for it to be considered a standardized file listing.

Events lists are more generally just a time stamp and a message associated with the time stamp. Often events lists will have a time range with a start and end time. There is a separate group working on an event list standard, and once that group has a schema, the HAPI team will be very quick to create a HAPI-based expression of that standard. For now, this document does not address Event Lists.

Recommended dataset naming for a listing of files is to end the dataset identifier string with "filelist", e.g., "MyDataset/filelist" (not "filelist/MyDataset"), and also add "datasetSchema="filelist" in the info response.

One option is to start with a base schema class which is just a "Listing of Times". But there seems to be little benefit to this, since every HAPI dataset has this property anyway.

Resource Listing Basics

We decided to call it Resource Listing since URIs are more general than just files (maybe S3 contents, DOI's, etc).

The Resource Listing schema should capture basic file characteristics from a generic operating system, and make them available in a maximally interoperable way. The schema should contain minimal required elements, but allow optional elements that can provide more details about the files, especially those characteristics that are often needed when dealing with lists of scientific data files. The core items are of course the time and the file name, but here the time has a special scientific meaning, namely the start time of the data in the file. Because HAPI already has this concept, the label startDate will be used for this parameter. The URI parameter will just be uri. Optional but important items are modification date and size (both normal characteristics from the OS) and optional science-specific concepts are stopDate, checksum and version.

There was some debate about calling it a File Lsting or URI Listing and we setteld on Resource Listing since since we really aren't listing just plain files because this is a network-based access mechanism, which is focused on URIs instead of files, and those URIs can point to a variety of resources, mostly files, but possibly other things.

Resource Listing Schema

So a Resource Listing schema needs to be able to describe time-ordered lists of data-centric URIs (files, S3 buckets, service endpoints, etc).

Note that the widest possible use of URIs is not supported here, because URIs can also capture mailto: and other non-data entities! This listing capability uses URI syntax, but only a subset of URIs are relevant for this kind of resource listing. The URI should point to an actionable, downloadable, data-centered content that is publicly accessible. The MIME type should indicate content that data users or systems could reasonably be expected to ingest.

The verifier could check for odd-ball URI types!

Resource Listing Schema

parameter name parameter type required / recommended / optional description
startDate isotime required start date/time of coverage
uri string (and stringType of uri) required name of resource (file, image, endpoint, etc)
modificationDate isotime recommended last modified time of the resource
size number - see below recommended size of the file in B (bytes)
checksum string (and stringType constrains the checkSumAlgorithm) optional value for checksum with the specified algorithm
creationDate isotime optional time the resource was created
accessDate isotime optional date & time the resource was last accessed
stopDate isotime optional stop date & time of coverage; exclusive (like HAPI request end times)
version string optional project-specific version indicator

About the size

Units must be in B and values formatted as an integer, unless the size is greater than 9,007,199,254,740,991 (~9 PB), in which case scientific notation should be used. Note that with this approach, a size is greater than ~9 PB cannot be represented with an exact number of bytes.

about version numbers

There are no constraints on the version strings. If there are multiple version of files over the life of a project, it may be desirable to offer a version indicator to distinguish between subsequent revisions (better calibration, revised background subtraction, etc).

Comment on column names: we tried to be consistent with the names used in the info response.

Ordering of "resourceListing" columns:

  • required columns must be present in the order given (startDate, then uri)
  • recommended and/or optional columns must follow required columns and can be in any order
  • any number of user-added columns can be present (other than the listed recommended / optional columns) and these can be interleaved among the optional columns

HAPI will need to define a new stringType for checksum (see ticket #273 in the HAPI specification GitHub repo). There are curated lists of hash algorithm names (one use case: HTTP headers). Consider these lists of has algorithm names:

The first link has a CSV file with agreed on names - the algorithm name must be one of the items in the first column of that CSV file.

A long would be helpful here, but that should be a separate discussion, because maybe we also add float too, for HAPI 4.0, and also complex numbers? Adding more numeric types to HAPI is beyond the scope of this discussion so we are going with integer for now for file size.

Events Lists are also listings and a basic representation could have these elements:

  • startDate of time coverage (required, isotime)
  • stopDate of time coverage (isotime, required) (Documentation acknowledges that this should be the same for an event that is just an instant, i.e., has no duration)
  • label required string label or name for this event

Some example extensions to Event List:

  • latitude
  • longitude

There will be lots of extensions to the event list, and these will be domain-specific. The Solar community within Heliophysics already has a very rich set of event characteristics for things that happen on (and now around) the Sun. See the event properties as described at the HEK site maintained by LMSAL.

We still need to talk about names for values of the datasetSchema entry, since this is the first of several potential schemas (availability or good-time intervals, event lists, and then also science types, like ground magnetometer data, space mag data, energetic particles, plasma parameters, plasma waves, etc).

Example proposed output, note datasetSchema 

{
    "HAPI": "3.2",
    "x_createdAt": "2017-02-21T17:27Z",
    "modificationDate": "2026-01-01T00:00Z",
    "datasetSchema": "list>resourceList>jpgFileList",
    "parameters": [
        {
            "length": 20,
            "name": "startDate",
            "type": "isotime",
            "x_format": "$Y-$m-$dT$H:$M:$SZ",
            "fill": null,
            "units": "UTC",
            "timeStampLocation" : "begin"
        },
        {
            "description": "Picture of the creek, unmodified",
            "fill": null,
            "name": "uri",
            "length": 26,
            "type": "string",
            "units": null,
            "stringType": {
                "uri": {
                    "base": "https://cottagesystems.com/data/hapi/pics/",
                    "mediaType": "image/jpeg"
                }
            }
        },
        {
            "description": "File modification time",
            "name": "modificationDate",
            "type": "isotime",
            "fill": null,
            "x_format": "$Y-$m-$dT$H:$MZ",
            "length": 17,
            "units": "UTC"
        },
        {
            "description": "File size in kilobytes",
            "name": "size",
            "fill": null,
            "type": "integer",
            "units": "B"
        },
        { "name": "checksum",
         "type": "string",
         "length": 32,
         "stringType": {"checksum": { "algorithm": "md5" } },
         "fill": null,
         "description": "pre-calculated checksum using MD5 algorithm"}
    ],
    "sampleStartDate": "2023-01-01T00:00Z",
    "sampleStopDate": "2023-02-01T00:00Z",
    "startDate": "2022-11-01T00:00Z",
    "stopDate": "2026-03-06T00:00Z",
    "cadence": "PT10M",
    "status": {
        "code": 1200,
        "message": "OK"
    }
}

How to handle duration of files and events

How to handle the fact that event listing and file listings involve content that has an intrinsic time range. Regular HAPI data content has each row associated with a point in time, at least with respect to the query for data.

We decided to keep the query mechanism and rules the same, and will just add a statement about the need to expand a query time range to include potential edge cases, something like this: "Because event lists and file listings refer to items with an implied durations, a HAPI query for items in this kind of list may need to be expanded, since the query will return only items whose start time falls in the query range. If a server wants to communicate a duration for a file or event (in a Resource Listing or, eventually, an Event List), the stopDate should be used."

How to handle duplicate times in file listings or event lists

Repeated time tags are allowed in fileListing or eventList data schemas. Equivalently, we could say that data must never be decreasing.

We just noticed that the HAPI spec never actually states that HAPI times must only ever increase. So we need to add that to the spec! The definitions for "monotonically increasing: vary, so we will avoid that language. The spec should say that values can only ever increase, with no duplicates.

Comments on case and capitalization

Three places where we have specific capitalization:

  • http query parameters: we use snake case, such as include_parameters
  • camelCase everywhere else
  • AlertCamelCase for the name of the first column, the Time parameter (sort of, since it's only one word)

Defining the schema for what the parameters are

Like the unitsSchema and coordinateSystemSchema, we will use parameterSchema as the keyword in the HAPI info response.

Other options considered:

  • datasetSchema - this means keywords outside the parameters have extra requirements; could datsetSchema be an array?
  • dataType - Should it just be called dataType? Do we need to worry about other usage of schema? We have stringType already.

We settled on datasetSchema, since this ties the schema to the whole dataset, not just the parameters and declaring a specific datasetSchema may actually then levy requirements on other items in the info response at the dataset level. For example, if we add a schema for ground-based datasets, we could require the location or geoLocation.

List of potential schemas:

  • resourceListing
  • availability (or good time intervals)
  • FAIR
  • eventList
  • groundMagnetometer (and other possible measurement types)
  • spaceMagnetometer
  • energeticParticles
  • radioSpectrum
  • plasmaParameters
  • streamFlow
  • earthquakeSeismograph
  • salinityTrace

What about multiple schemas in one info response? We should design the current mechanism to allow for this in the future, but not include it now.

What about the ability to map non-standard names of parameters into the standard ones needed to fulfill a specific datasetSchema? After discussion, we realized that HAPI server creators control the dataset level keyword names, so if a server wants to be compliant with a certain datasetSchema, that server should do the mapping of the names internally, and not express that mapping in HAPI info metadata. This saves HAPI from having to include syntax for renaming things, which if the server knows about that, it could just do it anyway. Although, having the name mapping block in HAPI would allow existing dataset to be merely augmented in a backward compatible way to become compliant with a new datasetSchema, and old code could still refer to the legacy names of the parameters and new code could leverage the name mapping to grab the standardized (via the datasetSchema) version of the data. This may still be worth considering.

The name mapping element could look like some of these options. Note that this is a mishmash info response with multiple illustrations of possibilities for the mapping (including no mapping), whereas a real info response would just have one mapping (or none). And it has comments (using #) which are illegal in real JSON.

{
   "HAPI": "3.3",
   "status": { "code": 1200, "message": "OK"},
   "startDate": "1998-001Z",
   "stopDate" : "2017-100Z",
   "coordinateSystemSchema" : { "schemaName": "SPASE", "schemaURI": "TBD"},
   "datasetSchema": "resourceListing", # no name mapping needed!
   "datasetSchema": { "resourceListing": {} },  # allowed?, but same as above

   "datasetSchema": { "resourceListing": {}, "FAIR": {} },  # can have more than one schema that is applicable; order not important

   "datasetSchema": { "resourceListing": { "schemaVersion": "1.0" }, "FAIR": { "schemaVersion":2.0} },  # can have more than one schema that is applicable; order not important

   # to be consistent with coordinateSystemSchema, each schema should be an object
   # any hierarchy could be expressed within the JSON schema itself
   "datasetSchema": [ { "schemaName": "resourceListing", "schemaURI": "<location of JSON for this schema"  },
                      { "schemaName": "FAIR", "schemaURI": "<uriLocation>" } 
                    ]

   # This allows for expansion if we ever want to define mappings:
   "datasetSchema": [ { "schemaName": "x_resourceListing", "schemaURI": "<location of JSON for this schema",
                              "x_paramNameMappings" : { "startDate" : "some_crazy_time_label", "uri": "epilo_filename" }
                    ]


   "datasetSchema": [ "fileListing": {}, "FAIR": {} ],  # allowed?, but same as above
   "datasetSchema": { "fileListing": { "parameterMap" : { "startDate" : "Time", "uri": "fileURL" } } },
   # or for data:
   "datasetSchema": { "groundMagnetometer": { "vectorField" : "field" } },
   "datasetSchema": { "groundMagnetometer": { "vectorField" : [ "bx", "by", "bz"] } },
   "datasetSchema": { "groundMagnetometer": { "vectorFieldBaselineSubtracted" : "fieldBGSubtr" } },
   "datasetSchema": { "groundMagnetometer": { "vectorFieldBaselineSubtracted" : [ "bx_b", "by_b", "bz_b"] } },
}

This contains a lot of ideas.

Here is what we settled on for how to structure the datasetSchema:

   # to be consistent with coordinateSystemSchema, each schema should be an object
   # any hierarchy could be expressed within the JSON schema itself
   "datasetSchema": [ { "schemaName": "resourceListing", "schemaURI": "<location of JSON for this schema"  },
                      { "schemaName": "FAIR", "schemaURI": "<uriLocation>" } 
                    ]

   # This allows for expansion if we ever want to define mappings:
   "datasetSchema": [ { "schemaName": "x_resourceListing", "schemaURI": "<location of JSON for this schema",
                              "x_paramNameMappings" : { "startDate" : "some_crazy_time_label", "uri": "epilo_filename" }
                    ]

We still need to make sure that a JSON schema can express the constraints we are placing on the info response.

The schema must inherit from the info response schema. Create a JSON schema based on the table above. The JSON schema is to be used that can validate a HAPI info response that has additional constraints beyond what is stated in the HAPI spec

  • this schema should describe an info response that contains FAIR data, as defined by the section in the HAPI spec on read HAPI spec and follow steps in section on how to make data FAIR
  • use the table of required and optional features 8 create a datset schema table

Clone this wiki locally