Skip to content

sof

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

sof

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
benches
benches
 
 
examples
examples
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
Cargo.toml
Cargo.toml
 
 
README.md
README.md
 
 

README.md

SQL-on-FHIR Implementation

This crate provides a complete implementation of the SQL-on-FHIR specification for Rust, enabling the transformation of FHIR resources into tabular data using declarative ViewDefinitions. It supports all major FHIR versions (R4, R4B, R5, R6) through a version-agnostic abstraction layer.

Overview

The sof crate implements the HL7 FHIR SQL-on-FHIR Implementation Guide, providing:

  • ViewDefinition Processing - Transform FHIR resources into tabular data using declarative configuration
  • Multi-Version Support - Works seamlessly with R4, R4B, R5, and R6 FHIR specifications
  • FHIRPath Integration - Complex data extraction using FHIRPath expressions
  • Multiple Output Formats - CSV, JSON, NDJSON, and Parquet support
  • Command Line Interface - Ready-to-use CLI tool for batch processing
  • Server Implementation - HTTP API for on-demand transformations (planned)

Python Developers

Looking to use SQL-on-FHIR from Python? Check out the pysof package, which provides Python bindings for this crate:

import pysof

# Transform FHIR data to CSV, JSON, NDJSON, or Parquet
result = pysof.run_view_definition(view_definition, bundle, "csv")

Features:

  • High Performance - Rust-powered processing with automatic multithreading (5-7x speedup)
  • Simple API - Easy-to-use functions with native Python types
  • Multiple Formats - Support for CSV, JSON, NDJSON, and Parquet outputs
  • FHIR Versions - Compatible with R4, R4B, R5, and R6 (configurable at build time)
  • PyPI Distribution - Install with pip install pysof

See the pysof README for installation and usage details.

Executables

This crate provides two executable targets:

sof-cli - Command Line Interface

A full-featured command-line (CLI) tool for running ViewDefinition transformations. The CLI tool accepts FHIR ViewDefinition and Bundle resources as input (either from files or stdin) and applies the SQL-on-FHIR transformation to produce structured output in formats like CSV, JSON, or other supported content types.

# Basic CSV output (includes headers by default)
sof-cli --view patient-view.json --bundle patient-data.json --format csv

# CSV output without headers
sof-cli --view patient-view.json --bundle patient-data.json --format csv --no-headers

# JSON output to file
sof-cli -v observation-view.json -b lab-results.json -f json -o output.json

# Read ViewDefinition from stdin, Bundle from file
cat view-definition.json | sof-cli --bundle patient-data.json --format csv

# Read Bundle from stdin, ViewDefinition from file
cat patient-bundle.json | sof-cli --view view-definition.json --format json

# Load data using --source parameter (supports local paths and URLs)
sof-cli -v view-definition.json -s ./data/bundle.json -f csv
sof-cli -v view-definition.json -s /absolute/path/to/bundle.json -f csv
sof-cli -v view-definition.json -s file:///path/to/bundle.json -f csv
sof-cli -v view-definition.json -s https://example.com/fhir/bundle.json -f json
sof-cli -v view-definition.json -s s3://my-bucket/fhir-data/bundle.json -f csv
sof-cli -v view-definition.json -s gs://my-bucket/fhir-data/bundle.json -f json
sof-cli -v view-definition.json -s azure://my-container/fhir-data/bundle.json -f ndjson

# Filter resources modified at or after a specific date
sof-cli -v view-definition.json -b patient-data.json --since 2024-01-01T00:00:00Z -f csv

# Limit results to first 100 rows
sof-cli -v view-definition.json -b patient-data.json --limit 100

# Combine filters: recent resources limited to 50 results
sof-cli -v view-definition.json -b patient-data.json --since 2024-01-01T00:00:00Z --limit 50

# Load NDJSON file (newline-delimited JSON) - automatically detected by .ndjson extension
sof-cli -v view-definition.json -b patient-data.ndjson -f csv
sof-cli -v view-definition.json -s file:///path/to/data.ndjson -f json
sof-cli -v view-definition.json -s s3://my-bucket/fhir-data/patients.ndjson -f csv

# NDJSON content detection (works even without .ndjson extension)
sof-cli -v view-definition.json -b patient-data.txt -f csv  # Auto-detects NDJSON content

# Streaming mode for large NDJSON files (memory-efficient chunked processing)
sof-cli -v view-definition.json -b large-patients.ndjson -f csv --chunk-size 500
sof-cli -v view-definition.json -b data.ndjson -f ndjson --skip-invalid

CLI Features

  • Flexible Input:
    • Read ViewDefinitions from file (-v) or stdin
    • Read Bundles from file (-b), stdin, or external sources (-s)
    • Use -s/--source to load from local paths (relative or absolute) or URLs: file://, http(s)://, s3://, gs://, azure://
    • Cannot read both ViewDefinition and Bundle from stdin simultaneously
  • Output Formats: CSV (with/without headers), JSON (pretty-printed array), NDJSON (newline-delimited), Parquet (columnar binary format)
  • Output Options: Write to stdout (default) or specified file with -o
  • Result Filtering:
    • Filter resources by modification time with --since (RFC3339 format). The boundary is inclusive: a resource is kept when its meta.lastUpdated is at or after the given instant, so one stamped exactly at the instant is kept (earlier versions excluded it). Resources without meta.lastUpdated are dropped. sof-server's _since uses the same boundary. --since is currently not applied in NDJSON streaming mode (below) or when --max-file-size splits Parquet output.
    • Limit number of results with --limit (1-10000)
  • Streaming Mode: Memory-efficient chunked processing for large NDJSON files
    • Automatically enabled when using --bundle with .ndjson files
    • Configurable chunk size with --chunk-size (default: 1000 resources)
    • Skip invalid lines with --skip-invalid for fault-tolerant processing
  • FHIR Version Support: R4 by default; other versions (R4B, R5, R6) require compilation with feature flags
  • Error Handling: Clear, actionable error messages for debugging

Command Line Options

-v, --view <VIEW>              Path to ViewDefinition JSON file (or use stdin if not provided)
-b, --bundle <BUNDLE>          Path to FHIR Bundle JSON file (or use stdin if not provided)
-s, --source <SOURCE>          Path or URL to FHIR data source (see Data Sources below)
-f, --format <FORMAT>          Output format (csv, json, ndjson, parquet) [default: csv]
    --no-headers               Exclude CSV headers (only for CSV format)
-o, --output <OUTPUT>          Output file path (defaults to stdout)
    --since <SINCE>            Filter resources modified at or after this time (RFC3339 format)
    --limit <LIMIT>            Limit the number of results (1-10000)
    --fhir-version <VERSION>   FHIR version to use [default: R4]
    --parquet-row-group-size <MB> Encoded MB at which a Parquet row group is cut (64-1024MB) [default: 256]
    --parquet-page-size <KB>   Page size for Parquet (64-8192KB) [default: 1024]
    --parquet-compression <ALG> Compression for Parquet [default: snappy]
                              Options: none, snappy, gzip, lz4, brotli, zstd
    --max-file-size <MB>       Maximum file size for Parquet output (10-10000MB) [default: 1000]
                              When exceeded, the first file keeps the plain name and later files are numbered (output.parquet, output_002.parquet, ...)
    --chunk-size <N>           Number of resources per chunk for streaming NDJSON [default: 1000]
    --skip-invalid             Skip invalid JSON lines in NDJSON files instead of failing
    --terminology-server <URL> Terminology server URL for FHIRPath functions
                               [env: SOF_TERMINOLOGY_SERVER]
-h, --help                     Print help

* Additional FHIR versions (R4B, R5, R6) available when compiled with corresponding features

Data Sources

The CLI provides two ways to specify FHIR data:

  • -b/--bundle: Direct path to a local file (simple, no protocol prefix needed)
  • -s/--source: URL-based loading with protocol support (more flexible)

The --source parameter supports loading FHIR data from various sources:

Local Files
# Using --bundle (simpler for local files)
sof-cli -v view.json -b /path/to/bundle.json

# Using --source with relative path
sof-cli -v view.json -s ./data/bundle.json

# Using --source with absolute path
sof-cli -v view.json -s /path/to/bundle.json

# Using --source with file:// protocol
sof-cli -v view.json -s file:///path/to/bundle.json
HTTP/HTTPS URLs
sof-cli -v view.json -s https://example.com/fhir/bundle.json
AWS S3
# Set AWS credentials
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1

# Load from S3 bucket
sof-cli -v view.json -s s3://my-bucket/fhir-data/bundle.json -f csv
S3-Compatible Services (MinIO, Ceph, LocalStack, etc.)
# Standard AWS credentials
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1

# Custom endpoint (required for S3-compatible services)
export AWS_ENDPOINT=http://localhost:9000

# Allow HTTP endpoints (set to true if not using HTTPS)
export AWS_ALLOW_HTTP=true

sof-cli -v view.json -s s3://my-bucket/fhir-data/bundle.json -f csv
S3-Compatible Services (MinIO, Ceph, LocalStack)
# Standard AWS credentials
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_REGION=us-east-1

# Custom endpoint for S3-compatible service
export AWS_ENDPOINT_URL=http://localhost:9000
export AWS_ALLOW_HTTP=true  # Required for HTTP (non-HTTPS) endpoints

sof-cli -v view.json -s s3://my-bucket/fhir-data/bundle.json -f csv
Google Cloud Storage
# Option 1: Service account credentials
export GOOGLE_SERVICE_ACCOUNT=/path/to/service-account.json

# Option 2: Application Default Credentials
gcloud auth application-default login

# Load from GCS bucket
sof-cli -v view.json -s gs://my-bucket/fhir-data/bundle.json -f json
Azure Blob Storage
# Option 1: Storage account credentials
export AZURE_STORAGE_ACCOUNT=myaccount
export AZURE_STORAGE_ACCESS_KEY=mykey

# Option 2: Azure managed identity (when running in Azure)
# No environment variables needed

# Load from Azure container
sof-cli -v view.json -s azure://my-container/fhir-data/bundle.json -f ndjson

The source can contain:

  • A FHIR Bundle (JSON)
  • A single FHIR resource (will be wrapped in a Bundle)
  • An array of FHIR resources (will be wrapped in a Bundle)
  • NDJSON file (newline-delimited FHIR resources, automatically detected)

NDJSON Input Format

In addition to standard JSON, the CLI and server support NDJSON (newline-delimited JSON) as an input format. NDJSON files contain one FHIR resource per line, making them ideal for streaming large datasets.

Format Detection:

  • Extension-based: Files with .ndjson extension are automatically parsed as NDJSON
  • Content-based fallback: If JSON parsing fails on a multi-line file, NDJSON parsing is attempted automatically
  • Works with all data sources: local files, HTTP(S), S3, GCS, and Azure

Example NDJSON file:

{"resourceType": "Patient", "id": "patient-1", "gender": "male"}
{"resourceType": "Patient", "id": "patient-2", "gender": "female"}
{"resourceType": "Observation", "id": "obs-1", "status": "final", "code": {"text": "Test"}}

Error Handling:

  • Invalid lines are skipped with warnings printed to stderr (or use --skip-invalid in streaming mode)
  • Processing continues as long as at least one valid FHIR resource is found
  • Empty lines and whitespace-only lines are ignored

Streaming Mode (Memory-Efficient Processing):

When processing large NDJSON files with --bundle, the CLI automatically uses streaming mode for memory-efficient processing:

# Stream large NDJSON file with default chunk size (1000 resources)
sof-cli -v view.json -b large-patients.ndjson -f csv

# Custom chunk size for memory-constrained environments
sof-cli -v view.json -b patients.ndjson -f csv --chunk-size 100

# Skip invalid lines and continue processing
sof-cli -v view.json -b patients.ndjson -f ndjson --skip-invalid

# Output to file with streaming
sof-cli -v view.json -b huge-dataset.ndjson -f csv -o output.csv --chunk-size 500

Streaming mode features:

  • Bounded memory: Only --chunk-size resources are loaded at a time (~10MB per 1000 resources)
  • Progressive output: Results are written incrementally, ideal for large datasets
  • Fault tolerance: Use --skip-invalid to continue past malformed JSON lines
  • Statistics: Reports resources processed, chunks, and output rows on completion

Usage Examples:

# Load from local NDJSON file
sof-cli -v view.json -b patients.ndjson -f csv

# Load from cloud storage
sof-cli -v view.json -s s3://bucket/patients.ndjson -f json

# Mix NDJSON source with JSON bundle
sof-cli -v view.json -s file:///data.ndjson -b additional-data.json -f csv

# Server API with NDJSON
curl -X POST "http://localhost:8080/$sql-run?source=s3://bucket/data.ndjson" \
  -H "Content-Type: application/json" \
  -d '{"resourceType": "Parameters", "parameter": [{"name": "subjectResource", "resource": {...}}]}'

Output Formats

The CLI supports multiple output formats via the -f/--format parameter:

  • csv (default) - Comma-separated values format

    • Includes headers by default
    • Use --no-headers flag to exclude column headers
    • All values are properly quoted according to CSV standards

  • json - JSON array format

    • Pretty-printed for readability
    • Each row is a JSON object with column names as keys
    • Suitable for further processing with JSON tools

  • ndjson - Newline-delimited JSON format

    • One JSON object per line
    • Streaming-friendly format
    • Ideal for processing large datasets

  • parquet - Apache Parquet columnar format

    • Efficient binary format for analytics workloads
    • Schema fixed before the first row from the declared column type; untyped columns are inferred from the first 100 rows
    • Configurable compression (snappy, gzip, lz4, brotli, zstd, none)
    • Encoded incrementally with bounded memory, so large datasets do not need to fit in RAM
    • Configurable row group and page sizes for performance tuning

sof-server - HTTP Server

A high-performance HTTP server providing SQL-on-FHIR ViewDefinition transformation capabilities with advanced Parquet support and streaming for large datasets. Use this server if you need a stateless, simple web service for SQL-on-FHIR implementations. Should you need to perform SQL-on-FHIR transformations using server-stored ViewDefinitions and server-stored FHIR data, use the full capabilities of the Helios FHIR Server in hfs.

An open test server is available at https://sof.heliossoftware.com/ for experimentation and evaluation.

# Start server with defaults
sof-server

# Custom configuration via command line
sof-server --port 3000 --host 0.0.0.0 --log-level debug

# Custom configuration via environment variables
SOF_SERVER_PORT=3000 SOF_SERVER_HOST=0.0.0.0 sof-server

# Check server health
curl http://localhost:8080/health

# Get CapabilityStatement
curl http://localhost:8080/metadata

Configuration

The server can be configured using either command-line arguments or environment variables. Command-line arguments take precedence when both are provided.

Request bodies sent with Content-Encoding: gzip (also deflate, br, zstd) are decompressed transparently; unsupported encodings are rejected with 415. Responses are compressed when the client sends Accept-Encoding, except parquet (application/vnd.apache.parquet, legacy application/parquet) and application/zip output, which is already compressed and returned as-is.

Environment Variables
Variable Description Default
SOF_SERVER_PORT Server port 8080
SOF_SERVER_HOST Server host address 127.0.0.1
SOF_LOG_LEVEL Log level (error, warn, info, debug, trace) info
SOF_MAX_BODY_SIZE Maximum request body size in bytes (applies to the decompressed body for compressed requests) 10485760 (10MB)
SOF_REQUEST_TIMEOUT Request timeout in seconds 30
SOF_ENABLE_CORS Enable CORS (true/false) true
SOF_CORS_ORIGINS Allowed CORS origins (comma-separated, * for any) *
SOF_CORS_METHODS Allowed CORS methods (comma-separated, * for any) GET,POST,PUT,DELETE,OPTIONS
SOF_CORS_HEADERS Allowed CORS headers (comma-separated, * for any) Common headers¹
SOF_TERMINOLOGY_SERVER Terminology server URL for FHIRPath functions (memberOf, subsumes) (none)
FHIRPATH_TERMINOLOGY_MAX_CALLS Max distinct terminology server calls per ViewDefinition run (0 disables); read by helios-fhirpath 1000

Terminology lookups (memberOf(), %terminologies.*) share one cache and one call budget per ViewDefinition run. A run is one sof-cli invocation, one streamed NDJSON input, or one view in a $sql-run; each depends-on view a $sql-run materializes is its own run with its own cache and budget. Identical lookups across rows, forEach/repeat items and chunks reach the terminology server once. A lookup that failed for a lasting reason (a 4xx answer such as an unknown ValueSet, or an invalid response body) is answered from the cache for the rest of the run; a timeout, connection error or HTTP 408/429/5xx (except 501) answer is not cached, so the next identical lookup retries it and counts against the cap again. FHIRPATH_TERMINOLOGY_MAX_CALLS (read by helios-fhirpath) caps the distinct lookups of the whole run; going over it fails the run with an error naming the variable, including in forEach/repeat item columns. The default cap of 1000 now applies to the whole run rather than each row, so views that check many distinct codes should raise FHIRPATH_TERMINOLOGY_MAX_CALLS or set it to 0. The cache is dropped when the run ends.

Any terminology error (the call cap, no terminology server configured, or a failed request) fails the run with SofError::FhirPathError wherever the lookup is made: where clauses, top-level columns, and columns evaluated on forEach/forEachOrNull/unionAll/repeat items. Other FHIRPath errors in item columns still fall back to property access and yield null.

Command-Line Arguments
Argument Short Description Default
--port -p Server port 8080
--host -H Server host address 127.0.0.1
--log-level -l Log level info
--max-body-size -m Max request body (bytes) 10485760
--request-timeout -t Request timeout (seconds) 30
--enable-cors -c Enable CORS true
--cors-origins Allowed origins (comma-separated) *
--cors-methods Allowed methods (comma-separated) GET,POST,PUT,DELETE,OPTIONS
--cors-headers Allowed headers (comma-separated) Common headers¹
--terminology-server Terminology server URL (none)
Examples
# Production configuration with environment variables
export SOF_SERVER_HOST=0.0.0.0
export SOF_SERVER_PORT=8080
export SOF_LOG_LEVEL=warn
export SOF_MAX_BODY_SIZE=52428800  # 50MB
export SOF_REQUEST_TIMEOUT=60
export SOF_ENABLE_CORS=false
sof-server

# Development configuration
sof-server --log-level debug --enable-cors true

# CORS configuration for specific frontend
sof-server --cors-origins "http://localhost:3000,http://localhost:3001" \
           --cors-methods "GET,POST,OPTIONS" \
           --cors-headers "Content-Type,Authorization"

# Disable CORS for internal services
sof-server --enable-cors false

# Show all configuration options
sof-server --help
Cloud Storage Configuration

When using the source parameter with cloud storage URLs, ensure the appropriate credentials are configured:

AWS S3 (s3:// URLs):

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1
sof-server

S3-Compatible Services (s3:// URLs with custom endpoint):

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1
export AWS_ENDPOINT=http://localhost:9000  # e.g. MinIO, Ceph, LocalStack
export AWS_ALLOW_HTTP=true                # omit if the endpoint uses HTTPS
sof-server

S3-Compatible Services (MinIO, Ceph, LocalStack):

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_REGION=us-east-1
export AWS_ENDPOINT_URL=http://localhost:9000
export AWS_ALLOW_HTTP=true  # Required for HTTP (non-HTTPS) endpoints
sof-server

Google Cloud Storage (gs:// URLs):

# Option 1: Service account
export GOOGLE_SERVICE_ACCOUNT=/path/to/service-account.json
sof-server

# Option 2: Application Default Credentials
gcloud auth application-default login
sof-server

Azure Blob Storage (azure:// URLs):

# Option 1: Storage account credentials
export AZURE_STORAGE_ACCOUNT=myaccount
export AZURE_STORAGE_ACCESS_KEY=mykey
sof-server

# Option 2: Use managed identity when running in Azure
sof-server
CORS Configuration

The server provides flexible CORS (Cross-Origin Resource Sharing) configuration to control which web applications can access the API:

  • Origins: Specify which domains can access the server

    • Use * to allow any origin (default)
    • Provide comma-separated list for specific origins: https://app1.com,https://app2.com

  • Methods: Control which HTTP methods are allowed

    • Default: GET,POST,PUT,DELETE,OPTIONS
    • Use * to allow any method
    • Provide comma-separated list: GET,POST,OPTIONS

  • Headers: Specify which headers clients can send

    • Default: Common headers¹
    • Use * to allow any header
    • Provide comma-separated list: Content-Type,Authorization,X-Custom-Header

Important Security Notes:

  1. When using wildcard (*) for origins, credentials (cookies, auth headers) are automatically disabled for security
  2. To enable credentials, you must specify exact origins, not wildcards
  3. In production, always specify exact origins rather than using * to prevent unauthorized access
# Development (permissive, no credentials)
sof-server  # Uses default wildcard origin

# Production CORS configuration (with credentials)
export SOF_CORS_ORIGINS="https://app.example.com"
export SOF_CORS_METHODS="GET,POST,OPTIONS"
export SOF_CORS_HEADERS="Content-Type,Authorization"
sof-server

¹ Default headers: Accept,Accept-Language,Content-Type,Content-Language,Authorization,X-Requested-With

Server Features

  • HTTP API: RESTful endpoints for ViewDefinition execution
  • CapabilityStatement: Discovery endpoint for server capabilities
  • ViewDefinition Runner: Synchronous execution of ViewDefinitions
  • Multi-format Output: Support for CSV, JSON, NDJSON, and Parquet responses
  • Advanced Parquet Support:
    • Configurable compression, row group size, and page size
    • Automatic file splitting when size limits are exceeded
    • ZIP archive generation for multi-file outputs
  • Streaming Response: Chunked transfer encoding for large datasets
  • FHIR Compliance: Proper OperationOutcome error responses
  • Configurable CORS: Fine-grained control over cross-origin requests with support for specific origins, methods, and headers

API Endpoints

GET /metadata

Returns the server's CapabilityStatement describing supported operations:

curl http://localhost:8080/metadata
POST /$sql-run

Execute a ViewDefinition transformation:

# JSON output (default)
curl -X POST http://localhost:8080/$sql-run \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "Parameters",
    "parameter": [{
      "name": "subjectResource",
      "resource": {
        "resourceType": "ViewDefinition",
        "status": "active",
        "resource": "Patient",
        "select": [{
          "column": [{
            "name": "id",
            "path": "id"
          }, {
            "name": "gender", 
            "path": "gender"
          }]
        }]
      }
    }, {
      "name": "patient",
      "resource": {
        "resourceType": "Patient",
        "id": "example",
        "gender": "male"
      }
    }]
  }'

# CSV output (includes headers by default)
curl -X POST "http://localhost:8080/$sql-run?_format=text/csv" \
  -H "Content-Type: application/json" \
  -d '{...}'

# CSV output without headers
curl -X POST "http://localhost:8080/$sql-run?_format=text/csv&header=false" \
  -H "Content-Type: application/json" \
  -d '{...}'

# NDJSON output
curl -X POST http://localhost:8080/$sql-run \
  -H "Content-Type: application/json" \
  -H "Accept: application/ndjson" \
  -d '{...}'

Parameters

The $sql-run POST operation accepts parameters either as query parameters or in a FHIR Parameters resource.

Parameter table:

Name Type Use Scope Min Max Documentation
_format code in system 0 1 Output format - application/json, application/ndjson, text/csv, application/vnd.apache.parquet (aliases: application/parquet, application/octet-stream), application/fhir+json (fhir). Defaults to application/x-ndjson when neither _format nor a usable Accept header is supplied.
header boolean in system 0 1 This parameter only applies to text/csv requests. true (default) - return headers in the response, false - do not return headers.
maxFileSize integer in system 0 1 Maximum Parquet file size in MB (10-10000). When exceeded, generates multiple files in a ZIP archive. Checked after each batch, so a file can overshoot by one batch plus the footer.
rowGroupSize integer in system 0 1 Encoded MB at which a Parquet row group is cut (64-1024, default: 256; REST minimum 64). Peak memory per request is about 3x this value.
pageSize integer in system 0 1 Parquet page size in KB (64-8192, default: 1024)
compression code in system 0 1 Parquet compression: none, snappy (default), gzip, lz4, brotli, zstd
subjectCanonical canonical in system 0 1 Canonical URL of the subject. Not supported here: sof-server is stateless and has nothing to resolve it against.
subjectReference Reference in system 0 1 Literal location of the subject. Not supported here, for the same reason.
subjectResource CanonicalResource in system 0 1 The ViewDefinition to execute, supplied inline.
patient Reference in system 0 * Filter resources by patient.
group Reference in system 0 * Filter resources by group.
source string in system 0 1 URL or path to FHIR data source. Supports file://, http(s)://, s3://, gs://, and azure:// protocols.
_limit integer in system 0 1 Limits the number of results. (1-10000)
_since instant in system 0 1 Return resources whose meta.lastUpdated is at or after the supplied time; resources without it are excluded. (RFC3339 format)
resource Resource in system 0 * Collection of FHIR resources to be transformed into a tabular projection.
Query Parameters

All parameters except the subject trio, patient, group, and resource can be provided as POST query parameters:

  • _format: Output format (optional; defaults to application/x-ndjson)
    • application/json - JSON array output
    • text/csv - CSV output
    • application/ndjson - Newline-delimited JSON
    • application/vnd.apache.parquet - Parquet file (aliases: application/parquet, application/octet-stream)
    • fhir / application/fhir+json - FHIR Parameters resource with one row per result row
  • header: Control CSV headers (only applies to CSV format)
    • true - Include headers (default for CSV)
    • false - Exclude headers
  • source: URL to FHIR data (file://, http://, s3://, gs://, azure://)
  • _limit: Limit results (1-10000)
  • _since: Filter by modification time (RFC3339 format)
  • maxFileSize: Maximum Parquet file size in MB (10-10000)
  • rowGroupSize: Parquet row group size in MB (64-1024)
  • pageSize: Parquet page size in KB (64-8192)
  • compression: Parquet compression algorithm
Body Parameters

For POST requests, parameters can be provided in a FHIR Parameters resource:

  • _format: As valueCode or valueString (overrides query params and Accept header)
  • header: As valueBoolean (overrides query params)
  • subjectCanonical / subjectReference: not supported (no resource store)
  • subjectResource: As resource (inline ViewDefinition)
  • patient: As valueReference.reference or valueString
  • group: As valueReference.reference or valueString
  • source: As valueString (URL to external FHIR data)
  • _limit: As valueInteger
  • _since: As valueInstant
  • resource: As resource (can be repeated)
  • maxFileSize: As valueInteger (for Parquet output)
  • rowGroupSize: As valueInteger (for Parquet output)
  • pageSize: As valueInteger (for Parquet output)
  • compression: As valueCode or valueString (for Parquet output)
Parameter Precedence

When the same parameter is specified in multiple places, the precedence order is:

  1. Parameters in request body (highest priority)
  2. Query parameters
  3. Accept header (for format only, lowest priority)
Response Headers

The server automatically sets appropriate response headers based on the output format and size:

Standard Response Headers:

  • Content-Type: Based on format parameter
    • application/json for JSON output
    • text/csv for CSV output
    • application/ndjson for NDJSON output
    • application/vnd.apache.parquet for single Parquet file
    • application/zip for multiple Parquet files
    • application/fhir+json for _format=fhir output and Binary-envelope responses

Streaming Response Headers (for large files):

  • Transfer-Encoding: chunked - Set for streamed Parquet and Arrow IPC responses
  • Content-Disposition: attachment; filename="..." - Suggests filename for downloads
    • Single Parquet: filename="data.parquet"
    • Multiple Parquet (ZIP): filename="data.zip"

Note: The Transfer-Encoding: chunked header is automatically managed by the server. Clients don't need to set any special headers to receive chunked responses - they will automatically receive data in chunks for streamed responses.

Error Responses

Each patient or group body entry must carry a nonblank valueReference.reference or valueString; the reference text is trimmed. Missing or blank strings, identifier-only or display-only references, and unsupported value[X] fields return 400 with an OperationOutcome naming the parameter. These entries are never silently discarded. HFS applies the same rule to its inline, stored-view and Library $sql-run paths and to $sql-export.

HFS's SQLite and PostgreSQL compilers return 422 for a direct ViewDefinition whose computed iteration source cannot be lowered, including nested focuses containing NUL string literals. Non-path forEach sources remain unsupported. The compiler also refuses non-numeric decimal IR literals; FHIRPath decimal syntax is already constrained by the parser, so this is a compiler safeguard, not a newly accepted request form. Dependency compilation failures in SQL Query or SQL View subjects still surface as 500 until #1864.

On HFS, stored-data runners for SQLite, PostgreSQL and S3 use the configured HFS_DEFAULT_FHIR_VERSION. Cardinality and compartment metadata therefore match that version, which can correct earlier results on a server configured for R5, R4B or R6. SQL Query and SQL View dependencies can materialize unionAll ViewDefinitions: repeated branch column names contribute a single table column with the first declaration's position and type, and duplicate rows are retained. The SQL emitters still require matching branch column order.

For HFS stored-data row order, collection cells and preview limits, see the stored-data ordering contract.

An invalid subjectResource (whether supplied as a bare ViewDefinition body or wrapped in a Parameters resource) is rejected with 422 Unprocessable Entity before any rows are evaluated. The ViewDefinition is linted structurally — resource shape, unknown/missing keys, duplicate column names, at most one of forEach/forEachOrNull/repeat, FHIRPath syntax, and undeclared %constant references — exactly as received, before any typed parsing. The response is an OperationOutcome with one issue per lint error:

{
  "resourceType": "OperationOutcome",
  "issue": [
    {
      "severity": "error",
      "code": "required",
      "diagnostics": "missing required key `resource`",
      "details": {
        "text": "missing required key `resource`",
        "coding": [{
          "system": "http://heliossoftware.com/fhir/CodeSystem/view-definition-lint",
          "code": "missing-required"
        }]
      },
      "expression": ["ViewDefinition"]
    }
  ]
}

issue.code follows a fixed mapping from the lint's own diagnostic code: structure for a malformed document, an unknown key, or a wrong JSON type; required for a missing or empty required key; invalid for everything else (duplicate column names, more than one iteration directive, an empty select, invalid FHIRPath syntax, or an undeclared constant). issue.expression[0] locates the offending node as a FHIRPath-style path (ViewDefinition.select[0].column[1].path). This is the same engine used by the ViewDefinition editor's /ui/sql/view-definitions/lint endpoint, and by validate_view_definition — so sof-cli and pysof report the same underlying messages (as plain text, not this OperationOutcome shape).

Examples
# Limit results - first 50 records as CSV
curl -X POST "http://localhost:8080/$sql-run?_limit=50&_format=text/csv" \
  -H "Content-Type: application/json" \
  -d '{...}'

# CSV without headers, limited to 20 results
curl -X POST "http://localhost:8080/$sql-run?_format=text/csv&header=false&_limit=20" \
  -H "Content-Type: application/json" \
  -d '{...}'

# Using header parameter in request body (overrides query params)
curl -X POST "http://localhost:8080/$sql-run?_format=text/csv" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "Parameters",
    "parameter": [{
      "name": "header",
      "valueBoolean": false
    }, {
      "name": "subjectResource",
      "resource": {...}
    }]
  }'

# Filter by modification time (requires resources with lastUpdated metadata)
curl -X POST "http://localhost:8080/$sql-run?_since=2024-01-01T00:00:00Z" \
  -H "Content-Type: application/json" \
  -d '{...}'

# Load data from S3 bucket
curl -X POST "http://localhost:8080/$sql-run?source=s3://my-bucket/bundle.json" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "Parameters",
    "parameter": [{
      "name": "subjectResource",
      "resource": {...}
    }]
  }'

# Load data from Azure with filtering
curl -X POST "http://localhost:8080/$sql-run?source=azure://container/data.json&_limit=100" \
  -H "Content-Type: application/json" \
  -d '{...}'

# Generate Parquet with custom compression and row group size
curl -X POST "http://localhost:8080/$sql-run?_format=application/parquet&compression=zstd&rowGroupSize=512" \
  -H "Content-Type: application/json" \
  -d '{...}' \
  --output result.parquet

# Generate large Parquet with file splitting (returns ZIP if multiple files)
curl -X POST "http://localhost:8080/$sql-run?_format=application/parquet&maxFileSize=100" \
  -H "Content-Type: application/json" \
  -d '{...}' \
  --output result.zip

# Using Parquet parameters in request body
curl -X POST "http://localhost:8080/$sql-run" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceType": "Parameters",
    "parameter": [{
      "name": "_format",
      "valueCode": "parquet"
    }, {
      "name": "maxFileSize",
      "valueInteger": 500
    }, {
      "name": "compression",
      "valueCode": "brotli"
    }, {
      "name": "subjectResource",
      "resource": {...}
    }]
  }' \
  --output result.zip

Core Features

ViewDefinition Processing

Transform FHIR resources using declarative ViewDefinitions:

use helios_sof::{SofViewDefinition, SofBundle, ContentType, run_view_definition};

// Parse ViewDefinition and Bundle
let view_definition: fhir::r4::ViewDefinition = serde_json::from_str(view_json)?;
let bundle: fhir::r4::Bundle = serde_json::from_str(bundle_json)?;

// Wrap in version-agnostic containers
let sof_view = SofViewDefinition::R4(view_definition);
let sof_bundle = SofBundle::R4(bundle);

// Transform to CSV with headers
let csv_output = run_view_definition(
    sof_view,
    sof_bundle,
    ContentType::CsvWithHeader
)?;

Multi-Version FHIR Support

Seamlessly work with any supported FHIR version:

// Version-agnostic processing
match fhir_version {
    FhirVersion::R4 => {
        let view = SofViewDefinition::R4(parse_r4_viewdef(json)?);
        let bundle = SofBundle::R4(parse_r4_bundle(json)?);
        run_view_definition(view, bundle, format)?
    },
    FhirVersion::R5 => {
        let view = SofViewDefinition::R5(parse_r5_viewdef(json)?);
        let bundle = SofBundle::R5(parse_r5_bundle(json)?);
        run_view_definition(view, bundle, format)?
    },
    // ... other versions
}

Advanced ViewDefinition Features

forEach Iteration

Process collections with automatic row generation:

{
  "resourceType": "ViewDefinition",
  "resource": "Patient",
  "select": [{
    "forEach": "name",
    "column": [{
      "name": "family_name",
      "path": "family"
    }, {
      "name": "given_name", 
      "path": "given.first()"
    }]
  }]
}

Constants and Variables

Define reusable values for complex expressions:

{
  "constant": [{
    "name": "loinc_system",
    "valueString": "http://loinc.org"
  }],
  "select": [{
    "where": [{
      "path": "code.coding.where(system = %loinc_system).exists()"
    }],
    "column": [{
      "name": "loinc_code",
      "path": "code.coding.where(system = %loinc_system).code"
    }]
  }]
}

Where Clauses

Filter resources using FHIRPath expressions:

{
  "where": [{
    "path": "status = 'final'"
  }, {
    "path": "effective.exists()"
  }, {
    "path": "value.exists()"
  }]
}

Union Operations

Combine multiple select statements:

{
  "select": [{
    "unionAll": [{
      "column": [{"name": "type", "path": "'observation'"}]
    }, {
      "column": [{"name": "type", "path": "'condition'"}]
    }]
  }]
}

Output Formats

Multiple output formats for different integration needs:

use helios_sof::ContentType;

// CSV without headers
let csv = run_view_definition(view, bundle, ContentType::Csv)?;

// CSV with headers  
let csv_headers = run_view_definition(view, bundle, ContentType::CsvWithHeader)?;

// Pretty-printed JSON array
let json = run_view_definition(view, bundle, ContentType::Json)?;

// Newline-delimited JSON (streaming friendly)
let ndjson = run_view_definition(view, bundle, ContentType::NdJson)?;

// Apache Parquet (columnar binary format)
let parquet = run_view_definition(view, bundle, ContentType::Parquet)?;

Parquet Export

The SOF implementation supports Apache Parquet format for efficient columnar data storage and analytics:

  • Declared-Type Schema: Column types follow the type (and optional ansi/type tag) declared in the ViewDefinition, using the SQL-on-FHIR 3.0.0-ballot default mappings (see Column Type Mapping). Columns that declare no type fall back to inference from the data
  • Type Mapping: see the tables below; collection: true columns become List<element type>
  • In-database runners ($viewdefinition-run on HFS): for Parquet and Arrow IPC output the SQLite and PostgreSQL runners append database values straight to typed Arrow builders (helios_sof::columnar), skipping the JSON row step, and the batches are streamed through the same encoder as the row path. Column types follow the declared mapping above (including the ansi/type tag), so an integer column that is entirely null is Int32 (not Utf8); an integer value outside the 32-bit range of an Int32 column is an error naming the column, not null. Undeclared columns are inferred from the first batch. Runners without a columnar path fall back to the row path.
  • Optimized for Large Datasets:
    • Incremental encoding in batches of 8,192 rows; the schema is fixed before the first row
    • Row groups are cut by encoded size, so memory stays bounded by about one row group (see docs/sof-parquet-streaming-benchmark.md)
    • For a ViewDefinition subject, $sql-run also streams json, csv and fhir once a result has more than 100 rows (SQL Query (Library) subjects still buffer); a row error after the first 100 rows aborts the body, and the request timeout no longer bounds a streamed run after that point
    • The $sql-run HTTP response is streamed as row groups are written; an error after the first bytes aborts the body instead of returning a truncated file with a clean 200
    • Configurable row group size (default: 256MB, range: 64-1024MB), measured in encoded bytes
    • File splitting (--max-file-size) has the granularity of one batch: a file can overshoot the limit by one batch plus the footer
    • Configurable page size (default: 1MB, range: 64KB-8MB)
  • Compression Options:
    • snappy (default): Fast compression with good ratios
    • gzip: Maximum compatibility, good compression
    • lz4: Fastest compression/decompression
    • zstd: Balanced speed and compression ratio
    • brotli: Best compression ratio
    • none: No compression for maximum speed
  • Null Handling: All fields are OPTIONAL to accommodate FHIR's nullable nature
  • Complex Types: Objects and nested structures are serialized as JSON strings

Example usage:

# CLI export with default settings (256MB row groups, snappy compression)
sof-cli --view view.json --bundle data.json --format parquet -o output.parquet

# Optimize for smaller files with better compression
sof-cli --view view.json --bundle data.json --format parquet \
  --parquet-compression zstd \
  --parquet-row-group-size 128 \
  -o output.parquet

# Maximize compression for archival
sof-cli --view view.json --bundle data.json --format parquet \
  --parquet-compression brotli \
  --parquet-row-group-size 512 \
  --parquet-page-size 2048 \
  -o output.parquet

# Fast processing with minimal compression
sof-cli --view view.json --bundle data.json --format parquet \
  --parquet-compression lz4 \
  --parquet-row-group-size 64 \
  -o output.parquet

# Split large datasets into multiple files (500MB each)
sof-cli --view view.json --bundle large-data.json --format parquet \
  --max-file-size 500 \
  -o output.parquet
# Creates: output.parquet (first 500MB)
#          output_002.parquet (next 500MB)
#          output_003.parquet (remaining data)

# Server API - single Parquet file
curl -X POST "http://localhost:8080/$sql-run?_format=application/parquet" \
  -H "Content-Type: application/json" \
  -d '{"resourceType": "Parameters", ...}' \
  --output result.parquet

# Server API - with file splitting (returns ZIP archive if multiple files)
curl -X POST "http://localhost:8080/$sql-run?_format=application/parquet&maxFileSize=100" \
  -H "Content-Type: application/json" \
  -d '{"resourceType": "Parameters", ...}' \
  --output result.zip

# Server API - optimized settings for large datasets
curl -X POST "http://localhost:8080/$sql-run?_format=application/parquet&compression=zstd&rowGroupSize=512&maxFileSize=500" \
  -H "Content-Type: application/json" \
  -d '{"resourceType": "Parameters", ...}' \
  --output result.zip

Performance Guidelines

  • Row Group Size: The value is the number of encoded MB at which a row group is cut, and peak memory is about 3x it (about 190 MB per request at the 64 MB minimum). Larger row groups (256-512MB) improve compression and columnar efficiency at the cost of memory
  • Page Size: Smaller pages (64-512KB) enable fine-grained reads and better predicate pushdown; larger pages (1-8MB) reduce metadata overhead
  • Compression:
    • Use snappy or lz4 for real-time processing
    • Use zstd for balanced storage and query performance
    • Use brotli or gzip for long-term storage where space is critical
  • Large Datasets: Rows are encoded incrementally in fixed 8,192-row batches, and a row group is written each time its encoded size reaches the configured row group size, so memory does not grow with the result size. Arrow IPC output is streamed the same way
  • File Splitting: When --max-file-size or maxFileSize is specified:
    • Files are split when they exceed the specified size in MB
    • Each file contains complete row groups and is independently queryable
    • CLI: Files are named with sequential numbering: base.parquet, base_002.parquet, base_003.parquet, etc.
    • Server: Multiple files are automatically packaged into a ZIP archive for convenient download; sof-server builds a split multi-file ZIP in memory before sending it
    • Ideal for distributed processing systems that parallelize across files
  • Streaming Response (Server only):
    • $sql-run Parquet and Arrow IPC responses are always sent with chunked transfer encoding and no Content-Length; a row error after the first bytes aborts the body
    • Reduces memory usage on both server and client
    • Multiple Parquet files are returned as a ZIP archive with proper content disposition headers
    • Enables processing of gigabyte-scale datasets without memory constraints
    • Response headers for streaming:
      • Transfer-Encoding: chunked - Automatically set by the server for streaming responses
      • Content-Type: application/vnd.apache.parquet or application/zip - Based on single or multi-file output
      • Content-Disposition: attachment; filename="data.parquet" or filename="data.zip" - For convenient file downloads
    • Chunked responses use 64KB chunks for optimal network efficiency

Column Type Mapping

SQL-on-FHIR 3.0.0-ballot defines default mappings from FHIR and FHIRPath types to ISO/IEC 9075 SQL types (viewdef-6, viewdef-7), and an ansi/type column tag to override them. Where a path has both a FHIR type and a FHIRPath type, the FHIR type wins. The resolver lives in column_types (helios_sof::resolve_columns) and is shared by every emitter, so they all agree on what a column's SQL type is; each emitter then maps that SQL type to the closest equivalent in its output format, as the specification asks for implementations without native SQL types.

FHIR type → SQL type (defaults)

FHIR type ISO/IEC 9075 type
boolean BOOLEAN
integer, positiveInt, unsignedInt INT
integer64 BIGINT
decimal CHARACTER VARYING (not a float; the digits are preserved, trailing zeros are not: 1.50 becomes 1.5)
instant TIMESTAMP WITH TIME ZONE
base64Binary BINARY
date, dateTime, time CHARACTER VARYING
every other type (string, code, uri, ...) CHARACTER VARYING

FHIRPath types (columns without a declared type): Boolean → BOOLEAN, Integer → INT, String/Decimal/Date/Time/DateTime → CHARACTER VARYING.

Comparison with each emitter

"Status" is conforms, fixed (diverged before this change, now follows the table) or deliberate (diverges on purpose, reason given).

Emitter decimal instant integer64 base64Binary integer* / boolean Status
Parquet / Arrow IPC (sof-cli, sof-server, pysof, $sql-run, $sql-export) Utf8 (was Float64) Timestamp(µs, UTC), offsets normalised (was Utf8) Int64 (was truncated to Int32) Binary (was Utf8) Int32 with a checked conversion / Boolean fixed
JSON / NDJSON / CSV JSON number, exact decimal digits, trailing zeros stripped string as evaluated JSON number (was the Rust debug string Integer64(5, None, None)) string number / boolean integer64 fixed; decimal deliberate
$sqlquery-run / $sql-run in-memory SQLite (sqlquery) REAL TEXT INTEGER (64-bit) TEXT INTEGER decimal and base64Binary deliberate
In-DB compilers (SQLite, PostgreSQL) PostgreSQL numeric; SQLite REAL text integer64 now decoded as a number (was text) text bigint / INTEGER; boolean integer64 fixed; decimal, instant, base64Binary deliberate
In-DB MongoDB compiler declared types are not applied (BSON is dynamically typed) same same same same deliberate (limitation); an invalid ansi/type is still rejected
_format=fhir (value[x]) valueDecimal valueInstant valueInteger64 valueBase64Binary valueInteger / valueBoolean conforms (the FHIR type picks the value[x] part)

Reasons for the deliberate choices:

  • decimal as a JSON number (JSON, NDJSON, CSV). The upstream conformance suite's expect rows compare decimals as JSON numbers. The digits are exact (no f64 round trip), but trailing zeros are stripped (1.50 is emitted as 1.5), because evaluation artefacts such as 3 / 2 or boundary functions would otherwise not compare equal. Parquet and Arrow carry the same digits as a string, which satisfies viewdef-23.
  • decimal as REAL in SQLite. With TEXT affinity WHERE dec > 5 would compare as text and return wrong rows. SQLite has no exact decimal type either. Use ansi/type: CHARACTER VARYING if you want text. PostgreSQL keeps numeric.
  • instant and base64Binary as text in SQLite and the in-DB runners. The values travel as JSON text; only Parquet and Arrow have native timestamp and binary types.
  • Columns without a type are inferred from the values, passed through the FHIRPath table (a number with a fractional part is a Decimal and so Utf8; integers are Int32, widened to Int64 if a value does not fit). An undeclared instant therefore stays Utf8; declare type: instant to get a timestamp.

ansi/type tag

A column may carry a tag with name ansi/type and a value naming the desired ISO/IEC 9075 type:

{ "name": "birth_date", "path": "birthDate", "type": "date",
  "tag": [{ "name": "ansi/type", "value": "DATE" }] }

The value is case-insensitive. Accepted values, the FHIR types each may be applied to, and the resulting Parquet / Arrow type:

ansi/type value (aliases) Applies to FHIR type Parquet / Arrow type
CHARACTER VARYING (VARCHAR, CHAR VARYING) any Utf8
BOOLEAN boolean Boolean
INT (INTEGER) integer, positiveInt, unsignedInt Int32
BIGINT the INT types, integer64 Int64
DOUBLE PRECISION (DOUBLE) decimal, the INT types, integer64 Float64
DATE date Date32
TIME time Time64(µs)
TIMESTAMP (TIMESTAMP WITHOUT TIME ZONE) dateTime, instant Timestamp(µs)
TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ) dateTime, instant Timestamp(µs, UTC)
BINARY (VARBINARY, BINARY VARYING) base64Binary Binary

A column with no declared type accepts any value and is checked per value. The specification leaves an unknown or incompatible value undefined; we reject it with an invalid-ViewDefinition error naming the column, in every emitter, rather than silently producing a column different from the one requested. This includes a tag whose value is not a string (for example 42): the raw JSON is checked before the typed ViewDefinition is built, so the error names the column instead of being a generic deserialization failure. Likewise, a value that does not fit the requested type (a partial date such as 1980 with DATE, invalid base64, an integer outside the INT range) is an error naming the column, never NULL. Only the Parquet / Arrow and sqlquery SQLite emitters change the stored type; JSON, NDJSON and CSV keep their shape.

Breaking change (Parquet / Arrow IPC). decimal columns change from FLOAT64 to UTF8, instant from UTF8 to TIMESTAMP(µs, UTC), integer64 from a truncated INT32 to INT64 and base64Binary from UTF8 to BINARY. Collection columns of non-string types previously declared one type and built another. To keep a numeric decimal column, add "tag": [{ "name": "ansi/type", "value": "DOUBLE PRECISION" }].

Performance

Multi-Threading

The SQL-on-FHIR implementation leverages multi-core processors for optimal performance through parallel resource processing:

  • Automatic Parallelization: FHIR resources are processed in parallel using rayon for both batch and streaming modes
  • Streaming Mode Benefits: Streaming NDJSON processing uses parallel chunk processing, achieving throughput comparable to or better than batch mode while using 35-150x less memory
  • Zero Configuration: Parallel processing is always enabled with intelligent work distribution
  • Thread Pool Control: Optionally control thread count via RAYON_NUM_THREADS environment variable

RAYON_NUM_THREADS Environment Variable

The RAYON_NUM_THREADS environment variable controls the number of threads used for parallel processing:

# Use all available CPU cores (default behavior)
sof-cli --view view.json --bundle large-bundle.json

# Limit to 4 threads for resource-constrained environments
RAYON_NUM_THREADS=4 sof-cli --view view.json --bundle large-bundle.json

# Use single thread (disables parallelization)
RAYON_NUM_THREADS=1 sof-cli --view view.json --bundle data.ndjson

# Server with custom thread pool
RAYON_NUM_THREADS=8 sof-server

# Python (pysof) also respects this variable
RAYON_NUM_THREADS=4 python my_script.py

When to adjust thread count:

  • Reduce threads (RAYON_NUM_THREADS=2-4): On shared systems, containers with CPU limits, or when running multiple instances
  • Increase threads: Rarely needed; rayon auto-detects available cores
  • Single thread (RAYON_NUM_THREADS=1): For debugging, profiling, or deterministic output ordering

Performance Benchmarks

Batch Mode (Bundle processing):

Bundle Size Sequential Time Parallel Time Speedup
10 patients 22.7ms 8.3ms 2.7x
50 patients 113.8ms 16.1ms 7.1x
100 patients 229.4ms 35.7ms 6.4x
500 patients 1109ms 152ms 7.3x

Streaming Mode (NDJSON processing):

Dataset Batch Mode Streaming Mode Memory Reduction
10k Patients (32MB) 2.66s, 1.6GB 0.93s, 45MB 35x less memory, 2.9x faster
93k Encounters (136MB) 3.97s, 3.9GB 2.75s, 25MB 155x less memory, 1.4x faster

The parallel processing ensures:

  • Each FHIR resource is processed independently on available threads
  • Column ordering is maintained consistently across parallel operations
  • Thread-safe evaluation contexts for FHIRPath expressions
  • Efficient load balancing through work-stealing algorithms
  • Both batch and streaming modes benefit from parallelization

Architecture

Version-Agnostic Design

The crate uses trait abstractions to provide uniform processing across FHIR versions:

// Core traits for version independence
pub trait ViewDefinitionTrait {
    fn resource(&self) -> Option<&str>;
    fn select(&self) -> Option<&[Self::Select]>;
    fn where_clauses(&self) -> Option<&[Self::Where]>;
    fn constants(&self) -> Option<&[Self::Constant]>;
}

pub trait BundleTrait {
    type Resource: ResourceTrait;
    fn entries(&self) -> Vec<&Self::Resource>;
}

Processing Pipeline

  1. Input Validation - Verify ViewDefinition structure and FHIR version compatibility
  2. Constant Extraction - Parse constants/variables for use in expressions
  3. Resource Filtering - Apply where clauses to filter input resources
  4. Row Generation - Process select statements with forEach support
  5. Output Formatting - Convert to requested format (CSV, JSON, etc.)

Error Handling

Comprehensive error types for different failure scenarios:

use helios_sof::SofError;

match run_view_definition(view, bundle, format) {
    Ok(output) => println!("Success: {} bytes", output.len()),
    Err(SofError::InvalidViewDefinition(msg)) => {
        eprintln!("ViewDefinition error: {}", msg);
    },
    Err(SofError::FhirPathError(msg)) => {
        eprintln!("FHIRPath evaluation error: {}", msg);
    },
    Err(SofError::UnsupportedContentType(format)) => {
        eprintln!("Unsupported format: {}", format);
    },
    Err(e) => eprintln!("Other error: {}", e),
}

Feature Flags

Enable support for specific FHIR versions:

[dependencies]
sof = { version = "1.0", features = ["R4", "R5"] }

# Or enable all versions
sof = { version = "1.0", features = ["R4", "R4B", "R5", "R6"] }

Available features:

  • R4 - FHIR 4.0.1 support (default)
  • R4B - FHIR 4.3.0 support
  • R5 - FHIR 5.0.0 support
  • R6 - FHIR 6.0.0 support

Integration Examples

Batch Processing Pipeline

use helios_sof::{SofViewDefinition, SofBundle, ContentType, run_view_definition};
use helios_std::fs;

fn process_directory(view_path: &str, data_dir: &str) -> Result<(), Box<dyn std::error::Error>> {
    let view_def = fs::read_to_string(view_path)?;
    let view: fhir::r4::ViewDefinition = serde_json::from_str(&view_def)?;
    let sof_view = SofViewDefinition::R4(view);
    
    for entry in fs::read_dir(data_dir)? {
        let bundle_path = entry?.path();
        let bundle_json = fs::read_to_string(&bundle_path)?;
        let bundle: fhir::r4::Bundle = serde_json::from_str(&bundle_json)?;
        let sof_bundle = SofBundle::R4(bundle);
        
        let output = run_view_definition(
            sof_view.clone(),
            sof_bundle,
            ContentType::CsvWithHeader
        )?;
        
        let output_path = bundle_path.with_extension("csv");
        fs::write(output_path, output)?;
    }
    
    Ok(())
}

Custom Error Handling

use helios_sof::{SofError, run_view_definition};

fn safe_transform(view: SofViewDefinition, bundle: SofBundle) -> Option<Vec<u8>> {
    match run_view_definition(view, bundle, ContentType::Json) {
        Ok(output) => Some(output),
        Err(SofError::InvalidViewDefinition(msg)) => {
            log::error!("ViewDefinition validation failed: {}", msg);
            None
        },
        Err(SofError::FhirPathError(msg)) => {
            log::warn!("FHIRPath evaluation issue: {}", msg);
            None
        },
        Err(e) => {
            log::error!("Unexpected error: {}", e);
            None
        }
    }
}

Testing

The crate includes comprehensive tests covering:

  • ViewDefinition Validation - Structure and logic validation
  • FHIRPath Integration - Expression evaluation and error handling
  • Multi-Version Compatibility - Cross-version processing
  • Output Format Validation - Correct CSV, JSON, and NDJSON generation
  • Edge Cases - Empty results, null values, complex nested structures
  • Query Parameter Validation - Pagination, filtering, and format parameters
  • Error Handling - Proper FHIR OperationOutcome responses for invalid parameters

Run tests with:

# All tests
cargo test

# Specific FHIR version
cargo test --features R5

# Integration tests only
cargo test --test integration