[ebariaux]

Description

ENTSO-E A44 publications could leave stale prices in predictions or produce incomplete forecasts because curve types and period boundaries were not fully respected. This change decodes A01/A03 curves, preserves explicit unavailability, and clears existing future predictions in validated gaps between periods. Both EMS implementations now handle unavailable tariffs without failing optimisation.

• Read curveType, default an omitted value to A01, and reject unsupported types. Require complete A01 positions; expand A03 constant-price blocks through the next point or the exclusive period end.
• Validate each period's own timestamps, resolution, positions, and prices before emitting updates. Malformed price amounts reject the whole period; an absent amount represents unavailability. Valid periods in the same response still update.
• Write timestamped nulls for unavailable prices. Identify gaps from merged coverage across all returned series and periods, and set existing future predictions in those gaps to null. If any series or period is rejected, skip gap clearing for the entire response. Historical predictions and predictions outside the returned coverage envelope remain unchanged.
• Use the core null-safe AssetDatapointAllQuery; OpenRemote 1.33.0 or later is required. The branch retains the existing 1.34.0-SNAPSHOT dependency.
• In EmsOptimisation and EmsOptimisationBeta, exclude moving-average windows containing null tariffs from charge/discharge zone selection while preserving the indices of later valid windows. Import and export forecasts are evaluated independently; other optimisation logic continues when no eligible tariff window exists.
• Restore the two skipped integration tests and verify every predicted timestamp and price.

Breaking behaviour and migration

Forecast consumers must handle unavailable values before deploying this update. For example, a previously stored (12:15, 80) prediction can now become (12:15, null). The entry retains its timestamp; it is not deleted. A later publication can restore a numeric price.

This affects custom rules, other extensions, and external applications as well as EMS. Unboxing or performing arithmetic on every datapoint without checking availability can fail. Treat null as unavailable, not zero, and do not carry a previous price through an explicitly unavailable interval. Dropping null entries must not cause remaining points to be treated as consecutive or shift decisions to different timestamps.

The core all query retains an unavailable entry's x timestamp but omits its JSON y property. LTTB omits unavailable points entirely and does not preserve gap markers. The core JSON-null fix alone does not adapt downstream consumers.

Detailed migration guidance is included in entsoe/RELEASE_NOTES.txt and linked from the ENTSO-E README.

Validation

• Full ENTSO-E suite: 139 passed, 0 failed, 0 skipped, independently under Europe/Brussels and UTC test JVM timezones. Both runs forced test execution without the former temporary UTC init script; worker logs confirmed the requested timezone.
• Full EMS suite: 138 passed, 0 failed, 0 skipped. The new tariff tests reproduced 26 null-pointer failures before the fix and passed afterward.
• Malformed-price regressions first failed for both A01 and A03, then passed after validation was added. Gap coverage includes repeated polling over already-null predictions.
• Changed files formatted with Spotless; git diff --check passes. The full repository test suite and a live ENTSO-E deployment were not exercised.

Scope and known limitation

Consumers outside this repository still require review. Changing resolution inside a covered period can leave old prediction timestamps; this PR handles gaps between periods, not removal of those older timestamps.

[ebariaux]

@igorrutka Might be worth you double check the changes in EMS and maybe you need to adapt some rules in custom projects when this gets used.