[ritchiecarroll]

Note: these changes are implemented in the GSF-based storage engine and are being copied over to the newer SNAPdb engine used by the Gemstone-based openHistorian. -- that said, would rather this GSF-based implementation review be completed before the Gemstone-based version copy is completed, in case there are changes.

When using non-synchrophasor data sources, e.g., SCADA-type data, these changes add a way to down-sample openHistorian data whose timestamps do not fall neatly on regular intervals. The existing interval-based seek filter is highly effective for synchrophasor measurements as it can jump to a narrow window around each requested time and skip the data between those windows. With irregular SCADA-like timestamps, however, a sample may fall just outside the target window and be omitted, even though it would be a reasonable representative value for that interval.

This change adds a new NearestTimestampSeekFilter that selects the nearest available sample for each requested measurement ID at each target time. For example, a four-hour sampling interval over a year provides roughly 2,190 target times per measurement. The returned records retain their actual timestamps and values; the reader does not interpolate values or move samples onto the target times.

A configurable distance limit controls how far the reader may search. By default, this is half the specified sampling interval. For a four-hour interval, the reader searches up to two hours from the target, within the overall query bounds. If no eligible sample exists, that measurement has no result for that target. This keeps long outages visible instead of filling them with increasingly distant samples. Equally distant candidates favor the earlier sample.

There are two search modes:

• Nearest considers samples before and after the target, with limits (described below), and
• AtOrAfter considers only samples at or after the target, within the configured distance limit. This avoids backward searching, but can select a different sample—or return nothing when the only nearby sample precedes the target (faster reads, forward only, and more memory efficient)

Understanding the backward search requires a little context about how the openHistorian archive storage works. Historian records are ordered primarily by timestamp, with measurement IDs ordered within that timeline, the standard HistorianKey. The tree index lets the reader seek efficiently to a position in time. It does not provide a separate timeline for every measurement ID.

Compressed records also introduce a distinction between "finding a location" and "decoding the records there". The compression algorithm used by the openHistorian advances forward, reconstructing records using state established while decoding previous records. It cannot simply start at an arbitrary compressed record and decode its predecessor.

The new reader handles this at the "leaf" level. A leaf is a small block of records near the bottom of the archive's tree structure. To allow for backwards searches, the reader decodes the relevant leaf in a forward direction, as required by the compression algorithm, into a cache of keys and values; this cache can then be processed in reverse order. If needed, the reader will move to the preceding leaf and repeat the process.

This operation provides backward search capability without changing the archive’s compression format or decompressing the entire query range, but at a cost. The buffer is local to the reader and reused one leaf at a time; it is not a persistent reverse index or a cache of the entire archive. Forward-only queries avoid this work. Nearest queries also skip backward searching when exact matches have already resolved the relevant IDs.

The reader stops searching a direction once the remaining timestamps cannot improve the candidates it has found. That preserves the main performance advantage of interval sampling: jumping between target times instead of reading every intervening record. However, it still ends up being a longer scan process, even when just using the forward-direction. For example, if one of the requested measurement IDs has no samples in the search window, the reader cannot stop early: it must scan the rest of that window to confirm the ID is absent, even if it has already found suitable samples for every other requested ID. To help alleviate this issue and improve search performance, you need to know when an ID exists in an archive file, or better, the available time-ranges for an ID.

The code implements two optional companion indexes to address this problem:

• Exact file-presence index (.pidx). This lists the point IDs present anywhere in a .d2 archive file. It lets a query exclude IDs that cannot be found in that file. This also helps mixed requests: an absent ID no longer forces the reader to continue searching after it has found suitable samples for the IDs that are present.
• Time-bucket Bloom index (.bidx). This optional index divides the timeline into fixed-duration buckets and summarizes which IDs occur in each time-range bucket. It addresses a finer distinction: an ID may exist somewhere in the file but have no data near a particular time-range.

I learned about Bloom filters from Josh Patterson, of machine learning fame. A Bloom filter can answer either "definitely absent" or "possibly present". The reader can safely exclude an ID from an archive's search window when every overlapping bucket rules it out. A possible match still executes an ordinary search. False positives therefore cause extra work, not incorrect results.

This initial implementation uses Bloom filters to eliminate IDs from complete search windows. It does not yet jump over individual empty buckets inside a window that might contain a matching sample.

Both indexes are separate external files that live next to a .d2 archive, commonly referred to as a "sidecar". The rollover process collects the needed index information while already streaming records into the new archive, avoiding a second archive scan. Readers validate each sidecar against the archive and captured snapshot before using it. Missing, stale, corrupt, or unsupported indexes fall back to ordinary searching, slow mode. Existing archives remain readable without either index, and no automatic back-fill operation is currently performed, but a process could be added to build or rebuild these indexes. I'll save this for a future PR.

The following controls distinguish writing indexes from using them:

• The LocalOutputAdapter, both GSF and Gemstone flavors, controls index generation. Exact-ID index, i.e., the .pidx, defaults to enabled -- it's a very small file, write impact is low; Bloom index, i.e., the .bidx, is not enabled by default -- the creation cost is higher and the file size is larger, although bucket duration, target false-positive probability, and the per-archive payload budget are configurable -- but for SCADA sources, the Bloom filter can make a huge difference in read speeds, so for data sources with non-aligned timestamps, the filter is recommended.
• Per-query reader options independently control whether .pidx and .bidx are used during read scans. These bypass flags support troubleshooting and performance comparisons without deleting associated index files.
• The seek filter controls sampling direction and maximum sample distance, because those choices affect which measurements a query returns.

FYI: Existing interval filters and Grafana query defaults currently remain unchanged but will need to be updated to take advantage of this new reader -- this will be a new PR. Applications opt into nearest sampling through the new seek filter. The result is a more suitable down-sampling option for irregular timestamps, with additional indexes that reduce unnecessary searching when measurements are absent from files or time windows, making the nearest seek filter much closer in speed to the existing standard one when both filters are enabled.

[StephenCWills]

Also note the build error from Jenkins. It seems openHistorian.Core.SqlClr is not using C# 10.

       "D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\openHistorian.Core.SqlClr.csproj" (default target) (24) ->
       (CoreCompile target) -> 
         D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\Net\HistorianServerDatabaseConfig.cs(40,1): error CS8400: Feature 'file-scoped namespace' is not available in C# 8.0. Please use language version 10.0 or greater. [D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\openHistorian.Core.SqlClr.csproj]

[ritchiecarroll]

Also note the build error from Jenkins. It seems openHistorian.Core.SqlClr is not using C# 10.

       "D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\openHistorian.Core.SqlClr.csproj" (default target) (24) ->
       (CoreCompile target) -> 
         D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\Net\HistorianServerDatabaseConfig.cs(40,1): error CS8400: Feature 'file-scoped namespace' is not available in C# 8.0. Please use language version 10.0 or greater. [D:\Jenkins\workspace\openHistorian\Build-Check_2\Source\Libraries\openHistorian.Core\openHistorian.Core.SqlClr.csproj]

This should be fixed now...