Centralize schema-safe writable ROOT I/O (preserve StreamerInfo on ROOT ops)

[cmargalejo]

Fixes #568

UPDATE 2026-08-19:

Also updating this PR significantly. The idea was essentially correct, but it didn't go far enough. The issue is that with the PR in the original form there would be many edge cases where we could have lost the StreamerInfo in the future. Either through existing code paths or more importantly due to the way the developers of REST need to use the ROOT I/O in order to not lose that information.

• It centralizes writable ROOT access in a move-only TRestRootFileHandle.
• Before entering UPDATE mode, REST inventories and prepares the file’s historical StreamerInfo and schema rules, including classes whose dictionaries are not currently loaded.
• Existing direct writable TFile call sites are migrated to this checked path.
• ROOT-file merges become transactional: build and validate a temporary sibling, then replace with rollback support.
• Remote ROOT files remain valid inputs, but writable destinations must be local.
• TRestRun becomes non-copyable because it now owns file handles and contains non-owning aliases into their contents.
• Comprehensive tests cover loaded/unloaded dictionaries, automatic vector<float> → vector<double> evolution, explicit renamed-member rules, update failures, merges, rollback, and remote paths.

So #566 repairs the specific damaged legacy detector-signal files; #567 prevents normal REST updates and merges from discarding the historical schema information needed to read old data safely.

Important note for developers: In the future it is mandatory that all ROOT I/O happens through the new safe interop. See the documentation about this here:

https://github.com/rest-for-physics/framework/blob/07c720c6cd89f594c2047a3bbdff609c2e7f3c89/doc/developer/Safe%20writable%20ROOT%20IO.md

In principle this should make our lives easier.

The addition of the new I/O interface and updating of all existing ROOT file I/O code is the reason the PR exploded in size now.

Original PR message

The problem

When a ROOT file is closed, TFile::WriteStreamerInfo() replaces the file's
StreamerInfo record, keeping only the infos of the classes streamed during that
session. TRestRun::MergeToOutputFile merges the threads' output files (which
preserves the infos), but then reopens the merged file in UPDATE mode to write
the metadata: closing that session wipes the StreamerInfos of all event classes,
keeping only the metadata classes.

As a consequence, no restManager output file contains event-class
StreamerInfos, which silently breaks ROOT schema evolution whenever an event
class definition changes. This is what made files written before the
TRestDetectorSignal vector<Float_t> → vector<Double_t> change unreadable
(rest-for-physics/detectorlib#125): without the on-disk layout description,
ROOT misreads the float payload as doubles and allocates GBs of garbage.

(restG4 files are unaffected: they are written in a single session.)

The fix

New TRestTools::PreserveStreamerInfos(TFile*): called right after opening a
file in UPDATE mode, it re-tags every StreamerInfo already stored in the file
(same marking as the deprecated TStreamerInfo::TagFile) so that
TFile::WriteStreamerInfo writes them out again on close. Called at the UPDATE
sessions that write into REST data files:

• TRestRun::MergeToOutputFile (the main restManager output path)
• TRestRun::UpdateOutputFile
• TRestProcessRunner split-file metadata update of the main file
• TRestDataSet::Export

Verification

Merging two files containing TRestDetectorSignalEvent trees via
TRestRun::MergeToOutputFile: without the fix the merged file keeps only
TRestRun/TRestMetadata infos; with it, TRestDetectorSignalEvent,
TRestEvent and TRestDetectorSignal survive the metadata session and the
file reads back correctly.

Note: this protects files written from now on. Existing files have already
lost their StreamerInfos; for the detector signal case the data is recoverable
with the tool in #566.

[cmargalejo]

@AlvaroEzq, @lobis, @hgmaluenda, @juanangp , small ping on this PR. We want to get a new stable REST release ready, and this is one of the PRs still waiting for review. Could you have a look when you get a chance?

[Vindaar]

The addition of the new changes should make it so that REST files should remain stable and readable in the future even under further schema changes. That's a big win for us, because there's nothing worse than breaking backwards compatibility for old data files. Especially given the context of the likely changes still to come for BabyIAXO once we get closer to actual data taking. At some point development will speed up and we are going to be happy this is done now and not later.

[cmargalejo]

Hey @Vindaar, reading doc/developer/Safe writable ROOT IO.md, two small thoughts.

1. Since the doc points new code at the handle, I'm assuming existing code is fine and doesn't need a updating, right?. If it's useful, I'm happy to grep the framework + lib submodules for any leftover "UPDATE" opens, just to confirm nothing dangerous is still there.

2. For discoverability I'd lean on CI rather than the doc: a tiny check that fails on a direct TFile::Open(…, "UPDATE") (pointing to the handle) would catch it automatically, since people writing the occasional process won't read a doc/developer/ page anyway.

What do you think?

[Vindaar]

I've added a few more commits which mostly simplify the PR a little bit. The main points remain unchanged though.

Given the large size of the PR, here is an HTML page to review the code more easily, where we have a clearer overview of which files contain what kind of change, what to focus on when reviewing etc. You might find it useful:

https://pages.vindaar.de/rest-for-physics/pr567_compact_review.html

Hey @Vindaar, reading doc/developer/Safe writable ROOT IO.md, two small thoughts.

1. Since the doc points new code at the handle, I'm assuming existing code is fine and doesn't need a updating, right?. If it's useful, I'm happy to grep the framework + lib submodules for any leftover "UPDATE" opens, just to confirm nothing dangerous is still there.

Yes, there seem to be ~4 existing usages in detectorlib and tracklib that we still have to update.

2. For discoverability I'd lean on CI rather than the doc: a tiny check that fails on a direct TFile::Open(…, "UPDATE") (pointing to the handle) would catch it automatically, since people writing the occasional process won't read a doc/developer/ page anyway.

What do you think?

Yes, this is a good idea in principle. We can have a small script that checks for existing lines that use the old style model and then write a warning or even CI failure if they are detected. The thing is, lines using TFile::Open(..., "UPDATE") are not guaranteed to break the StreamerInfo data, but there is a chance depending on what one does. But obviously things still continue to work, so we don't necessarily force every script to use it immediately. It's just preferable.