Centralize schema-safe writable ROOT I/O (preserve StreamerInfo on ROOT ops)

CONTRIBUTING.md

@@ -22,6 +22,17 @@ It is recommended that new contributions and functionalities added to REST have
 
 TODO : Explain doxygen formatting, tutorials, where official doc is located. ETC.
 
+### ROOT file I/O changes
+
+Code that creates, updates, replaces, or merges ROOT files must follow the
+[safe writable ROOT I/O guide](doc/developer/Safe%20writable%20ROOT%20IO.md). It documents the required
+schema preflight, ownership, transactional replacement, and local/remote path rules.
+
+This also applies to **existing library code and user-written macros**, not only new code. The handle does not
+intercept direct `TFile` calls: audit and migrate remaining UPDATE opens, including constructors and `ReOpen`.
+The [user macro migration guide](doc/tutorials/Updating%20ROOT%20files%20from%20macros.md) provides a before/after
+example. A passing framework test suite does not establish that external macros have been migrated.
+
 ### Pipeline validation tests
 
 TODO : Explain how pipeline validation tests should be implemented

README.md

@@ -8,6 +8,19 @@
 The REST-for-Physics (Rare Event Searches Toolkit) Framework is mainly written in C++ and it is fully integrated with [ROOT](https://root.cern.ch) I/O interface.
 REST was initially born as a collaborative software effort to provide common tools for acquisition, simulation, and data analysis of gaseous Time Projection Chambers (TPCs). However, the framework is already extending its usage to be non-exclusive of detector data analysis. The possibilities of the framework are provided by the different libraries and packages written for REST in our community.
 
+## Important: review existing macros that update ROOT files
+
+If your macro adds histograms, changes metadata, or otherwise modifies an existing ROOT file, replace direct
+`TFile` UPDATE opens with `TRestRootFileHandle`. This applies to **existing user macros and library code**, not
+just new development. Direct ROOT calls are not automatically protected by REST and can still lose historical
+schema information, even when the macro only writes a histogram. Read-only macros do not need this migration.
+
+See [Updating ROOT files from macros](doc/tutorials/Updating%20ROOT%20files%20from%20macros.md) for a migration
+example, ownership rules, and what to do if an update is refused. The new interface addresses an existing risk;
+it does not make previously written macros newly unsafe.
+
+## Framework overview
+
 The REST Framework provides 3 interfaces that prototype the use of **event types**, **metadata** and **event processes** through `TRestEvent`, `TRestMetadata` and `TRestEventProcess` abstract class definitions.
 Any REST library will implement **specific objects** that inherit from those 3 basic interfaces.
 
@@ -63,8 +76,10 @@ Any **metadata** object written with REST **will be stamped** with few metadata
 If different REST versions were used to write a ROOT file, e.g. at different steps of the data processing chain, the historic metadata objects will preserve their original version.
 However, the `TRestRun` metadata object **will always store** the version used to write the ROOT file.
 
-After REST release 2.2.1., REST implements correctly the `ROOT schema evolution`. Therefore, any new REST version should always be backwards compatible.
-I.e. Any file written after v2.2.1 should be readable without problems with any future version.
+REST uses ROOT schema evolution to support reading historical data. Compatibility depends on usable historical
+schema information and compatible class definitions/evolution rules; it is not guaranteed by a file's release
+number alone. Use the checked writable interface described above to preserve existing schema information.
+Files already missing required information may need separate recovery.
 
 A major change at 2.3 will prevent from backwards compatibility, since class names have been reviewed.
 

doc/developer/Safe writable ROOT IO.md

@@ -0,0 +1,152 @@
+# Safe writable ROOT I/O
+
+REST ROOT files may contain several historical `TStreamerInfo` entries and embedded schema rules. Opening such
+a file directly with `TFile::Open(..., "UPDATE")` bypasses REST's schema preflight and can let ROOT rewrite
+schema metadata before REST has established that every historical class layout is usable. Framework code that
+creates or mutates ROOT files must therefore use `TRestRootFileHandle`.
+
+## Existing code and user macros need attention too
+
+**This is not only a convention for new code.** Existing framework/library code and user-written macros that
+open ROOT files directly for UPDATE still bypass the protection. The handle does not intercept `TFile::Open`,
+`TFile` constructors, or `ReOpen`; installing a newer REST version does not redirect those calls automatically.
+Even an update that only adds a histogram can rewrite file-level schema metadata. Whether information is lost
+depends on the file's historical schemas and ROOT's handling of them; this change addresses a pre-existing risk.
+
+Audit existing writable opens and migrate them to the handle, or use the borrowed-file adapter below where
+ownership cannot be changed. The adapter must receive a READ-mode file, not one already opened for UPDATE.
+Read-only opens do not need migration for this schema-preservation protection. See the
+[user-facing migration example](../tutorials/Updating%20ROOT%20files%20from%20macros.md), which also covers error
+handling, borrowed pointers, and existing-file safety. Do not assume all library or external macros are covered
+merely because framework CI passes.
+
+## Opening files
+
+Use `TRestRootFileHandle::Open` for new code and when migrating existing writable opens:
+
+```cpp
+#include "TRestTools.h"
+
+auto file = TRestRootFileHandle::Open(filename, TRestRootFileMode::Update);
+if (!file) {
+    ReportError(file.Error());
+    return false;
+}
+
+file->cd();
+WriteObjects();
+
+if (!file.Close()) {
+    ReportError(file.Error());
+    return false;
+}
+```
+
+The available modes are `Read`, `Recreate`, and `Update`. `Recreate` intentionally replaces an existing file
+and must not be used as a shortcut for `Update`. Like ROOT's UPDATE mode, `Update` creates a missing local
+file; REST uses CREATE for this case so that a concurrently appearing file cannot be overwritten.
+An existing file is initially opened read-only. REST inventories the
+exact class-name, class-version, and checksum tuples stored in the file, collects the embedded schema rules,
+asks ROOT to resolve the on-disk entries into loaded or emulated classes, and only then registers the embedded
+rules. Inventory and resolution share one read of the on-disk schema record. REST checks that each resolved
+user schema still has its original name, version, and checksum; a conflicting cached layout is rejected before
+the file becomes writable. This follows ROOT's own `TFile::ReadStreamerInfo`/`TStreamerInfo::BuildCheck` ownership and resolution
+rules, including unloaded classes and entries such as `ROOT::TIOFeatures`. REST then transitions the same
+`TFile` to update mode and checks that its filesystem identity still matches the identity captured before
+preflight. It then marks the historical class-index entries required for writing.
+
+`PrepareBorrowedUpdate(TFile&, std::string*)` provides the same update preparation when legacy code already
+owns a `TFile`. The supplied file must be valid, open in `READ` mode, and not writable:
+
+```cpp
+std::unique_ptr<TFile> file(TFile::Open(filename.c_str(), "READ"));
+std::string error;
+if (!file || !TRestRootFileHandle::PrepareBorrowedUpdate(*file, &error)) {
+    ReportError(error);
+    return false;
+}
+```
+
+Prefer `TRestRootFileHandle` whenever ownership can be changed. A borrowed file remains the caller's
+responsibility, including checking its close/write status. If preparation fails, propagate the error and do not
+attempt to write through that file.
+
+The preflight preserves semantically required historical user StreamerInfos and schema rules; it does not make
+an incompatible class change or an incorrect schema rule valid. ROOT may normalize or omit generated
+standard-library implementation metadata (for example libstdc++ `__pair_base` descriptors) when writing a
+file. REST accepts that ROOT-defined normalization but still requires exact identities for ordinary user and
+historical class schemas, plus every embedded rule. Class authors must still increment class versions as
+required, write correct evolution rules, and test representative old files both before and after a writable
+open.
+
+## Ownership and error handling
+
+`TRestRootFileHandle` is move-only. Pass it by reference while it remains owned by a component, or transfer it
+with `std::move`. Pointers returned by `Get()` and `operator->` are non-owning and must not be deleted or retained
+beyond the handle's lifetime.
+
+The destructor closes an open file, but cannot report failure. Code that writes must call `Close()` explicitly
+and handle a false result using `Error()`. Close a live destination before move-assigning another handle to it,
+because move assignment cannot return a close error for the previous file.
+
+## Replacing or merging files
+
+Use `TRestTools::MergeRootFilesTransactionally` instead of merging directly into the destination or manually
+renaming a partially written file:
+
+```cpp
+std::string error;
+const std::string existing = outputAlreadyExists ? output : "";
+if (!TRestTools::MergeRootFilesTransactionally(output, newInputs, existing, true, &error)) {
+    ReportError(error);
+    return false;
+}
+```
+
+When non-empty, `existingTarget` is copied byte-for-byte to the same-directory temporary file and opened through
+the checked UPDATE path. Only `newInputs` are passed to ROOT's merger. This deliberately preserves ROOT's
+historical UPDATE behavior: target-only objects are not deserialized or rewritten, while a same-named object
+from the new inputs replaces the old target object. Callers updating an existing output must pass it explicitly;
+otherwise its existing contents are not part of the merge.
+
+The helper inventories every input, rejects incompatible classes at the same key path, and constructs the
+result in a temporary sibling of the local destination. Before replacement it validates the expected user
+StreamerInfos and schema rules, recursive key paths and classes, and `TTree` entry counts using ROOT's UPDATE
+semantics (summed across new inputs, replacing a same-named target tree). It validates the installed file again
+and attempts to restore the previous destination from a rollback backup on failure.
+Destination paths and cleanup exclusions resolve filesystem aliases, so using `./` or a symbolic link cannot
+cause input cleanup to delete the completed output. Local input files are removed only after successful replacement and validation when
+`removeInputsOnSuccess` is true.
+
+A false return can also mean that the merged output is valid but a backup or input could not be removed.
+Always inspect the returned error before deciding how to recover. Replacement uses platform filesystem
+operations on sibling paths, but this is not a promise of power-loss durability or atomic behavior on every
+mounted filesystem. Rollback can itself fail; preserve and report the detailed error, including any retained
+backup path.
+
+Internally, the update and merge operations translate failures into exceptions to unwind ROOT file owners
+before cleanup. Their public interfaces retain boolean results and error strings. Both temporary and installed
+merge results pass the same schema/content validator, and any exception during installed-result validation
+triggers rollback before the failure is returned.
+
+## Local and remote paths
+
+Remote ROOT files may be read and may be merge inputs if the installed ROOT transports can open them. Writable
+opens and transactional merge destinations must resolve to local paths; remote URLs are rejected before
+mutation. Local `file://` URLs are accepted. Use `TRestTools::IsRemoteRootPath` when a caller needs to validate
+or explain this policy before opening a file.
+
+Only local entries in `inputFiles` are candidates for removal after a successful merge. Remote inputs are not
+deleted by the helper.
+
+## `TRestRun` is intentionally non-copyable
+
+`TRestRun` owns live input and output handles and also holds raw aliases to file-owned objects. The previous
+implicit copy would have shallow-copied that state, making ownership and lifetime unsafe. Its copy constructor
+and copy assignment operator are therefore deleted.
+
+APIs should pass runs as `TRestRun&`, `const TRestRun&`, or pointers rather than by value. Use
+`std::unique_ptr<TRestRun>` when ownership of a run object itself must be transferred, or construct a separate
+`TRestRun` from the appropriate filename/configuration when an independent instance is required. `TRestRun`
+does not currently expose move construction or move assignment, so do not rely on `std::move` to transfer the
+object directly.

doc/tutorials/Updating ROOT files from macros.md

@@ -0,0 +1,87 @@
+# Updating ROOT files from macros
+
+**Review your existing macros as well as new ones.** If a macro adds histograms, writes a readout, changes
+metadata, or otherwise modifies an existing ROOT file, use `TRestRootFileHandle` instead of a direct `TFile`
+UPDATE open. This applies to personal macros outside the REST repositories too.
+
+ROOT files can contain historical class descriptions (`StreamerInfo`) and schema-evolution rules needed to
+read their data. A direct UPDATE open can allow ROOT to rewrite this metadata without REST's preservation
+checks, even if your macro only adds a histogram. Not every UPDATE loses information, but successful execution
+alone does not prove historical data remains readable.
+
+This is a pre-existing risk, not a new incompatibility introduced by the handle. Updating REST protects code
+that uses the checked interface; it does **not** automatically redirect direct ROOT calls in your macros.
+Macros that only read files do not need this migration for schema preservation.
+
+## What to look for
+
+Look for `TFile::Open(..., "UPDATE")`, stack/heap `TFile` constructors using UPDATE (including lowercase
+`"update"`), and calls to `ReOpen("UPDATE")`. Check Python/PyROOT macros and helper functions too. For example:
+
+```sh
+rg -n -i 'update|reopen' --glob '*.{C,cxx,cpp,h,py}' path/to/your/macros
+```
+
+This is only a starting point: inspect mode variables and wrapper functions manually. A search or CI check is
+not proof that every writable open is safe, and repository CI cannot inspect private macros on your machine.
+
+## Replace the open, preserve the write, check the close
+
+The old pattern bypasses REST's preflight:
+
+```cpp
+TFile* file = TFile::Open(filename, "UPDATE");
+// Write histograms, metadata, etc.
+file->Close();
+delete file;
+```
+
+Use a REST build that provides `TRestRootFileHandle`, with REST loaded in your macro environment. This complete
+example adds a small metadata note; use the same structure around your own histogram or metadata writes:
+
+```cpp
+#include <TFile.h>
+#include <TNamed.h>
+#include <iostream>
+
+#include "TRestTools.h"
+
+bool AddAnalysisNote(const char* filename) {
+    auto file = TRestRootFileHandle::Open(filename, TRestRootFileMode::Update);
+    if (!file) {
+        std::cerr << file.Error() << '\n';
+        return false;
+    }
+
+    file->cd();
+    TNamed note("analysisNote", "Updated with checked REST ROOT I/O");
+    const bool written = note.Write() > 0;
+    const bool closed = file.Close();
+    if (!written) std::cerr << "Could not write analysisNote\n";
+    if (!closed) std::cerr << file.Error() << '\n';
+    return written && closed;
+}
+```
+
+The handle owns the file and closes it automatically on destruction, but writers must explicitly check
+`Close()` to report errors. Pass `file.Get()` to APIs expecting a `TFile*`; that pointer is borrowed. Do not
+delete it or use it, or file-owned objects, after the handle closes. Do not retain the old `delete file` line.
+
+Like ROOT UPDATE, `Update` creates a missing local file. If your macro requires an existing input, keep that
+existence check. `Recreate` intentionally replaces existing contents: never substitute it for an UPDATE that
+failed. Writable destinations must be local; remote reads remain subject to ROOT's available transports.
+
+## If the update is refused
+
+Report the error and stop; do not fall back to direct ROOT UPDATE or change the mode to RECREATE. Consult the
+error before deciding whether a compatible dictionary, corrected schema rule, or separate legacy recovery is
+needed. The handle preserves usable schema information already present; it does not reconstruct missing
+StreamerInfo or repair incompatible evolution rules.
+
+Keep a backup before modifying valuable data. Checked UPDATE is not a rollback transaction for your entire
+macro: failed writes may leave partial changes, and it does not provide concurrent-writer locking or power-loss
+durability. To work on a remote file, explicitly create a local copy and update that copy.
+
+For existing code that must retain ownership of a READ-mode `TFile`, see `PrepareBorrowedUpdate` in the
+[developer guide](../developer/Safe%20writable%20ROOT%20IO.md). The same guide covers transactional merging and
+the intentionally non-copyable `TRestRun` interface.

macros/REST_AddComponentDataSet.C

@@ -1,5 +1,6 @@
 #include "TRestComponent.h"
 #include "TRestTask.h"
+#include "TRestTools.h"
 
 #ifndef RestTask_AddComponent
 #define RestTask_AddComponent
@@ -29,18 +30,17 @@ Int_t REST_AddComponentDataSet(std::string cfgFile, std::string sectionName,
     TRestComponentDataSet comp(cfgFile.c_str(), sectionName.c_str());
     comp.Initialize();
 
-    TFile* f;
-    if (update)
-        f = TFile::Open(outputFile.c_str(), "UPDATE");
-    else
-        f = TFile::Open(outputFile.c_str(), "RECREATE");
+    auto file = TRestRootFileHandle::Open(outputFile,
+                                          update ? TRestRootFileMode::Update : TRestRootFileMode::Recreate);
+    if (!file) {
+        RESTError << file.Error() << RESTendl;
+        return -1;
+    }
 
     if (componentName == "") componentName = sectionName;
 
     comp.Write(componentName.c_str());
 
-    f->Close();
-
-    return 0;
+    return file.Close() ? 0 : -1;
 }
 #endif

macros/REST_AddComponentFormula.C

@@ -1,5 +1,6 @@
 #include "TRestComponent.h"
 #include "TRestTask.h"
+#include "TRestTools.h"
 
 #ifndef RestTask_AddComponentFormula
 #define RestTask_AddComponentFormula
@@ -29,18 +30,17 @@ Int_t REST_AddComponentFormula(std::string cfgFile, std::string sectionName,
     TRestComponentFormula comp(cfgFile.c_str(), sectionName.c_str());
     comp.Initialize();
 
-    TFile* f;
-    if (update)
-        f = TFile::Open(outputFile.c_str(), "UPDATE");
-    else
-        f = TFile::Open(outputFile.c_str(), "RECREATE");
+    auto file = TRestRootFileHandle::Open(outputFile,
+                                          update ? TRestRootFileMode::Update : TRestRootFileMode::Recreate);
+    if (!file) {
+        RESTError << file.Error() << RESTendl;
+        return -1;
+    }
 
     if (componentName == "") componentName = sectionName;
 
     comp.Write(componentName.c_str());
 
-    f->Close();
-
-    return 0;
+    return file.Close() ? 0 : -1;
 }
 #endif

macros/REST_CreateHisto.C

@@ -5,6 +5,7 @@
 #include <TMath.h>
 #include <TRestRun.h>
 #include <TRestTask.h>
+#include <TRestTools.h>
 #include <TSystem.h>
 
 #ifndef RestTask_CreateHisto
@@ -54,9 +55,13 @@ Int_t REST_CreateHisto(string varName, string rootFileName, TString histoName, i
 
     h->Scale(normFactor);
 
-    TFile* f = new TFile((TString)rootFileName, "update");
+    auto file = TRestRootFileHandle::Open(rootFileName, TRestRootFileMode::Update);
+    if (!file) {
+        RESTLog << file.Error() << RESTendl;
+        return -1;
+    }
     h->Write(histoName);
-    f->Close();
+    if (!file.Close()) return -1;
 
     RESTLog << "Written histogram " << histoName << " into " << rootFileName << RESTendl;