Immediate flush mode by franzpoeschel · Pull Request #1846 · openPMD/openPMD-api

franzpoeschel · 2026-02-02T14:30:23Z

Support immediate (synchronous) flushing via `flush_immediately`

Base branch: adios2-memory-selection · Feature branch: synchronous-flush. Use this for the Diff: Diff until then: https://github.com/franzpoeschel/openPMD-api/compare/adios2-memory-selection...franzpoeschel:openPMD-api:synchronous-flush?expand=1

Motivation

The openPMD-api is a deferred data API: load/store operations are enqueued and only executed at explicit flush points. This is efficient but easy to get wrong (missed flushes, ordering surprises). This PR adds an opt-in immediate flush mode where every load/store call acts as an implicit flush point.

What changes

New option flush_immediately (JSON/TOML key or OPENPMD_FLUSH_IMMEDIATELY=1). Default: false in C++, true in Python (merged in via json::merge()).
New FlushLevel::ImmediateFlush with updated flush_level::* helpers, so an immediate flush can write datasets/attributes/hierarchy like a UserFlush without being a user-visible global flush point; determineUnsetDirty() treats it as a no-op.
Immediate path in push_chunk(): performs a frontend hierarchy flush at ImmediateFlush, enqueues the chunk directly, and flushes the backend queue
all within the store/load call. RecordComponent::flush() fixed so the read path only drains m_chunks at a true global flush point.
LS_API { legacy, chaining }: only the legacy API (and PatchRecordComponent) respects immediate flushing; the chaining API manages its own deferred flush and stays deferred; storeChunkSpan forces flush_immediately=false to keep span buffers valid.
IOHandler refactor: shared state extracted into internal::GlobalParameters (from which AbstractIOHandler derives), plus internal::AbstractIOHandlerInitFrom replacing the (path, access, initialize_from) packs in all handler constructors / createIOHandler(). Deferred init now carries options into the real handler via GlobalParameters. Adds Series::flushImmediately().
Tests & CI: full test suite now runs in both deferred and immediate modes on Linux/macOS; tests updated to branch on the active mode (chunk-queue sizes, buffer use-counts, adios2_bp5_flush size, immediate throw on closed-iteration, skips for tests needing non-collective deferred store).
Docs: backendconfig.rst and workflow.rst document the option and that immediate flushing introduces implicit flush points.

Compatibility

Deferred default for C++ is fully preserved; purely additive.

Testing

Full build clean; Core/Auxiliary/JSON/Serial and Python unit suites pass (Python exercises the immediate default).
Full serial CTest passes with both OPENPMD_FLUSH_IMMEDIATELY=0 and =1 (36/36 each).
Parallel (MPI) suite passes in both modes, except the pre-existing adios2_streaming (SST) abort that also reproduces on the base branch (environment-related, not introduced here).

TODO:

Merge Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection #1620 first
Merge Write scientific defaults upon close (Iteration::close or Series::close) #1834 first
Diff until then: https://github.com/franzpoeschel/openPMD-api/compare/adios2-memory-selection...franzpoeschel:openPMD-api:synchronous-flush?expand=1
Make this the default in Python? - Yes
Documentation
Dedicate a CI run to this mode
Check the interplay with the new chaining syntax

+    auto init_directly = [this, &comm..., &filepath](
                             std::unique_ptr<ParsedInput> parsed_input,
-                             json::TracingJSON tracing_json) {
+                             json::TracingJSON tracing_json,


+        else
+        {
+            auto read_again = E_x_read.loadChunk<int>({0, 0}, {mpi_size, 4});
+            // REQUIRE_THROWS(read.flush());


+                std::move(ls_cfg),
+                /*flush_immediately=*/false);
        }
+        // storeChunk(std::move(data), std::move(o), std::move(e));


 template <typename T>
-void RecordComponent::loadChunk(std::shared_ptr<T> data, Offset o, Extent e)
+void RecordComponent::loadChunk_impl(
+    std::shared_ptr<T> const &data, internal::LoadStoreConfigWithBuffer cfg)


-        std::static_pointer_cast<T>(std::move(ptr)),
-        std::move(offset),
-        std::move(extent));
+    // static_assert(!std::is_same_v<T_with_extent, std::string>, "EVIL");


 #endif
 }
+
+TEST_CASE("future_test", "[auxiliary]")


+inline DynamicMemoryView<T> RecordComponent::storeChunkSpanCreateBuffer_impl(
+    internal::LoadStoreConfig cfg, F &&createBuffer)
+{
+    [[maybe_unused]] auto [o, e, api] = std::move(cfg);


+#endif
+}
+
+TEST_CASE("unsafe_no_automatic_flush_immediate_flush_test", "[core]")


for more information, see https://pre-commit.ci

…tics The chaining API's push_chunk() ignores OPENPMD_FLUSH_IMMEDIATELY for safety. When unsafeNoAutomaticFlush() is used, that guarantee no longer applies, so run legacy flushing semantics instead: unsafeNoAutomaticFlush_impl() now sets the internal api to LS_API::legacy, making push_chunk() honor the immediate-flush setting again. Adds a Core test (unsafe_no_automatic_flush_immediate_flush_test) verifying that chunks are flushed immediately when m_flush_immediately is set and buffered otherwise.

Explain how the synchronous flush option (flush_immediately/ OPENPMD_FLUSH_IMMEDIATELY) interacts with the two load/store APIs: - Legacy API (storeChunk()/loadChunk() and friends) only enqueues operations performed at flush points, so the sync-flush option makes each such call its own flush point. - Chaining API (prepareLoadStore()) flushes automatically upon evaluation of its DeferredComputation object and is unaffected by the sync-flush option, unless unsafeNoAutomaticFlush() is used, which falls back to legacy flushing semantics. - The span-based API always defers; Python defaults to immediate flushing. Documented in Doxygen/comments in the source code, in the .rst docs (workflow.rst and backendconfig.rst) and in the Python bindings docstring.

franzpoeschel added the api: new additions to the API label Feb 2, 2026

franzpoeschel force-pushed the synchronous-flush branch from 53bdf60 to 9c02a26 Compare February 2, 2026 16:24

github-advanced-security AI found potential problems Feb 2, 2026

View reviewed changes

franzpoeschel force-pushed the synchronous-flush branch 2 times, most recently from 3b5c07f to c725c7b Compare February 3, 2026 09:46

franzpoeschel mentioned this pull request Feb 13, 2026

Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection #1620

Open

5 tasks

franzpoeschel force-pushed the synchronous-flush branch from 84d0f30 to 6e9c239 Compare March 20, 2026 12:56

github-advanced-security AI found potential problems Mar 20, 2026

View reviewed changes

franzpoeschel force-pushed the synchronous-flush branch from af38436 to 822c03f Compare March 20, 2026 14:01

franzpoeschel force-pushed the synchronous-flush branch from 33b4186 to 7b99854 Compare April 28, 2026 10:00

github-advanced-security AI found potential problems Apr 28, 2026

View reviewed changes

franzpoeschel force-pushed the synchronous-flush branch from 7b99854 to 48a7af8 Compare June 10, 2026 09:00

franzpoeschel mentioned this pull request Jun 29, 2026

update the 5min tutorial ComputationalRadiationPhysics/picongpu#5715

Merged

franzpoeschel force-pushed the synchronous-flush branch from 2f456f4 to c0475c4 Compare July 10, 2026 14:50

franzpoeschel force-pushed the synchronous-flush branch from d11d683 to 588b813 Compare September 3, 2026 12:29

github-advanced-security AI found potential problems Sep 3, 2026

View reviewed changes

github-advanced-security AI found potential problems Sep 10, 2026

View reviewed changes

github-advanced-security AI found potential problems Sep 14, 2026

View reviewed changes

franzpoeschel force-pushed the synchronous-flush branch 3 times, most recently from 2bca70c to f29b628 Compare September 17, 2026 09:31

franzpoeschel added 9 commits September 17, 2026 13:38

Add memory selection task to backend

b35960a

Add future helper

c00658d

Fixes for UniquePtr.hpp

11cf8e1

Main implementation of LoadStoreChunk.hpp

3454e9c

Adapt RecordComponent implementation based on new API

82c334e

Testing

dfbfcea

Enable support for const unique pointers?

348df67

Use a better trick to cheat clang-tidy

74336f5

clang-tidy fixes

48f12ef

franzpoeschel and others added 29 commits September 17, 2026 13:45

Better initialization of global params

efd7bf8

First attempt at immediate flushing

4d9d51e

Fix coretests

3701cf8

wip: try splitting m_work in two

5f681cf

push_chunk: distinguish immediate from deferred flushes

df68897

Avoid flush_immediately for Span operations

2083858

Fix dirty handling

470c1a7

Bug fix in RecordComponent::flush

f848979

tmp: deactivate some tests

34cef84

Properly distinguish flush modi in tests

f8a456e

CI fixes

458a929

CI fixes

f8c7e0b

[pre-commit.ci] auto fixes from pre-commit.com hooks

ea4de28

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks

3b130bc

for more information, see https://pre-commit.ci

Add immediate flush modes to CI

e188b52

Add documentation

45c52a4

Python: flush immediately by default

24b2690

Also have some explicitly deferred CI runs

d9ff9ef

post-rebase fixes

d077779

Fix dir handling in CI

9b5609e

Distinguish immediate flushing depending on used API

12fc13b

Restore prepareLoadStore() API

da7c2a6

Rename API to LS_API

e8f16e0

subtle fixes

b1893a2

Python: Parallel JSON config reading

695f599

nodiscard

1257a30

Parameterize unsafeNoAutomaticFlush()

ea5fa08

franzpoeschel force-pushed the synchronous-flush branch from 8dd3f15 to ea5fa08 Compare September 17, 2026 11:45

Navigation Menu

Immediate flush mode - #1846

Immediate flush mode#1846
franzpoeschel wants to merge 82 commits into
openPMD:devfrom
franzpoeschel:synchronous-flush

franzpoeschel commented Feb 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Check notice

Uh oh!

Uh oh!

Check notice

Check notice

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Check notice

Check notice

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

franzpoeschel commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Support immediate (synchronous) flushing via flush_immediately

Motivation

What changes

Compatibility

Testing

Uh oh!

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Check notice

Check notice

Uh oh!

Uh oh!

Check notice

Uh oh!

Check notice

Check notice

Uh oh!

Check notice

Check notice

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

franzpoeschel commented Feb 2, 2026 •

edited

Loading

Support immediate (synchronous) flushing via `flush_immediately`