diff --git a/docs/wiki/api-references/Graph-Utilities-API.md b/docs/wiki/api-references/Graph-Utilities-API.md index d5f381d25..fc1f03886 100644 --- a/docs/wiki/api-references/Graph-Utilities-API.md +++ b/docs/wiki/api-references/Graph-Utilities-API.md @@ -5,6 +5,7 @@ The main utility functions of CSP are documented here. These functions enable us - [Table of Contents](#table-of-contents) - [`csp.run`](#csprun) - [`csp.run_on_thread`](#csprun_on_thread) +- [`csp.is_configured_realtime`](#cspis_configured_realtime) - [`csp.show_graph`](#cspshow_graph) ## `csp.run` @@ -73,6 +74,38 @@ Returns a runner object, which has the following methods: """request engine to stop ( async )""" ``` +## `csp.is_configured_realtime` + +```python +csp.is_configured_realtime() +``` + +Returns the `realtime` flag passed to `csp.run`, `csp.run_on_thread`, or `csp.build_graph` (default `False`). Takes no arguments and raises `RuntimeError` outside a graph run/build context. + +During graph construction, this flag can select historical or live input adapters. It does not report the engine's current phase: a realtime-configured run can replay history first. Inside a node, `csp.in_realtime()` reports that phase. See [Execution Modes](Execution-Modes#replay-before-going-realtime). + +This example uses constants in place of adapters: + +```python +from datetime import datetime, timedelta + +import csp + + +def graph(): + if csp.is_configured_realtime(): + return csp.const("live") + return csp.const("historical") + + +start = datetime(2024, 1, 1) +for realtime, expected in ((False, "historical"), (True, "live")): + result = csp.run(graph, starttime=start, endtime=timedelta(0), realtime=realtime) + assert result[0] == [(start, expected)] +``` + +Both runs end at the historical start time. The second selects `"live"` because its configured mode is realtime, even though no realtime processing occurs. + ## `csp.show_graph` ```python diff --git a/docs/wiki/api-references/csp.Struct-API.md b/docs/wiki/api-references/csp.Struct-API.md index 38769d35c..dbfab7633 100644 --- a/docs/wiki/api-references/csp.Struct-API.md +++ b/docs/wiki/api-references/csp.Struct-API.md @@ -139,13 +139,14 @@ print(f"Using FastList field: value {s.a}, type {type(s.a)}, is Python list: {is - **`update_from(self, rhs)`**: copy only the set fields from the `rhs` into this instance - **`update(self, **kwargs)`**: in this instance, set the provided fields with the provided values - **`fromts(self, trigger=None, /, **kwargs)`**: similar to `collectts` above, `fromts` will create a ticking Struct timeseries from the valid values of all the provided inputs whenever any of them tick. `trigger` is an optional position-only argument which is used as a trigger timeseries for when to convert inputs into a struct tick. By default any input tick will generate a new struct of valid inputs -- **`from_dict(self, dict)`**: convert a regular python dict to an instance of the struct +- **`from_dict(cls, json, use_pydantic=False)`**: convert a dictionary to a struct. With `use_pydantic=True`, validate it using pydantic (see [Pydantic integration](#pydantic-integration)) - **`metadata(self)`**: returns the struct's metadata as a dictionary of key : type pairs - **`to_dict(self, callback=None, preserve_enums=False)`**: convert struct instance to a python dictionary, if callback is not None it is invoked for any values encountered when processing the struct that are not basic Python types, datetime types, tuples, lists, sets, dicts, csp.Structs, or csp.Enums. If preserve_enums=True, then enums are not converted to strings in the output dictionary. - **`postprocess_to_dict(cls, obj)`**: if this method is defined on a `csp.Struct` class, the `to_dict` method will invoke this method on the dict obtained after processing the struct of this class to allow for further customization. - **`to_dict_depr(self)`**: convert struct instance to a python dictionary. \[DEPRECATED\]: This is a python only and slower implementation of to_dict which will be removed. - **`to_json(self, callback=lambda x: x)`**: convert struct instance to a json string, callback is invoked for any values encountered when processing the struct that are not basic Python types, datetime types, tuples, lists, dicts, csp.Structs, or csp.Enums. The callback should convert the unhandled type to a combination of the known types. - **`all_fields_set(self)`**: returns `True` if all the fields on the struct are set. Note that this will not recursively check sub-struct fields +- **`type_adapter(cls)`**: return a cached `pydantic.TypeAdapter` for the struct type ## Note on inheritance @@ -189,3 +190,81 @@ By default, CSP Struct objects allow any field to be unset. This means that acce MyStruct(y="hello") # bad: required fields x and z are not set MyStruct(x=None, z=None) # bad: cannot set field x to None as its not optional ``` + +## Pydantic integration + +`csp.Struct` and `csp.Enum` support [pydantic](https://docs.pydantic.dev/) v2 as `BaseModel` fields or through a `TypeAdapter`. `Struct.type_adapter()` returns a cached adapter; `Struct.from_dict(data, use_pydantic=True)` calls its `validate_python` method. + +```python +from datetime import datetime + +from pydantic import BaseModel + +import csp + + +class Side(csp.Enum): + BUY = 1 + SELL = 2 + + +class Order(csp.Struct): + symbol: str + price: float + qty: int = 0 + side: Side = Side.BUY + tags: list[str] + + +class Fill(BaseModel): + order: Order + time: datetime + + +order = Order.type_adapter().validate_python({"symbol": "AAPL", "price": "101.5", "side": "SELL"}) +assert order.price == 101.5 +assert order.side is Side.SELL +assert not hasattr(order, "tags") +assert Order.type_adapter().dump_json(order) == b'{"symbol":"AAPL","price":101.5,"qty":0,"side":"SELL"}' + +fill = Fill.model_validate({"order": {"symbol": "AAPL", "price": 101.5}, "time": "2024-01-02T10:00:00"}) +assert isinstance(fill.order, Order) +assert fill.time == datetime(2024, 1, 2, 10) +``` + +### Struct validation and serialization + +- Fields use pydantic's validation and coercion rules. Schema validation failures and unknown fields raise `pydantic.ValidationError`; CSP's constructor also checks field types. +- Missing fields retain their defaults or remain unset. With `strict=True`, fields without defaults are required. Existing struct instances are returned without revalidation. +- Fields starting with `_` bypass pydantic validation but still pass through CSP's constructor checks. Pydantic serialization omits them and unset fields. +- Field types without a pydantic schema fall back to less specific validation, with CSP's constructor checks still applied. + +Pydantic serialization differs from `to_dict` and `to_json`: + +- `dump_python` preserves `csp.Enum` members; `to_dict` converts them to names unless `preserve_enums=True`. +- Serializers attached through `typing.Annotated`, such as pydantic's `PlainSerializer`, apply only to pydantic serialization. +- JSON formats can differ: a 90-second `timedelta` is `"+90.000000"` in `to_json` and `"PT1M30S"` in pydantic. + +### Enums + +A `csp.Enum` field accepts an enum member, a member name (`"SELL"`) or a value (`2`). JSON serialization uses the member name; the JSON schema lists member names as a string enum. + +### NumPy arrays + +`csp.typing.Numpy1DArray[T]` and `NumpyNDArray[T]` validate by converting inputs with `numpy.asarray(value, dtype=T)`. `Numpy1DArray` also requires one dimension. Serialization uses the array's `tolist()` method. + +```python +from pydantic import BaseModel + +from csp.typing import Numpy1DArray, NumpyNDArray + + +class Prices(BaseModel): + last: Numpy1DArray[float] + grid: NumpyNDArray[float] + + +prices = Prices(last=[1, 2, 3], grid=[[1, 2], [3, 4]]) +assert prices.last.tolist() == [1.0, 2.0, 3.0] +assert prices.model_dump_json() == '{"last":[1.0,2.0,3.0],"grid":[[1.0,2.0],[3.0,4.0]]}' +``` diff --git a/docs/wiki/concepts/CSP-Graph.md b/docs/wiki/concepts/CSP-Graph.md index 08f89e257..1101b0566 100644 --- a/docs/wiki/concepts/CSP-Graph.md +++ b/docs/wiki/concepts/CSP-Graph.md @@ -3,6 +3,7 @@ - [Table of Contents](#table-of-contents) - [Anatomy of a `csp.graph`](#anatomy-of-a-cspgraph) - [Graph Propagation and Single-dispatch](#graph-propagation-and-single-dispatch) +- [Graph Memoization](#graph-memoization) - [Graph Pruning](#graph-pruning) - [Collecting Graph Outputs](#collecting-graph-outputs) @@ -49,6 +50,56 @@ For example, consider the graph below: Individuals nodes are executed in *rank order* where the rank of a node is defined as the longest path between the node and an input adapter. The "mid" node is at rank 1, while "spread" is at rank 2 and "quote" is rank 3. Therefore, if "bid" ticks on a given engine cycle then "mid" will be executed before "spread" and "quote". Note that the order of node execution *within* a rank is undefined, and users should never rely on the execution order of nodes at the same rank. +## Graph Memoization + +By default, repeated calls to a `@csp.node` or `@csp.graph` with the same inputs and scalar arguments reuse the same node or subgraph during graph construction. Consumers share its state, alarms and buffers. This caches graph objects, not the values computed on each tick. Plain Python functions are not automatically memoized, although the nodes they call may be. + +- `@csp.node(memoize=False)` creates an independent node for each call, for example an independent random source (see [More with CSP](More-with-CSP)). +- `@csp.graph(memoize=False)` reruns the graph function for each call. Nodes and subgraphs called inside it retain their own memoization settings and may still be shared. +- `with csp.memoize(False):` disables memoization for calls within the block, except definitions with `force_memoize=True`. The previous setting is restored on exit. + +In this example, `left` and `right` share a counter, while `independent` has its own state: + +```python +from datetime import datetime, timedelta + +import csp + + +@csp.node +def running_count(x: csp.ts[int]) -> csp.ts[int]: + with csp.state(): + count = 0 + if csp.ticked(x): + count += 1 + return count + + +@csp.graph +def example() -> csp.Outputs(shared=csp.ts[int], independent=csp.ts[int]): + x = csp.const(1) + left = running_count(x) + right = running_count(x) + assert left is right + + with csp.memoize(False): + independent = running_count(x) + assert independent is not left + assert running_count(x) is left + + return csp.output(shared=left, independent=independent) + + +start = datetime(2024, 1, 1) +result = csp.run(example, starttime=start, endtime=timedelta(0)) +assert result["shared"] == [(start, 1)] +assert result["independent"] == [(start, 1)] +``` + +The cache belongs to the wiring context. Separate `csp.run` calls normally build fresh contexts and node state; nested or global contexts can share cached graph objects. Reusing a built graph starts a new engine run, not a continuation of its previous state or history. + +`@csp.csp_memoized` also caches return objects from Python factory functions during graph construction. These objects are shared, not copied, so factories should not depend on changing external state or mutate shared results. Arguments that cannot form a cache key cause a warning and an uncached call; `force_memoize=True` raises instead. + ## Graph Pruning Any node in a graph that is not connected to an output will be pruned from the graph and will not exist during runtime. diff --git a/docs/wiki/concepts/CSP-Node.md b/docs/wiki/concepts/CSP-Node.md index 18292a133..178e749de 100644 --- a/docs/wiki/concepts/CSP-Node.md +++ b/docs/wiki/concepts/CSP-Node.md @@ -2,6 +2,7 @@ - [Table of Contents](#table-of-contents) - [Anatomy of a `csp.node`](#anatomy-of-a-cspnode) +- [Alarms](#alarms) - [Basket inputs](#basket-inputs) - [Node Outputs](#node-outputs) - [Basket Outputs](#basket-outputs) @@ -103,6 +104,42 @@ Valid means that an input has had at least one tick and so it has a "current val 27\) We tick our running "sum" as an output here every second. +## Alarms + +Alarms let a node schedule ticks for timers, delays and timeouts. Each node instance owns its alarms, declared with `csp.alarm(typ)` inside `with csp.alarms():`. An alarm behaves like an input: `csp.ticked`, `csp.valid` and `csp.num_ticks` apply, and its name gives its current value. + +These functions can be called in the node body or a `csp.start()` block: + +- **`csp.schedule_alarm(alarm, time, value)`** schedules a tick and returns its handle. `time` is a `timedelta` from engine time or an absolute `datetime`; scheduling in the past raises `ValueError`. An alarm can have multiple pending ticks. Ticks at the same time run in separate engine cycles, in scheduling order. `timedelta(0)` schedules a tick at the current engine time. +- **`csp.reschedule_alarm(alarm, handle, time)`** moves a pending tick without changing its value. It returns a new handle and invalidates the old one. Rescheduling a fired or cancelled tick raises `ValueError`. +- **`csp.cancel_alarm(alarm, handle)`** cancels a pending tick. Cancelling a fired or cancelled tick does nothing. + +This node reports when `x` has not ticked for `timeout`. Each input tick replaces the pending timeout: + +```python +from datetime import timedelta + +import csp + + +@csp.node +def stale(x: csp.ts[float], timeout: timedelta) -> csp.ts[bool]: + with csp.alarms(): + a_timeout = csp.alarm(bool) + + with csp.state(): + s_handle = None + + if csp.ticked(x): + if s_handle is not None: + csp.cancel_alarm(a_timeout, s_handle) + s_handle = csp.schedule_alarm(a_timeout, timeout, True) + elif csp.ticked(a_timeout): + return True +``` + +With a five-second timeout and input ticks at 1, 2, 3 and 10 seconds, `stale` emits `True` at 8 and 15 seconds. It starts waiting after the first input tick; if input and timeout tick in the same cycle, the input takes precedence. + ## Basket inputs In addition to single time-series inputs, a node can also accept a **basket** of time series as an argument. diff --git a/docs/wiki/concepts/Execution-Modes.md b/docs/wiki/concepts/Execution-Modes.md index 8a95f7d40..3de6eca25 100644 --- a/docs/wiki/concepts/Execution-Modes.md +++ b/docs/wiki/concepts/Execution-Modes.md @@ -13,6 +13,7 @@ Since engines can run in both simulated and realtime mode, users should **always - [Table of Contents](#table-of-contents) - [Simulation Mode](#simulation-mode) - [Realtime Mode](#realtime-mode) + - [Replay Before Going Realtime](#replay-before-going-realtime) - [csp.PushMode](#csppushmode) - [Handling Duplicate Timestamps](#handling-duplicate-timestamps) - [Realtime Group Event Synchronization](#realtime-group-event-synchronization) @@ -38,6 +39,39 @@ All time based inputs such as `csp.timer` and alarms will switch to executing in As always, `csp.now()` should still be used in `csp.node` code, even when running in realtime mode. `csp.now()` will be the time assigned to the current engine cycle. +### Replay Before Going Realtime + +Set `starttime` in the past and `endtime` in the future to replay history and continue in realtime: + +```python +from datetime import datetime, timedelta, timezone + +import csp + + +@csp.node +def show_times(tick: csp.ts[bool]): + if csp.ticked(tick): + print(f"engine={csp.now()} wallclock={datetime.now(timezone.utc)} realtime={csp.in_realtime()}") + + +def replay_to_realtime(): + show_times(csp.timer(timedelta(seconds=1))) + + +now = datetime.now(timezone.utc) +csp.run( + replay_to_realtime, + starttime=now - timedelta(seconds=5), + endtime=now + timedelta(seconds=3), + realtime=True, +) +``` + +The historical ticks print in quick succession; live ticks follow at one-second intervals. `csp.in_realtime()` reports the current phase inside a node. Use [`csp.is_configured_realtime()`](Graph-Utilities-API#cspis_configured_realtime) during graph construction to check the run's configured mode instead. + +Historical adapters must supply data for the replay interval; realtime adapters do not provide history automatically. An absolute future `endtime` allows time for live processing. A `timedelta` end time is measured from `starttime`, so it must cover both replay and live processing. + ## csp.PushMode When consuming data from input adapters there are three choices on how one can consume the data: diff --git a/docs/wiki/references/Glossary.md b/docs/wiki/references/Glossary.md index 6ab953bb1..01840a5fe 100644 --- a/docs/wiki/references/Glossary.md +++ b/docs/wiki/references/Glossary.md @@ -63,7 +63,7 @@ The current time of the engine can be accessed at any time within a `csp.node` b ### Alarm - +A node-owned time series scheduled with `csp.schedule_alarm` for timers, delays or timeouts. See [Alarms](CSP-Node#alarms). ### Adapter