Skip to content

feat(strings): length-delimited blobs; as_numpy returns raw bytes - #125

Merged
cboulay merged 3 commits into
mainfrom
feat/cf_bytes
Sep 21, 2026
Merged

cboulay merged 3 commits into
mainfrom
feat/cf_bytes

Conversation

@cboulay

@cboulay cboulay commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Resolves #54 without inventing a channel format. LSL's cf_string is a variable-length blob format ("variable-length ASCII strings or data blobs, such as video frames" per liblsl's header) and is length-delimited on the wire and in XDF. pylsl now treats it that way.

Outlet (cf_string)

  • push_sample / push_chunk accept str (UTF-8 encoded for you) or any bytes-like value (bytes, bytearray, memoryview) per channel, mixed freely.
  • Always push through lsl_push_sample_buftp / lsl_push_chunk_buftp / lsl_push_chunk_buftnp with an explicit uint32 lengths array, so NUL bytes survive. The chunk path reuses the joined-buffer pointer table from perf: numpy-native chunk path with as_numpy opt-in #123; the lengths were already computed there.

Inlet (cf_string)

  • Always pull through lsl_pull_sample_buf / lsl_pull_chunk_buf and slice each pointer by its length. Decoded str values now preserve embedded NULs instead of being truncated.
  • as_numpy=True (inlet default or per pull_chunk call) returns the raw, undecoded bytes as a dtype=object array. This gives as_numpy one meaning across all formats: no Python-level conversion. False is Python-native (list of numbers / str), True is the payload as numpy (typed array / object array of bytes). No second flag, no mutually exclusive arguments.
  • pull_sample now honours the inlet's as_numpy default for every format (1-D typed array, or 1-D object array of bytes). No per-call kwarg added there.

Also fixes a pre-existing leak: pull_sample on string streams never called lsl_destroy_string on the per-channel strings liblsl mallocs, despite the liblsl docs requiring it. Now freed after copying out.

README: new "Binary data in string streams" section. Decoding is the receiver's decision because only the receiver knows what the bytes mean.

No new liblsl requirement: the _buf functions predate pylsl's 1.16 minimum. as_numpy is unreleased, so redefining it for string streams is not a compatibility break.

Benchmark

String paths on the _buf functions are within noise of the _str ones (see comment below for numbers).

Test plan

test/test_string_bytes.py (8 tests) plus an updated as_numpy string test in test_pull_chunk.py, passing on Linux/Windows/macOS:

  • as_numpy=True round-trips embedded NUL, invalid UTF-8, all 256 byte values, empty values as bytes in an object array.
  • Default decoding preserves NULs; bytes in, str out.
  • bytearray / memoryview inputs; mixed str/bytes rows; flat input; per-sample timestamps preserve spacing.
  • Per-call as_numpy=False overrides an inlet default of True.
  • Invalid UTF-8 with default decoding raises UnicodeDecodeError (use as_numpy=True for binary).
  • Free accounting: every liblsl-allocated string handed to the free routine, no stale pointer re-freed.

Full suite: 61 passed, 1 expected skip. ruff check/format clean under 0.14 and 0.16.

LSL's cf_string is a variable-length blob format that is length-delimited
on the wire. pylsl now treats it that way instead of as C strings:

- Outlets accept str (UTF-8 encoded) or any bytes-like value per channel
  and always push through lsl_push_*_buf* with explicit lengths, so NUL
  bytes survive.
- Inlets always pull through lsl_pull_*_buf. By default values are
  decoded to str (NULs preserved). With as_numpy=True, string streams
  return a dtype=object array of the raw bytes, undecoded; as_numpy now
  means "no Python-level conversion" for every format.
- pull_sample honours the inlet's as_numpy default for all formats.

Also fixes a long-standing leak: pull_sample on string streams never
freed the strings liblsl malloc'd for each channel.

Closes #54
@cboulay cboulay changed the title feat: cf_bytes, a NUL-safe raw-bytes view of string streams feat(strings): length-delimited blobs; as_numpy returns raw bytes Sep 20, 2026
@cboulay

cboulay commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

Benchmark, string streams, Apple Silicon, liblsl 1.18.0.b4, 1000-sample chunks, µs per call:

path main (C-string _str) this PR (length-delimited _buf)
push_sample ch=64 9.1 14.3
push_chunk ch=64 3990 4655
pull_sample ch=64 5.2 6.3
pull_chunk ch=64 3617 4425
pull_chunk ch=8 500 613

The ~20% on pull is the per-value length comparison that makes NUL bytes survive; values without NULs never hit a per-element call. The push delta is the uint32 lengths array plus accepting str or bytes-like per value. Numeric paths are untouched.

@cboulay
cboulay merged commit bfa0f15 into main Sep 21, 2026
19 checks passed
@cboulay
cboulay deleted the feat/cf_bytes branch September 21, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for pushing arbitrary byte sequences in cf_string channels

1 participant