Repository navigation
feat(strings): length-delimited blobs; as_numpy returns raw bytes - #125
Merged
Merged
Conversation
LSL's cf_string is a variable-length blob format that is length-delimited on the wire. pylsl now treats it that way instead of as C strings: - Outlets accept str (UTF-8 encoded) or any bytes-like value per channel and always push through lsl_push_*_buf* with explicit lengths, so NUL bytes survive. - Inlets always pull through lsl_pull_*_buf. By default values are decoded to str (NULs preserved). With as_numpy=True, string streams return a dtype=object array of the raw bytes, undecoded; as_numpy now means "no Python-level conversion" for every format. - pull_sample honours the inlet's as_numpy default for all formats. Also fixes a long-standing leak: pull_sample on string streams never freed the strings liblsl malloc'd for each channel. Closes #54
cboulay
force-pushed
the
feat/cf_bytes
branch
from
September 20, 2026 18:15
d2c4906 to
e2370b9
Compare
Contributor
Author
|
Benchmark, string streams, Apple Silicon, liblsl 1.18.0.b4, 1000-sample chunks, µs per call:
The ~20% on pull is the per-value length comparison that makes NUL bytes survive; values without NULs never hit a per-element call. The push delta is the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #54 without inventing a channel format. LSL's
cf_stringis a variable-length blob format ("variable-length ASCII strings or data blobs, such as video frames" per liblsl's header) and is length-delimited on the wire and in XDF. pylsl now treats it that way.Outlet (
cf_string)push_sample/push_chunkacceptstr(UTF-8 encoded for you) or any bytes-like value (bytes,bytearray,memoryview) per channel, mixed freely.lsl_push_sample_buftp/lsl_push_chunk_buftp/lsl_push_chunk_buftnpwith an explicituint32lengths array, so NUL bytes survive. The chunk path reuses the joined-buffer pointer table from perf: numpy-native chunk path with as_numpy opt-in #123; the lengths were already computed there.Inlet (
cf_string)lsl_pull_sample_buf/lsl_pull_chunk_bufand slice each pointer by its length. Decodedstrvalues now preserve embedded NULs instead of being truncated.as_numpy=True(inlet default or perpull_chunkcall) returns the raw, undecoded bytes as adtype=objectarray. This givesas_numpyone meaning across all formats: no Python-level conversion.Falseis Python-native (listof numbers /str),Trueis the payload as numpy (typed array / object array ofbytes). No second flag, no mutually exclusive arguments.pull_samplenow honours the inlet'sas_numpydefault for every format (1-D typed array, or 1-D object array ofbytes). No per-call kwarg added there.Also fixes a pre-existing leak:
pull_sampleon string streams never calledlsl_destroy_stringon the per-channel strings liblsl mallocs, despite the liblsl docs requiring it. Now freed after copying out.README: new "Binary data in string streams" section. Decoding is the receiver's decision because only the receiver knows what the bytes mean.
No new liblsl requirement: the
_buffunctions predate pylsl's 1.16 minimum.as_numpyis unreleased, so redefining it for string streams is not a compatibility break.Benchmark
String paths on the
_buffunctions are within noise of the_strones (see comment below for numbers).Test plan
test/test_string_bytes.py(8 tests) plus an updatedas_numpystring test intest_pull_chunk.py, passing on Linux/Windows/macOS:as_numpy=Trueround-trips embedded NUL, invalid UTF-8, all 256 byte values, empty values asbytesin an object array.strout.bytearray/memoryviewinputs; mixedstr/bytesrows; flat input; per-sample timestamps preserve spacing.as_numpy=Falseoverrides an inlet default ofTrue.UnicodeDecodeError(useas_numpy=Truefor binary).Full suite: 61 passed, 1 expected skip. ruff check/format clean under 0.14 and 0.16.