Skip to content

feat(scheduling): priority bands in ThreadPool + SocketReactor, RT priorities on host Task - #735

Open
finger563 wants to merge 12 commits into
mainfrom
feat/priority-scheduling
Open

feat(scheduling): priority bands in ThreadPool + SocketReactor, RT priorities on host Task#735
finger563 wants to merge 12 commits into
mainfrom
feat/priority-scheduling

Conversation

@finger563

@finger563 finger563 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description

Phase 1 of priority-aware scheduling (foundation for prioritized RTPS endpoints, but useful to any espp user of these components). Design agreed up front: bucketed bands, not fine-grained — and everything defaults to today's behavior byte-for-byte when unused.

espp::QosBand (new, thread_pool/qos_band.hpp)

{ Critical=0, High, Normal (default), Low } — four buckets, shared by the pool and the reactor.

espp::ThreadPool — Linux-scheduler-inspired band queues

  • submit(job, band) / try_submit(job, band); per-band FIFO queues, workers drain most-urgent-first; existing overloads forward to Normal.
  • Aging anti-starvation (default 100 ms, 0 = strict): front-of-band promotion to the back of the next band with clock reset — O(bands) per pop. The threshold is an eligibility interval (evaluated when a worker next dequeues), not an at-most wait bound; promoted work beats all later arrivals, so progress toward Critical is guaranteed under sustained load and nothing starves.
  • Worker bands (opt-in, band_worker_counts + band_task_priorities): band-k workers run at band-k OS priority and service bands 0..k (the deepest configured band's workers service every band, so no band is unreachable) — with an otherwise-empty Critical queue, a new Critical job waits at most one in-flight job's remaining duration for a high-OS-priority worker, and that worker preempts lower ones via the OS scheduler (FreeRTOS on ESP; on host, RT scheduling is a further explicit opt-in — see below).
  • Per-band Stats (submitted/executed/aged).

espp::SocketReactor — priority-aware dispatch

  • Every registration (ReceiveConfig, TCP listener/stream, raw fd) takes a QosBand; each select() batch is stable-sorted by band and submitted band-aware, so urgent sockets also win the remaining pool slots under saturation (the kernel-buffer backpressure for flooding sockets is unchanged — it was already the right behavior).
  • Optional ReceiveConfig::dscp applies IP_TOS (e.g. 46/EF) for network-level marking.

espp::Task — host priority application (explicit opt-in)

Config::priority was silently ignored off-ESP. It can now be applied on host, but only when BaseConfig::host_realtime is set (default false — every existing caller keeps today's default host scheduling; ESP semantics unchanged). With the opt-in: Linux maps 1–25 → SCHED_FIFO [min,max] via pthread_setschedparam, applied by the worker thread itself before its first callback (true preemption under PREEMPT_RT; graceful fallback that explicitly resets to SCHED_OTHER without CAP_SYS_NICE/RLIMIT_RTPRIO — start never fails), macOS best-effort SCHED_FIFO, Windows SetThreadPriority. Live set_priority() works on all platforms (stored priority is atomic); new get_configured_priority(). ThreadPool::Config::band_workers_realtime is the pool-level switch that sets it on per-band workers.

Notable pre-existing bug found

Applying real RT priorities on macOS made workers deschedule between a job's completion-signal and the post-job executed_++, deterministically failing three long-standing test assertions. Stats::executed is now counted at job start (documented) — the only observable stats-timing change.

Testing

  • pc/tests/thread_pool.cpp: 59/59, repeated runs — RT-fallback round-trip, strict ordering (Critical overtakes queued Low), default-band FIFO equivalence, aging rescue under a continuous Normal stream, per-band stats, worker-band mixed-load smoke (no job lost), unreachable-band coverage fallback ({1,0,0,0} strict still executes Normal/Low).
  • pc/tests/socket_reactor.cpp (new): 45/45, repeated runs — the flood scenario (Critical socket delivers all sparse messages with an asserted worst_latency < 500ms bound while a Low socket floods), a deterministic queue-jump test (gated single-worker pool: 4 Low handlers queued first, Critical submitted last must run first — fails under band-less FIFO dispatch), and DSCP verification via getsockopt(IP_TOS) including the out-of-range-rejection path.
  • esp32 on-target example tests: the socket example gains a "Reactor priority bands" scenario (Critical + DSCP EF vs Low under flood on real lwIP), and the task example exercises get_configured_priority() round-trips and the live FreeRTOS priority change.
  • Python: full parity — espp.QosBand, band-aware submit/try_submit, ThreadPool.Config/Stats band fields, Task.BaseConfig.host_realtime, Task.get_configured_priority, ReceiveConfig.band/.dscp, banded SocketReactor.add_udp_receiver — all smoke-tested end-to-end; committed stub updated (and qos_band.hpp added to autogenerate_bindings.py).
  • esp32 thread_pool, socket, task, and rtps examples build; cppcheck clean.

Phase 2 (RTPS wiring: per-channel transport bands, endpoint priority + dedicated-port SEDP locators, facade config) follows separately.

🤖 Generated with Claude Code

…iorities on host Task

Phase 1 of priority-aware scheduling across the espp concurrency stack:
bucketed QosBand priority bands (Critical=0 / High / Normal / Low, new
components/thread_pool/include/qos_band.hpp), band-aware job dispatch, and
real cross-platform application of espp::Task priorities on host builds.
All defaults preserve existing behavior exactly: no-band submits go to
QosBand::Normal and behave as the old single FIFO queue, and every existing
API/overload keeps working unchanged.

ThreadPool (band model):
- One FIFO deque per band; workers always drain the most urgent (lowest
  index) non-empty band first. New submit(job, band) / try_submit(job, band)
  overloads; the no-band overloads forward with QosBand::Normal.
- Aging starvation guard (Config::aging_threshold, default 100ms, 0 =
  strict): before popping, a worker promotes any front-of-band entry whose
  wait exceeds the threshold up one band (to the BACK of the next band, aging
  clock restarted). Approximate by design - only band fronts are examined,
  O(bands) per pop - but since bands are FIFO the front is always the
  longest waiter, so the bound holds: at most aging_threshold per band hop
  (<= 3x threshold Low->Critical) plus the destination band's backlog at
  promotion time; promoted entries enter ahead of all later arrivals, so
  progress is guaranteed under any sustained load.
- Worker bands (opt-in, Config::band_worker_counts +
  band_task_priorities{10,7,5,1}): per-band workers where a band-k worker
  services bands 0..k (its own and more urgent) at a descending espp::Task
  priority. Guarantee: every worker drains band 0 first and the band-0
  workers run at the highest OS priority, so a Critical arrival waits at
  most one in-progress job's remaining duration before a high-OS-priority
  worker takes it (true preemption on FreeRTOS / PREEMPT_RT).
- Stats extended with per-band submitted/executed/aged counters;
  max_queue_size bounds the TOTAL across bands (unchanged semantics).
- stats().executed is now counted when a worker BEGINS executing a job, so
  a job's own side effects always observe it. This fixes a pre-existing
  count-after-run race that real-time worker scheduling on macOS exposed
  deterministically (worker descheduled between the job's completion signal
  and the counter increment).

SocketReactor (priority-aware dispatch):
- Every registration carries a QosBand: UdpSocket::ReceiveConfig::band, and
  band parameters (default Normal) on add_tcp_listener/add_tcp_stream/add_fd.
- Ready sockets from one select() round are stable-sorted by band and each
  handler is submitted at its band, so urgent sockets dispatch first and win
  the remaining pool slots under saturation; a failed submit still reverts
  to re-arm + kernel-buffer backpressure exactly as before.
- UdpSocket::ReceiveConfig::dscp (optional): applied as IP_TOS = dscp << 2
  at registration (best-effort, logged on failure) to mark transmitted
  packets (e.g. 46 = EF); network treatment only, not local scheduling.

Task (host priority application; previously ESP-only):
- Linux/macOS: priority 0 -> default scheduling (SCHED_OTHER); priority
  1..25 (FreeRTOS-convention ceiling) maps linearly onto
  [sched_get_priority_min(SCHED_FIFO), sched_get_priority_max(SCHED_FIFO)],
  applied via pthread_setschedparam after thread start and on
  set_priority() of a live task. On EPERM (unprivileged Linux without
  CAP_SYS_NICE / RLIMIT_RTPRIO) it falls back gracefully to default
  scheduling with a one-time process-wide warning - task start NEVER fails
  due to scheduling policy. Under PREEMPT_RT this yields true preemptive
  RT scheduling.
- Windows: best-effort SetThreadPriority mapping (0 -> NORMAL, 1-8 ->
  ABOVE_NORMAL, 9-16 -> HIGHEST, >=17 -> TIME_CRITICAL).
- New Task::get_configured_priority() accessor; BaseConfig::priority and
  set_priority() docs now spell out the per-platform semantics.

Tests / verification:
- pc/tests/thread_pool.cpp: new sections for host Task priority fallback
  (start/set_priority succeed unprivileged, priority round-trips), strict
  band ordering (Critical overtakes queued Low, single gated worker),
  default-band FIFO equivalence, aging rescue of a Low job under a
  continuous Normal stream (bounded, counter-based), per-band stats, and a
  per-band-workers smoke test (mixed 80-job load, nothing lost, latencies
  logged). All 56 checks pass repeatedly.
- pc/tests/socket_reactor.cpp: new priority section - Critical-band socket
  vs flooded Low-band socket on a single-worker pool; all 10 sparse Critical
  messages dispatched with bounded latency (worst ~6ms observed) while the
  flood keeps progressing; dscp registration exercised. 28/28 checks pass.
- Host: lib/build.sh + pc/build.sh clean; python bindings updated for the
  new submit overloads (disambiguated member pointers) and
  python/socket_reactor_test.py passes 20/20.
- ESP: thread_pool, socket, and rtps examples all build for esp32.
- cppcheck clean over thread_pool, socket, and task; doxygen snippet
  markers verified paired, with a new "Priority Bands" example section.

Explicitly deferred to Phase 2: wiring RTPS (and other consumers) onto the
new bands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 21, 2026 19:09
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

✅Static analysis result - no issues found! ✅

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a cross-component, bucketed priority-band scheduling foundation (espp::QosBand) and wires it through ThreadPool (banded queues + aging + optional per-band workers), SocketReactor (band-aware dispatch + optional DSCP marking), and Task (host OS priority application). This extends ESPP’s scheduling capabilities to support prioritized work while aiming to preserve existing behavior when bands aren’t used.

Changes:

  • Added espp::QosBand and implemented priority-band queuing + aging + per-band worker support in espp::ThreadPool, including per-band stats.
  • Updated espp::SocketReactor registrations and dispatch to be band-aware; added optional DSCP (IP_TOS) application for UDP receiver registrations.
  • Implemented host-side thread priority application in espp::Task (Linux/macOS pthread_setschedparam, Windows SetThreadPriority), and added/updated PC tests and an example.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pc/tests/thread_pool.cpp Adds host-priority and band/aging/worker-band behavior tests with bounded waits.
pc/tests/socket_reactor.cpp Adds a flood test validating band-aware dispatch and DSCP exercise.
lib/python_bindings/pybind_espp.cpp Disambiguates ThreadPool::submit/try_submit overloads for pybind.
components/thread_pool/src/thread_pool.cpp Implements per-band queues, aging promotions, per-band workers, and per-band stats accounting.
components/thread_pool/include/thread_pool.hpp Exposes QosBand APIs, aging/per-band worker config, and per-band stats in the public interface/docs.
components/thread_pool/include/thread_pool_format_helpers.hpp Extends formatting to include per-band stats arrays.
components/thread_pool/include/qos_band.hpp Introduces the shared 4-band QoS enum and band count constant.
components/thread_pool/example/main/thread_pool_example.cpp Adds a deterministic example demonstrating Critical overtaking Low and stats usage.
components/task/src/task.cpp Applies host thread priorities on start and during set_priority() (best-effort fallback).
components/task/include/task.hpp Documents cross-platform priority semantics and adds get_configured_priority().
components/socket/src/socket_reactor.cpp Adds per-registration band storage, stable-sorted band dispatch, and DSCP-to-IP_TOS application for UDP receivers.
components/socket/include/udp_socket.hpp Extends UdpSocket::ReceiveConfig with band and optional dscp.
components/socket/include/socket_reactor.hpp Updates Reactor API/docs for band-aware registration/dispatch and DSCP notes.
Suppressed comments (1)

components/thread_pool/include/thread_pool.hpp:145

  • worker_task_config defaults priority=5; on host platforms this now requests SCHED_FIFO for every ThreadPool worker even when priority bands are unused. That can cause unexpected RT scheduling (or an unconditional one-time warning) for existing host users who previously had priorities ignored.
        ///< Base configuration applied to every worker task. (For per-band workers the priority
        ///< field is overridden by band_task_priorities.)
        .name = "thread_pool_worker",
        .stack_size_bytes = 4096,
        .priority = 5,
        .core_id = -1,
    };

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread components/task/src/task.cpp
Comment thread components/thread_pool/include/thread_pool.hpp Outdated
- esp32-timer-cam / xiao-esp32s3-sense examples: add local thread_pool to
  EXTRA_COMPONENT_DIRS/COMPONENTS so the in-tree component (with qos_band.hpp)
  wins over the stale registry copy the component manager was resolving.
- task: guard sched_get_priority_min/max(SCHED_OTHER) failure (-1) before
  computing a priority to pass to pthread_setschedparam.
- thread_pool: new Config::band_workers_realtime (default false) - host
  per-band workers no longer request SCHED_FIFO unless explicitly opted in
  (band ordering stays queue-level); ESP always applies FreeRTOS priorities.
  One-time info log when running without OS priority differentiation.
- pc/tests: cppcheck constParameterReference fix; opt the per-band worker
  test into band_workers_realtime to keep covering the host RT path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Suppressed comments (5)

Previously missed (4) — in code that hasn't changed since the last review.

components/task/src/task.cpp:184

  • The new thread begins executing thread_function() immediately, before the parent calls apply_thread_priority(). A short callback can therefore run or even finish entirely under the old scheduler, and a latency-sensitive callback can perform its first work before the promised priority takes effect. Gate callback execution until the startup priority attempt completes, or apply the priority from inside the new thread before entering the callback loop.
    thread_ = std::thread(&Task::thread_function, this);
#if !defined(ESP_PLATFORM)
    // On ESP the priority was applied via esp_pthread above; on host platforms
    // apply it to the newly-created thread now (best-effort: an unprivileged
    // failure falls back to default scheduling and never fails the start).
    apply_thread_priority(thread_, config_.priority);

pc/tests/socket_reactor.cpp:252

  • These checks only prove that receiver registration succeeded; add_udp_receiver() deliberately treats setsockopt(IP_TOS) failure as non-fatal, so the test still passes if DSCP marking is broken or removed. Read back IP_TOS with getsockopt() and assert the expected DSCP bits for both sockets.

This issue also appears on line 287 of the same file.

      check(low_id != espp::SocketReactor::INVALID_ID, "Low-band receiver registered (with dscp)");
      check(crit_id != espp::SocketReactor::INVALID_ID,
            "Critical-band receiver registered (with dscp)");

components/socket/include/socket_reactor.hpp:175

  • The socket documentation remains stale after adding banded dispatch and DSCP: neither components/socket/README.md nor doc/en/network/socket_reactor.rst explains these options, and the latter still describes only unprioritized pool submission. Update the component and example documentation with the new registration behavior and DSCP semantics.
   * @note The receive_config's band field selects the priority band this
   *       socket's handling is dispatched at, and its dscp field (if set) is
   *       applied to the socket via IP_TOS here (best-effort, marks
   *       transmitted packets - see UdpSocket::ReceiveConfig).

lib/python_bindings/pybind_espp.cpp:2703

  • This resolves overload ambiguity by binding only the legacy one-argument overload, so Python callers cannot submit a QosBand at all. QosBand and the new ThreadPool configuration/stat fields are also not exposed in this module, leaving the advertised priority feature unavailable through the existing Python API. Bind the enum and add the band-aware overload and associated fields rather than discarding the overload here.
      .def("submit",
           static_cast<bool (espp::ThreadPool::*)(espp::ThreadPool::Job &&)>(
               &espp::ThreadPool::submit),
           py::arg("job"),

pc/tests/socket_reactor.cpp:290

  • This assertion does not establish priority-aware reactor dispatch. Because the reactor permits only one in-flight handler per socket, one flooding Low socket can keep at most one 5 ms Low handler ahead of a Critical message; all messages would therefore meet the 2 s per-message timeout even with the new band sorting/submission removed. Add a deterministic case that blocks the worker, makes both sockets readable in the same batch, releases it, and asserts that the Critical callback runs first.
      check(delivered == num_critical_msgs,
            "all Critical messages dispatched promptly during the flood");
      check(flood_processed.load() >= flood_before + 5,
            "Low-band flood kept making progress alongside Critical traffic");

Comment thread components/task/include/task.hpp Outdated
Comment thread components/thread_pool/include/thread_pool.hpp
Comment thread components/task/include/task.hpp Outdated
…ates

- task: new BaseConfig::host_realtime (default false). Host platforms only
  apply priority to the OS thread (SCHED_FIFO on Linux/macOS, SetThreadPriority
  on Windows) when explicitly opted in - existing callers (ThreadPool /
  SocketReactor default priority 5, etc.) keep espp's historical default
  scheduling. ESP always applies FreeRTOS priorities, unchanged.
- thread_pool: band_workers_realtime now simply sets host_realtime on the
  per-band worker tasks (band priorities always stored; queue-level ordering
  always enforced).
- docs: thread_pool README/rst document bands, aging, per-band workers and the
  host-RT opt-in; task README/rst document per-platform priority semantics,
  privilege fallback and the SCHED_FIFO starvation risk; example README gains
  the priority-band test row; Doxyfile indexes qos_band.hpp.
- python: expose Task.BaseConfig.host_realtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

components/socket/src/socket_reactor.cpp:210

  • dscp is documented as 0–63, but the public type accepts 0–255 and this mask silently wraps invalid values (64 becomes DSCP 0, 255 becomes 63). Validate the value before binding/applying the option and report invalid configuration (or explicitly choose and document clamping) rather than assigning a different traffic class.
    const int tos = (receive_config.dscp.value() & 0x3F) << 2;

Comment thread components/thread_pool/src/thread_pool.cpp
Comment thread components/task/src/task.cpp Outdated
Comment thread lib/python_bindings/pybind_espp.cpp
- thread_pool: per-band worker configs can no longer leave bands unreachable -
  the deepest configured band's workers service ALL bands (warn if that band
  is not Low), so e.g. {1,0,0,0} with aging_threshold==0 still executes
  Normal/Low submissions instead of queueing them forever (and a bounded
  blocking submit can no longer deadlock on them). New pc test covers it.
- task: host_realtime scheduling is now applied by the worker thread itself at
  the top of thread_function(), before the first callback invocation - the
  parent-side application raced thread startup, so a short callback could run
  entirely (or exit, ESRCH) at default priority. apply_thread_priority() now
  delegates to a native-handle variant shared with the self-application path;
  live set_priority() unchanged.
- python: bind espp.QosBand, band-aware submit()/try_submit() overloads,
  ThreadPool.Config aging_threshold/band_worker_counts/band_task_priorities/
  band_workers_realtime, and per-band Stats arrays; smoke-tested end-to-end
  (band submit + per-band stats + coverage fallback).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (5) — in code that hasn't changed since the last review.

components/socket/include/socket_reactor.hpp:52

  • The existing user-facing SocketReactor documentation was not updated for this behavior change. components/socket/README.md:63-90 and doc/en/network/socket_reactor.rst:4-34 still describe unbanded dispatch and omit the new per-registration bands, saturation ordering, and UDP DSCP option. Update both documents so users can discover and correctly configure the new API.
 *          Each registration carries a priority band (@ref espp::QosBand,
 *          default Normal). When one @c select() wakeup reports several
 *          sockets readable at once, their handlers are submitted to the pool
 *          in band order (most urgent first) and each is submitted AT its band,
 *          so band-aware pools (see @ref espp::ThreadPool) run urgent sockets'

pc/tests/socket_reactor.cpp:237

  • This scenario does not actually distinguish band-aware dispatch from the old FIFO behavior. One-shot arming permits at most one Low handler from this single socket to be running or queued, so even an unprioritized Critical handler waits at most one 5 ms callback and easily satisfies the 2 s check. Gate the worker and queue work from multiple independent Low registrations (or otherwise build a Low backlog), then assert the Critical callback overtakes that queued work.

This issue also appears on line 238 of the same file.

           .band = espp::QosBand::Low,

components/socket/src/socket_reactor.cpp:210

  • dscp is documented as a 0–63 code point, but values above 63 are silently masked into a different valid value here (for example, 64 becomes DSCP 0). Reject out-of-range input with a clear error instead of applying an unintended marking.
    const int tos = (receive_config.dscp.value() & 0x3F) << 2;

lib/python_bindings/pybind_espp.cpp:2611

  • Binding QosBand alone does not make SocketReactor priority dispatch usable from Python: UdpSocket.ReceiveConfig does not expose its new band/dscp fields, and the hand-written SocketReactor.add_udp_receiver wrapper accepts neither argument. Extend both binding paths so Python users can configure the reactor feature introduced here.
  py::enum_<espp::QosBand>(
      m, "QosBand",
      "*\n * @brief Priority band for queued work. Critical is the most urgent and Low the "
      "least;\n * Normal is the default for band-less submissions.\n")
      .value("Critical", espp::QosBand::Critical)
      .value("High", espp::QosBand::High)
      .value("Normal", espp::QosBand::Normal)
      .value("Low", espp::QosBand::Low);

lib/python_bindings/pybind_espp.cpp:2132

  • host_realtime is exposed, but the new Task::get_configured_priority() API is not bound, so Python cannot perform the documented/requested priority round trip (especially when a best-effort live change returns false). Add a get_configured_priority binding and update the existing set_priority binding text, which still says live changes are ESP-only.
            .def_readwrite("host_realtime", &espp::Task::BaseConfig::host_realtime,
                           "*< Opt-in to applying the priority to the OS thread on host "
                           "platforms (SCHED_FIFO on Linux/macOS; ignored on ESP).");

pc/tests/socket_reactor.cpp:238

  • The DSCP check only verifies that registration succeeds, but setsockopt() is explicitly best-effort and registration also succeeds when it fails, so this test passes even if DSCP is never applied. Read back IP_TOS with getsockopt() and assert the configured DSCP bits to cover the new marking behavior.
           .dscp = 8}); // CS1 "low-priority data"

Comment thread components/thread_pool/include/thread_pool.hpp
Comment thread components/task/src/task.cpp Outdated
…cket+task esp32 examples

- task: new std::atomic<size_t> priority_ is the single source of truth after
  construction - set_priority() wrote config_.priority unlocked while the
  worker thread's startup priority application (and get_configured_priority())
  read it concurrently, an unsynchronized data race. All reads (esp_pthread
  startup config, host self-application, getter) now go through the atomic.
- socket example: new 'Reactor priority bands' scenario - Critical (with DSCP
  EF via IP_TOS on lwIP) + Low banded UDP receivers on one reactor; floods the
  Low port and verifies Critical echoes during the flood and Low after it.
  Referenced as a doc snippet from the SocketReactor class docs.
- task example: priority section now exercises get_configured_priority()
  round-trips (construction / after live set_priority) and the FreeRTOS-side
  observation of the live change, and documents BaseConfig::host_realtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 8 comments.

Comment thread components/task/src/task.cpp
Comment thread components/task/include/task.hpp
Comment thread lib/python_bindings/pybind_espp.cpp Outdated
Comment thread components/socket/include/udp_socket.hpp Outdated
Comment thread components/socket/include/socket_reactor.hpp
Comment thread components/socket/src/socket_reactor.cpp Outdated
Comment thread pc/tests/socket_reactor.cpp
Comment thread components/thread_pool/include/thread_pool.hpp Outdated
…on, real ordering test, full python parity, socket docs)

- task: when applying SCHED_FIFO fails, explicitly reset the thread to
  SCHED_OTHER so a previously-RT thread cannot silently keep its old policy
  (contradicting the documented fallback); a failed reset gets its own warning.
- socket_reactor: reject DSCP values > 63 with a warning instead of silently
  masking them to a different code point.
- pc/tests/socket_reactor: new deterministic queue-jump test - a single-worker
  external pool (aging off, queue_size() observable) is blocked by a gated
  handler, 4 Low handlers queue first, Critical is submitted LAST and must run
  FIRST on release; this fails under band-less FIFO dispatch. The flood test
  now also asserts the worst Critical latency bound (<500ms).
- thread_pool: class-doc contract now states the deepest-band-worker coverage
  exception explicitly.
- python: full parity for the reactor path - ReceiveConfig gains band/dscp
  (ctor kwargs + attributes), SocketReactor.add_udp_receiver takes band/dscp,
  Task.get_configured_priority bound; QosBand enum moved before its first use
  as a default argument; qos_band.hpp added to autogenerate_bindings.py; the
  committed stub (__init__.pyi) updated for QosBand, the band submit overloads,
  Config/Stats band fields, BaseConfig.host_realtime, ReceiveConfig band/dscp,
  and Task.get_configured_priority. All smoke-tested end-to-end from Python.
- docs: socket README + doc/en/network/socket_reactor.rst document bands +
  DSCP (incl. out-of-range rejection); socket example README lists the reactor
  scenarios including the new priority-bands one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

components/task/src/task.cpp:449

  • A live set_priority() can race this startup application and leave the OS thread at the old priority. For example, the worker can load priority 10 here, set_priority(3) can then apply 3 under thread_mutex_, and the worker can resume and apply the stale 10 even though get_configured_priority() reports 3. Serialize the startup load/application with the same mutex used by set_priority() so whichever operation runs last observes/applies the latest value.
  if (config_.host_realtime) {
#if defined(_WIN32)
    apply_thread_priority_to_handle(GetCurrentThread(), priority_.load());
#elif defined(__linux__) || defined(__APPLE__)
    apply_thread_priority_to_handle(pthread_self(), priority_.load());

Comment thread lib/python_bindings/espp/__init__.pyi
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (5)

Previously missed (4) — in code that hasn't changed since the last review.

components/task/src/task.cpp:169

  • Initial ESP priorities still bypass the documented clamp: unlike set_priority(), start() copies an arbitrary configured value directly into esp_pthread_cfg_t. A value at or above configMAX_PRIORITIES can therefore make startup fail instead of being clamped to configMAX_PRIORITIES - 1. Clamp the loaded value here and update priority_ before calling esp_pthread_set_cfg().
  thread_config.prio = priority_.load();

components/task/include/task.hpp:171

  • The PR description says host Config::priority is now applied, but this new default keeps it unapplied unless callers explicitly set host_realtime = true. The opt-in is intentional per the implementation and documentation, so the PR description should also state it; otherwise users are told the default behavior changed when it did not.
    bool host_realtime{
        false}; /**< Opt-in to applying the priority to the OS thread on HOST platforms (ignored
                   on ESP, where the FreeRTOS priority is always applied). When false (the
                   default) the thread uses the OS default scheduling, matching espp's historical

components/socket/src/socket_reactor.cpp:219

  • The new tests pass nonzero DSCP values and check registration/delivery, but never observe IP_TOS; deleting this setsockopt() call would leave all tests passing. Since this is the implementation of the new DSCP API, add a supported-host assertion using getsockopt(IP_TOS) (and ideally cover the out-of-range path) so the marking behavior is actually verified.
      const int tos = dscp << 2;
      if (::setsockopt(fd, IPPROTO_IP, IP_TOS, reinterpret_cast<const char *>(&tos), sizeof(tos)) <
          0) {

components/thread_pool/include/thread_pool.hpp:46

  • This advertised upper bound is not provided by the implementation. Aging is checked only when a worker next dequeues work, so a long in-flight job can make a hop occur arbitrarily later than aging_threshold; destination-band backlog adds further delay. Describe the threshold as an eligibility interval rather than an at-most wait bound.

This issue also appears on line 62 of the same file.

 * front is always the longest-waiting entry of its band. The resulting bound:
 * an entry waits at most aging_threshold per band hop (so at most
 * 3 * aging_threshold to reach Critical from Low) plus the backlog of each
 * destination band at promotion time; since promoted entries enter ahead of
 * all later arrivals, progress is guaranteed under any sustained load. Set

components/thread_pool/include/thread_pool.hpp:67

  • The stated latency guarantee ignores older Critical jobs. With a Critical backlog, a newly arrived Critical job can wait for every earlier Critical job, not just one currently running job. Qualify the claim so consumers do not treat worker bands as a hard one-job queueing bound.
 * because every worker drains band 0 first and the band-0 workers run at the
 * highest OS priority, a newly arrived Critical job waits at most the
 * remaining duration of one already-running job before a high-OS-priority
 * worker picks it up (and on a preemptive OS - e.g. FreeRTOS or Linux
 * PREEMPT_RT with granted RT scheduling - that worker preempts lower-priority
 * ones the moment it becomes runnable).

… stub import)

- stub: import datetime/enum in __init__.pyi (aging_threshold uses
  datetime.timedelta; the enum module was already referenced).
- task: clamp the initial ESP priority in start() to configMAX_PRIORITIES - 1
  exactly like set_priority() - an out-of-range configured value must not make
  startup fail (priority_ is updated so the clamp is observable).
- thread_pool docs: aging_threshold is documented as an ELIGIBILITY interval
  (evaluated at dequeue; a long in-flight job can delay a hop arbitrarily),
  not an at-most wait bound; the worker-band Critical latency claim is
  qualified for the empty-Critical-queue case and states the FIFO wait behind
  an existing Critical backlog.
- pc/tests/socket_reactor: verify DSCP actually lands via getsockopt(IP_TOS)
  (CS1 and EF read back) and cover the out-of-range path (registration
  succeeds, TOS stays at the OS default). Suite now 45/45.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@finger563

Copy link
Copy Markdown
Contributor Author

Addressed the four "suppressed" Copilot comments (which never surfaced as review threads) in 0bd91af, plus the stub datetime import (already autofixed in 758d1aa0bd91af also adds import enum, which the stub referenced without importing):

  • task.cpp ESP startup clamp: start() now clamps the loaded priority to configMAX_PRIORITIES - 1 exactly like set_priority() (and updates the stored atomic), so an out-of-range configured value can no longer make startup fail.
  • PR description accuracy: the description now states host priority application is an explicit opt-in via BaseConfig::host_realtime (plus the corrected aging and Critical-latency wording, and current test numbers).
  • DSCP actually verified: the pc test reads back getsockopt(IP_TOS) and asserts CS1/EF landed on the Low/Critical sockets — deleting the setsockopt now fails the suite — and covers the out-of-range path (registration succeeds, TOS stays at the OS default). socket_reactor suite: 45/45.
  • Aging doc: aging_threshold is now documented as an eligibility interval evaluated at dequeue (a long in-flight job can delay a hop arbitrarily; destination backlog adds further wait) with the actual guarantee (progress toward Critical under sustained load) stated instead of an at-most bound.
  • Worker-band latency claim: qualified for the otherwise-empty-Critical-queue case, and now states that a Critical backlog drains FIFO ahead of a new arrival.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

components/task/src/task.cpp:458

  • The startup priority application can overwrite a newer live set_priority() request. The worker reads priority_ without serializing with set_priority()'s OS call, so this interleaving is possible: startup reads 10, set_priority(3) stores/applies 3, then startup applies the stale 10. get_configured_priority() then reports 3 while the thread actually runs at 10. Re-read and reapply until no concurrent update occurred, or serialize both OS applications with a dedicated mutex.
  if (config_.host_realtime) {
#if defined(_WIN32)
    apply_thread_priority_to_handle(GetCurrentThread(), priority_.load());
#elif defined(__linux__) || defined(__APPLE__)
    apply_thread_priority_to_handle(pthread_self(), priority_.load());

components/task/src/task.cpp:99

  • On macOS this fallback warning recommends Linux-only CAP_SYS_NICE/RLIMIT_RTPRIO remediation. Since this branch also compiles under __APPLE__, macOS users receive an inapplicable error message; use platform-specific text here.
    } else if (!rt_unavailable_warned.exchange(true)) {
      logger_.warn("Could not apply SCHED_FIFO priority {} to task '{}' ({}); running without "
                   "realtime priority; grant CAP_SYS_NICE or configure RLIMIT_RTPRIO for RT "
                   "scheduling (e.g. PREEMPT_RT)",
                   param.sched_priority, config_.name, strerror(err));

…opriate RT warning

- The worker's startup self-application could overwrite a newer concurrent
  set_priority(): startup reads 10, set_priority(3) stores+applies 3, startup
  applies the stale 10 (thread runs at 10, get_configured_priority() says 3).
  Both OS applications now hold a dedicated priority_apply_mutex_ and apply
  priority_.load() inside the lock, so every application converges on the last
  stored value. A dedicated mutex (not thread_mutex_) because notify_and_join()
  holds thread_mutex_ across join() - the worker taking it at startup could
  deadlock a stop() issued right after start().
- The SCHED_FIFO-unavailable warning no longer recommends Linux-only
  CAP_SYS_NICE/RLIMIT_RTPRIO remediation on macOS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@finger563

Copy link
Copy Markdown
Contributor Author

Addressed the two new "suppressed" Copilot comments (again no review threads to resolve) in 17a4f0c:

  • Startup vs live set_priority() application race (task.cpp): the worker's startup self-application could overwrite a newer concurrent set_priority() with a stale value (thread runs at the old priority while get_configured_priority() reports the new one). Both OS applications now hold a dedicated priority_apply_mutex_ and apply priority_.load() inside the lock, so every application converges on the last stored value. A dedicated mutex rather than thread_mutex_ because notify_and_join() holds thread_mutex_ across join() — the worker taking it at startup could deadlock a stop() issued right after start().
  • macOS warning text: the SCHED_FIFO-unavailable fallback warning no longer recommends the Linux-only CAP_SYS_NICE/RLIMIT_RTPRIO remediation when built for macOS.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

components/thread_pool/README.md:16

  • Stats does not expose per-band rejected counters; only submitted, executed, and aged are broken down by band. This currently promises a metric users cannot retrieve.
- Thread-safe stats for submitted / executed / rejected jobs (total and per band)

components/thread_pool/include/thread_pool.hpp:240

  • The return documentation lists per-band rejected counts, but Stats has no band_rejected field. Limit the per-band claim to the three arrays actually returned.
  /// @return Stats struct with total and per-band submitted / executed / rejected / aged counts.

Application code writes Dscp::EF / Dscp::CS1 / Dscp::AF41 instead of magic
numbers. New header socket/include/dscp.hpp with the IANA-registered DiffServ
code points (CS0-CS7 class selectors, AF11-AF43 assured forwarding, EF,
VoiceAdmit, LE - RFC 2474/2597/3246/5865/8622) plus constexpr dscp_to_tos().
UdpSocket::ReceiveConfig::dscp is now std::optional<espp::Dscp>; a custom code
point remains expressible via static_cast<Dscp>(0-63) and out-of-range values
are still rejected with a warning.

- socket_reactor: applies dscp_to_tos(); pc test + esp32 example use the named
  values (invalid-path test uses static_cast<Dscp>(200)).
- python: espp.Dscp enum bound (registered before its first default-arg use);
  ReceiveConfig.dscp and SocketReactor.add_udp_receiver take espp.Dscp; stub
  updated; dscp.hpp added to autogenerate_bindings.py and the Doxyfile
  (inc/dscp.inc referenced from socket_reactor.rst).
- docs: socket README + socket_reactor.rst show the named code points.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The README and stats() docs promised per-band rejected counts that Stats did
not provide. Add the missing counter rather than weakening the docs: every
rejection path (null job, stopped/stopping, queue full, blocking submit woken
by stop) attributes to the submitted band, and stop()'s dropped-queued-jobs
accounting attributes each dropped job to the band it was queued in. Exposed
in the fmt formatter, python bindings, and stub; pc test asserts both the
band-less (Normal) and banded (Critical) attribution. Suite now 62/62.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@finger563

Copy link
Copy Markdown
Contributor Author

Addressed the two latest "suppressed" Copilot comments (README/stats() docs promising per-band rejected counts that Stats did not provide) in fcaaf43 — by adding the missing counter rather than weakening the docs: new Stats::band_rejected array, incremented on every rejection path (null job, stopped/stopping, queue full, blocking submit woken by stop) attributed to the submitted band, with stop()'s dropped-queued-jobs accounting attributed to the band each job was queued in. Exposed through the fmt formatter, the Python bindings, and the stub; the pc test asserts both band-less (Normal) and banded (Critical) attribution, and the Python smoke test verified band_rejected == [1, 0, 0, 0] for a Critical rejection end-to-end. thread_pool suite: 62/62.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

components/thread_pool/include/thread_pool.hpp:240

  • This return description promises per-band rejected counts, but Stats only contains a total rejected counter; only submitted, executed, and aged are per-band. Correct the public API documentation to match the struct.
  /// @return Worker thread count.

lib/python_bindings/pybind_espp.cpp:2714

  • The Python-facing stats documentation does not preserve the new start-time semantics: band_executed says only “executed,” while the adjacent total still says “successfully executed.” Since both counters are incremented before job() runs, Python users could incorrectly treat them as completion counters. Document that both mean execution has begun.
            .def_readwrite("band_executed", &espp::ThreadPool::Stats::band_executed,
                           "/< Jobs executed per band (by the band they were popped from, i.e. "
                           "after any aging promotions).")

lib/python_bindings/espp/init.pyi:4175

  • The regenerated stub likewise describes these as completed executions, but the implementation now increments both counters at job start. Keep the stub aligned with the runtime/C++ API so IDE documentation does not present these as completion counters.
        band_executed: List[int]                             #/< Jobs executed per band (by the band they were popped from, i.e. after any aging promotions).

components/task/include/task.hpp:304

  • The live-update contract is stated unconditionally here, but the host implementation only calls the OS API when BaseConfig::host_realtime is true; otherwise it stores the value and returns false. Make that opt-in explicit in this method's documentation so callers do not expect a live host change from set_priority().
   *          currently running, the change is also applied to the live task
   *          immediately: via vTaskPrioritySet() on ESP, via
   *          pthread_setschedparam() (SCHED_FIFO for priority >= 1, default
   *          scheduling for priority 0) on Linux/macOS, and via
   *          SetThreadPriority() on Windows. See BaseConfig::priority for the
   *          per-platform semantics (including the unprivileged-Linux graceful
   *          fallback).

Comment thread doc/Doxyfile
Comment on lines 418 to 422
$(PROJECT_PATH)/components/socket/include/socket.hpp \
$(PROJECT_PATH)/components/socket/include/dscp.hpp \
$(PROJECT_PATH)/components/socket/include/udp_socket.hpp \
$(PROJECT_PATH)/components/socket/include/tcp_socket.hpp \
$(PROJECT_PATH)/components/socket/include/socket_reactor.hpp \
… termios macros

The esp32-timer-cam CI build failed because newlib's <sys/termios.h> defines
CS5/CS6/CS7 as macros (character-size bits), which preprocessed into the
all-caps Dscp enumerators and destroyed the enum in any TU including both
headers (e.g. examples using the cli component). Renamed to espp-style
CamelCase - Cs0..Cs7, Af11..Af43, Ef, Le, VoiceAdmit - which is macro-proof
and consistent with QosBand::Critical; the doc comment explains the naming
and keeps the RFC names in prose. All C++/python/stub/doc usages updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants