feat(scheduling): priority bands in ThreadPool + SocketReactor, RT priorities on host Task - #735
feat(scheduling): priority bands in ThreadPool + SocketReactor, RT priorities on host Task#735finger563 wants to merge 12 commits into
Conversation
…iorities on host Task
Phase 1 of priority-aware scheduling across the espp concurrency stack:
bucketed QosBand priority bands (Critical=0 / High / Normal / Low, new
components/thread_pool/include/qos_band.hpp), band-aware job dispatch, and
real cross-platform application of espp::Task priorities on host builds.
All defaults preserve existing behavior exactly: no-band submits go to
QosBand::Normal and behave as the old single FIFO queue, and every existing
API/overload keeps working unchanged.
ThreadPool (band model):
- One FIFO deque per band; workers always drain the most urgent (lowest
index) non-empty band first. New submit(job, band) / try_submit(job, band)
overloads; the no-band overloads forward with QosBand::Normal.
- Aging starvation guard (Config::aging_threshold, default 100ms, 0 =
strict): before popping, a worker promotes any front-of-band entry whose
wait exceeds the threshold up one band (to the BACK of the next band, aging
clock restarted). Approximate by design - only band fronts are examined,
O(bands) per pop - but since bands are FIFO the front is always the
longest waiter, so the bound holds: at most aging_threshold per band hop
(<= 3x threshold Low->Critical) plus the destination band's backlog at
promotion time; promoted entries enter ahead of all later arrivals, so
progress is guaranteed under any sustained load.
- Worker bands (opt-in, Config::band_worker_counts +
band_task_priorities{10,7,5,1}): per-band workers where a band-k worker
services bands 0..k (its own and more urgent) at a descending espp::Task
priority. Guarantee: every worker drains band 0 first and the band-0
workers run at the highest OS priority, so a Critical arrival waits at
most one in-progress job's remaining duration before a high-OS-priority
worker takes it (true preemption on FreeRTOS / PREEMPT_RT).
- Stats extended with per-band submitted/executed/aged counters;
max_queue_size bounds the TOTAL across bands (unchanged semantics).
- stats().executed is now counted when a worker BEGINS executing a job, so
a job's own side effects always observe it. This fixes a pre-existing
count-after-run race that real-time worker scheduling on macOS exposed
deterministically (worker descheduled between the job's completion signal
and the counter increment).
SocketReactor (priority-aware dispatch):
- Every registration carries a QosBand: UdpSocket::ReceiveConfig::band, and
band parameters (default Normal) on add_tcp_listener/add_tcp_stream/add_fd.
- Ready sockets from one select() round are stable-sorted by band and each
handler is submitted at its band, so urgent sockets dispatch first and win
the remaining pool slots under saturation; a failed submit still reverts
to re-arm + kernel-buffer backpressure exactly as before.
- UdpSocket::ReceiveConfig::dscp (optional): applied as IP_TOS = dscp << 2
at registration (best-effort, logged on failure) to mark transmitted
packets (e.g. 46 = EF); network treatment only, not local scheduling.
Task (host priority application; previously ESP-only):
- Linux/macOS: priority 0 -> default scheduling (SCHED_OTHER); priority
1..25 (FreeRTOS-convention ceiling) maps linearly onto
[sched_get_priority_min(SCHED_FIFO), sched_get_priority_max(SCHED_FIFO)],
applied via pthread_setschedparam after thread start and on
set_priority() of a live task. On EPERM (unprivileged Linux without
CAP_SYS_NICE / RLIMIT_RTPRIO) it falls back gracefully to default
scheduling with a one-time process-wide warning - task start NEVER fails
due to scheduling policy. Under PREEMPT_RT this yields true preemptive
RT scheduling.
- Windows: best-effort SetThreadPriority mapping (0 -> NORMAL, 1-8 ->
ABOVE_NORMAL, 9-16 -> HIGHEST, >=17 -> TIME_CRITICAL).
- New Task::get_configured_priority() accessor; BaseConfig::priority and
set_priority() docs now spell out the per-platform semantics.
Tests / verification:
- pc/tests/thread_pool.cpp: new sections for host Task priority fallback
(start/set_priority succeed unprivileged, priority round-trips), strict
band ordering (Critical overtakes queued Low, single gated worker),
default-band FIFO equivalence, aging rescue of a Low job under a
continuous Normal stream (bounded, counter-based), per-band stats, and a
per-band-workers smoke test (mixed 80-job load, nothing lost, latencies
logged). All 56 checks pass repeatedly.
- pc/tests/socket_reactor.cpp: new priority section - Critical-band socket
vs flooded Low-band socket on a single-worker pool; all 10 sparse Critical
messages dispatched with bounded latency (worst ~6ms observed) while the
flood keeps progressing; dscp registration exercised. 28/28 checks pass.
- Host: lib/build.sh + pc/build.sh clean; python bindings updated for the
new submit overloads (disambiguated member pointers) and
python/socket_reactor_test.py passes 20/20.
- ESP: thread_pool, socket, and rtps examples all build for esp32.
- cppcheck clean over thread_pool, socket, and task; doxygen snippet
markers verified paired, with a new "Priority Bands" example section.
Explicitly deferred to Phase 2: wiring RTPS (and other consumers) onto the
new bands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
✅Static analysis result - no issues found! ✅ |
There was a problem hiding this comment.
Pull request overview
Introduces a cross-component, bucketed priority-band scheduling foundation (espp::QosBand) and wires it through ThreadPool (banded queues + aging + optional per-band workers), SocketReactor (band-aware dispatch + optional DSCP marking), and Task (host OS priority application). This extends ESPP’s scheduling capabilities to support prioritized work while aiming to preserve existing behavior when bands aren’t used.
Changes:
- Added
espp::QosBandand implemented priority-band queuing + aging + per-band worker support inespp::ThreadPool, including per-band stats. - Updated
espp::SocketReactorregistrations and dispatch to be band-aware; added optional DSCP (IP_TOS) application for UDP receiver registrations. - Implemented host-side thread priority application in
espp::Task(Linux/macOSpthread_setschedparam, WindowsSetThreadPriority), and added/updated PC tests and an example.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pc/tests/thread_pool.cpp | Adds host-priority and band/aging/worker-band behavior tests with bounded waits. |
| pc/tests/socket_reactor.cpp | Adds a flood test validating band-aware dispatch and DSCP exercise. |
| lib/python_bindings/pybind_espp.cpp | Disambiguates ThreadPool::submit/try_submit overloads for pybind. |
| components/thread_pool/src/thread_pool.cpp | Implements per-band queues, aging promotions, per-band workers, and per-band stats accounting. |
| components/thread_pool/include/thread_pool.hpp | Exposes QosBand APIs, aging/per-band worker config, and per-band stats in the public interface/docs. |
| components/thread_pool/include/thread_pool_format_helpers.hpp | Extends formatting to include per-band stats arrays. |
| components/thread_pool/include/qos_band.hpp | Introduces the shared 4-band QoS enum and band count constant. |
| components/thread_pool/example/main/thread_pool_example.cpp | Adds a deterministic example demonstrating Critical overtaking Low and stats usage. |
| components/task/src/task.cpp | Applies host thread priorities on start and during set_priority() (best-effort fallback). |
| components/task/include/task.hpp | Documents cross-platform priority semantics and adds get_configured_priority(). |
| components/socket/src/socket_reactor.cpp | Adds per-registration band storage, stable-sorted band dispatch, and DSCP-to-IP_TOS application for UDP receivers. |
| components/socket/include/udp_socket.hpp | Extends UdpSocket::ReceiveConfig with band and optional dscp. |
| components/socket/include/socket_reactor.hpp | Updates Reactor API/docs for band-aware registration/dispatch and DSCP notes. |
Suppressed comments (1)
components/thread_pool/include/thread_pool.hpp:145
- worker_task_config defaults priority=5; on host platforms this now requests SCHED_FIFO for every ThreadPool worker even when priority bands are unused. That can cause unexpected RT scheduling (or an unconditional one-time warning) for existing host users who previously had priorities ignored.
///< Base configuration applied to every worker task. (For per-band workers the priority
///< field is overridden by band_task_priorities.)
.name = "thread_pool_worker",
.stack_size_bytes = 4096,
.priority = 5,
.core_id = -1,
};
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- esp32-timer-cam / xiao-esp32s3-sense examples: add local thread_pool to EXTRA_COMPONENT_DIRS/COMPONENTS so the in-tree component (with qos_band.hpp) wins over the stale registry copy the component manager was resolving. - task: guard sched_get_priority_min/max(SCHED_OTHER) failure (-1) before computing a priority to pass to pthread_setschedparam. - thread_pool: new Config::band_workers_realtime (default false) - host per-band workers no longer request SCHED_FIFO unless explicitly opted in (band ordering stays queue-level); ESP always applies FreeRTOS priorities. One-time info log when running without OS priority differentiation. - pc/tests: cppcheck constParameterReference fix; opt the per-band worker test into band_workers_realtime to keep covering the host RT path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Suppressed comments (5)
Previously missed (4) — in code that hasn't changed since the last review.
components/task/src/task.cpp:184
- The new thread begins executing
thread_function()immediately, before the parent callsapply_thread_priority(). A short callback can therefore run or even finish entirely under the old scheduler, and a latency-sensitive callback can perform its first work before the promised priority takes effect. Gate callback execution until the startup priority attempt completes, or apply the priority from inside the new thread before entering the callback loop.
thread_ = std::thread(&Task::thread_function, this);
#if !defined(ESP_PLATFORM)
// On ESP the priority was applied via esp_pthread above; on host platforms
// apply it to the newly-created thread now (best-effort: an unprivileged
// failure falls back to default scheduling and never fails the start).
apply_thread_priority(thread_, config_.priority);
pc/tests/socket_reactor.cpp:252
- These checks only prove that receiver registration succeeded;
add_udp_receiver()deliberately treatssetsockopt(IP_TOS)failure as non-fatal, so the test still passes if DSCP marking is broken or removed. Read backIP_TOSwithgetsockopt()and assert the expected DSCP bits for both sockets.
This issue also appears on line 287 of the same file.
check(low_id != espp::SocketReactor::INVALID_ID, "Low-band receiver registered (with dscp)");
check(crit_id != espp::SocketReactor::INVALID_ID,
"Critical-band receiver registered (with dscp)");
components/socket/include/socket_reactor.hpp:175
- The socket documentation remains stale after adding banded dispatch and DSCP: neither
components/socket/README.mdnordoc/en/network/socket_reactor.rstexplains these options, and the latter still describes only unprioritized pool submission. Update the component and example documentation with the new registration behavior and DSCP semantics.
* @note The receive_config's band field selects the priority band this
* socket's handling is dispatched at, and its dscp field (if set) is
* applied to the socket via IP_TOS here (best-effort, marks
* transmitted packets - see UdpSocket::ReceiveConfig).
lib/python_bindings/pybind_espp.cpp:2703
- This resolves overload ambiguity by binding only the legacy one-argument overload, so Python callers cannot submit a
QosBandat all.QosBandand the new ThreadPool configuration/stat fields are also not exposed in this module, leaving the advertised priority feature unavailable through the existing Python API. Bind the enum and add the band-aware overload and associated fields rather than discarding the overload here.
.def("submit",
static_cast<bool (espp::ThreadPool::*)(espp::ThreadPool::Job &&)>(
&espp::ThreadPool::submit),
py::arg("job"),
pc/tests/socket_reactor.cpp:290
- This assertion does not establish priority-aware reactor dispatch. Because the reactor permits only one in-flight handler per socket, one flooding Low socket can keep at most one 5 ms Low handler ahead of a Critical message; all messages would therefore meet the 2 s per-message timeout even with the new band sorting/submission removed. Add a deterministic case that blocks the worker, makes both sockets readable in the same batch, releases it, and asserts that the Critical callback runs first.
check(delivered == num_critical_msgs,
"all Critical messages dispatched promptly during the flood");
check(flood_processed.load() >= flood_before + 5,
"Low-band flood kept making progress alongside Critical traffic");
…ates - task: new BaseConfig::host_realtime (default false). Host platforms only apply priority to the OS thread (SCHED_FIFO on Linux/macOS, SetThreadPriority on Windows) when explicitly opted in - existing callers (ThreadPool / SocketReactor default priority 5, etc.) keep espp's historical default scheduling. ESP always applies FreeRTOS priorities, unchanged. - thread_pool: band_workers_realtime now simply sets host_realtime on the per-band worker tasks (band priorities always stored; queue-level ordering always enforced). - docs: thread_pool README/rst document bands, aging, per-band workers and the host-RT opt-in; task README/rst document per-platform priority semantics, privilege fallback and the SCHED_FIFO starvation risk; example README gains the priority-band test row; Doxyfile indexes qos_band.hpp. - python: expose Task.BaseConfig.host_realtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
components/socket/src/socket_reactor.cpp:210
dscpis documented as 0–63, but the public type accepts 0–255 and this mask silently wraps invalid values (64becomes DSCP 0,255becomes 63). Validate the value before binding/applying the option and report invalid configuration (or explicitly choose and document clamping) rather than assigning a different traffic class.
const int tos = (receive_config.dscp.value() & 0x3F) << 2;
- thread_pool: per-band worker configs can no longer leave bands unreachable -
the deepest configured band's workers service ALL bands (warn if that band
is not Low), so e.g. {1,0,0,0} with aging_threshold==0 still executes
Normal/Low submissions instead of queueing them forever (and a bounded
blocking submit can no longer deadlock on them). New pc test covers it.
- task: host_realtime scheduling is now applied by the worker thread itself at
the top of thread_function(), before the first callback invocation - the
parent-side application raced thread startup, so a short callback could run
entirely (or exit, ESRCH) at default priority. apply_thread_priority() now
delegates to a native-handle variant shared with the self-application path;
live set_priority() unchanged.
- python: bind espp.QosBand, band-aware submit()/try_submit() overloads,
ThreadPool.Config aging_threshold/band_worker_counts/band_task_priorities/
band_workers_realtime, and per-band Stats arrays; smoke-tested end-to-end
(band submit + per-band stats + coverage fallback).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Suppressed comments (6)
Previously missed (5) — in code that hasn't changed since the last review.
components/socket/include/socket_reactor.hpp:52
- The existing user-facing SocketReactor documentation was not updated for this behavior change.
components/socket/README.md:63-90anddoc/en/network/socket_reactor.rst:4-34still describe unbanded dispatch and omit the new per-registration bands, saturation ordering, and UDP DSCP option. Update both documents so users can discover and correctly configure the new API.
* Each registration carries a priority band (@ref espp::QosBand,
* default Normal). When one @c select() wakeup reports several
* sockets readable at once, their handlers are submitted to the pool
* in band order (most urgent first) and each is submitted AT its band,
* so band-aware pools (see @ref espp::ThreadPool) run urgent sockets'
pc/tests/socket_reactor.cpp:237
- This scenario does not actually distinguish band-aware dispatch from the old FIFO behavior. One-shot arming permits at most one Low handler from this single socket to be running or queued, so even an unprioritized Critical handler waits at most one 5 ms callback and easily satisfies the 2 s check. Gate the worker and queue work from multiple independent Low registrations (or otherwise build a Low backlog), then assert the Critical callback overtakes that queued work.
This issue also appears on line 238 of the same file.
.band = espp::QosBand::Low,
components/socket/src/socket_reactor.cpp:210
dscpis documented as a 0–63 code point, but values above 63 are silently masked into a different valid value here (for example, 64 becomes DSCP 0). Reject out-of-range input with a clear error instead of applying an unintended marking.
const int tos = (receive_config.dscp.value() & 0x3F) << 2;
lib/python_bindings/pybind_espp.cpp:2611
- Binding
QosBandalone does not make SocketReactor priority dispatch usable from Python:UdpSocket.ReceiveConfigdoes not expose its newband/dscpfields, and the hand-writtenSocketReactor.add_udp_receiverwrapper accepts neither argument. Extend both binding paths so Python users can configure the reactor feature introduced here.
py::enum_<espp::QosBand>(
m, "QosBand",
"*\n * @brief Priority band for queued work. Critical is the most urgent and Low the "
"least;\n * Normal is the default for band-less submissions.\n")
.value("Critical", espp::QosBand::Critical)
.value("High", espp::QosBand::High)
.value("Normal", espp::QosBand::Normal)
.value("Low", espp::QosBand::Low);
lib/python_bindings/pybind_espp.cpp:2132
host_realtimeis exposed, but the newTask::get_configured_priority()API is not bound, so Python cannot perform the documented/requested priority round trip (especially when a best-effort live change returns false). Add aget_configured_prioritybinding and update the existingset_prioritybinding text, which still says live changes are ESP-only.
.def_readwrite("host_realtime", &espp::Task::BaseConfig::host_realtime,
"*< Opt-in to applying the priority to the OS thread on host "
"platforms (SCHED_FIFO on Linux/macOS; ignored on ESP).");
pc/tests/socket_reactor.cpp:238
- The DSCP check only verifies that registration succeeds, but
setsockopt()is explicitly best-effort and registration also succeeds when it fails, so this test passes even if DSCP is never applied. Read backIP_TOSwithgetsockopt()and assert the configured DSCP bits to cover the new marking behavior.
.dscp = 8}); // CS1 "low-priority data"
…cket+task esp32 examples - task: new std::atomic<size_t> priority_ is the single source of truth after construction - set_priority() wrote config_.priority unlocked while the worker thread's startup priority application (and get_configured_priority()) read it concurrently, an unsynchronized data race. All reads (esp_pthread startup config, host self-application, getter) now go through the atomic. - socket example: new 'Reactor priority bands' scenario - Critical (with DSCP EF via IP_TOS on lwIP) + Low banded UDP receivers on one reactor; floods the Low port and verifies Critical echoes during the flood and Low after it. Referenced as a doc snippet from the SocketReactor class docs. - task example: priority section now exercises get_configured_priority() round-trips (construction / after live set_priority) and the FreeRTOS-side observation of the live change, and documents BaseConfig::host_realtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…on, real ordering test, full python parity, socket docs) - task: when applying SCHED_FIFO fails, explicitly reset the thread to SCHED_OTHER so a previously-RT thread cannot silently keep its old policy (contradicting the documented fallback); a failed reset gets its own warning. - socket_reactor: reject DSCP values > 63 with a warning instead of silently masking them to a different code point. - pc/tests/socket_reactor: new deterministic queue-jump test - a single-worker external pool (aging off, queue_size() observable) is blocked by a gated handler, 4 Low handlers queue first, Critical is submitted LAST and must run FIRST on release; this fails under band-less FIFO dispatch. The flood test now also asserts the worst Critical latency bound (<500ms). - thread_pool: class-doc contract now states the deepest-band-worker coverage exception explicitly. - python: full parity for the reactor path - ReceiveConfig gains band/dscp (ctor kwargs + attributes), SocketReactor.add_udp_receiver takes band/dscp, Task.get_configured_priority bound; QosBand enum moved before its first use as a default argument; qos_band.hpp added to autogenerate_bindings.py; the committed stub (__init__.pyi) updated for QosBand, the band submit overloads, Config/Stats band fields, BaseConfig.host_realtime, ReceiveConfig band/dscp, and Task.get_configured_priority. All smoke-tested end-to-end from Python. - docs: socket README + doc/en/network/socket_reactor.rst document bands + DSCP (incl. out-of-range rejection); socket example README lists the reactor scenarios including the new priority-bands one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
components/task/src/task.cpp:449
- A live
set_priority()can race this startup application and leave the OS thread at the old priority. For example, the worker can load priority 10 here,set_priority(3)can then apply 3 underthread_mutex_, and the worker can resume and apply the stale 10 even thoughget_configured_priority()reports 3. Serialize the startup load/application with the same mutex used byset_priority()so whichever operation runs last observes/applies the latest value.
if (config_.host_realtime) {
#if defined(_WIN32)
apply_thread_priority_to_handle(GetCurrentThread(), priority_.load());
#elif defined(__linux__) || defined(__APPLE__)
apply_thread_priority_to_handle(pthread_self(), priority_.load());
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.
Suppressed comments (5)
Previously missed (4) — in code that hasn't changed since the last review.
components/task/src/task.cpp:169
- Initial ESP priorities still bypass the documented clamp: unlike
set_priority(),start()copies an arbitrary configured value directly intoesp_pthread_cfg_t. A value at or aboveconfigMAX_PRIORITIEScan therefore make startup fail instead of being clamped toconfigMAX_PRIORITIES - 1. Clamp the loaded value here and updatepriority_before callingesp_pthread_set_cfg().
thread_config.prio = priority_.load();
components/task/include/task.hpp:171
- The PR description says host
Config::priorityis now applied, but this new default keeps it unapplied unless callers explicitly sethost_realtime = true. The opt-in is intentional per the implementation and documentation, so the PR description should also state it; otherwise users are told the default behavior changed when it did not.
bool host_realtime{
false}; /**< Opt-in to applying the priority to the OS thread on HOST platforms (ignored
on ESP, where the FreeRTOS priority is always applied). When false (the
default) the thread uses the OS default scheduling, matching espp's historical
components/socket/src/socket_reactor.cpp:219
- The new tests pass nonzero DSCP values and check registration/delivery, but never observe
IP_TOS; deleting thissetsockopt()call would leave all tests passing. Since this is the implementation of the new DSCP API, add a supported-host assertion usinggetsockopt(IP_TOS)(and ideally cover the out-of-range path) so the marking behavior is actually verified.
const int tos = dscp << 2;
if (::setsockopt(fd, IPPROTO_IP, IP_TOS, reinterpret_cast<const char *>(&tos), sizeof(tos)) <
0) {
components/thread_pool/include/thread_pool.hpp:46
- This advertised upper bound is not provided by the implementation. Aging is checked only when a worker next dequeues work, so a long in-flight job can make a hop occur arbitrarily later than
aging_threshold; destination-band backlog adds further delay. Describe the threshold as an eligibility interval rather than an at-most wait bound.
This issue also appears on line 62 of the same file.
* front is always the longest-waiting entry of its band. The resulting bound:
* an entry waits at most aging_threshold per band hop (so at most
* 3 * aging_threshold to reach Critical from Low) plus the backlog of each
* destination band at promotion time; since promoted entries enter ahead of
* all later arrivals, progress is guaranteed under any sustained load. Set
components/thread_pool/include/thread_pool.hpp:67
- The stated latency guarantee ignores older Critical jobs. With a Critical backlog, a newly arrived Critical job can wait for every earlier Critical job, not just one currently running job. Qualify the claim so consumers do not treat worker bands as a hard one-job queueing bound.
* because every worker drains band 0 first and the band-0 workers run at the
* highest OS priority, a newly arrived Critical job waits at most the
* remaining duration of one already-running job before a high-OS-priority
* worker picks it up (and on a preemptive OS - e.g. FreeRTOS or Linux
* PREEMPT_RT with granted RT scheduling - that worker preempts lower-priority
* ones the moment it becomes runnable).
… stub import) - stub: import datetime/enum in __init__.pyi (aging_threshold uses datetime.timedelta; the enum module was already referenced). - task: clamp the initial ESP priority in start() to configMAX_PRIORITIES - 1 exactly like set_priority() - an out-of-range configured value must not make startup fail (priority_ is updated so the clamp is observable). - thread_pool docs: aging_threshold is documented as an ELIGIBILITY interval (evaluated at dequeue; a long in-flight job can delay a hop arbitrarily), not an at-most wait bound; the worker-band Critical latency claim is qualified for the empty-Critical-queue case and states the FIFO wait behind an existing Critical backlog. - pc/tests/socket_reactor: verify DSCP actually lands via getsockopt(IP_TOS) (CS1 and EF read back) and cover the out-of-range path (registration succeeds, TOS stays at the OS default). Suite now 45/45. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the four "suppressed" Copilot comments (which never surfaced as review threads) in 0bd91af, plus the stub
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
components/task/src/task.cpp:458
- The startup priority application can overwrite a newer live
set_priority()request. The worker readspriority_without serializing withset_priority()'s OS call, so this interleaving is possible: startup reads 10,set_priority(3)stores/applies 3, then startup applies the stale 10.get_configured_priority()then reports 3 while the thread actually runs at 10. Re-read and reapply until no concurrent update occurred, or serialize both OS applications with a dedicated mutex.
if (config_.host_realtime) {
#if defined(_WIN32)
apply_thread_priority_to_handle(GetCurrentThread(), priority_.load());
#elif defined(__linux__) || defined(__APPLE__)
apply_thread_priority_to_handle(pthread_self(), priority_.load());
components/task/src/task.cpp:99
- On macOS this fallback warning recommends Linux-only
CAP_SYS_NICE/RLIMIT_RTPRIOremediation. Since this branch also compiles under__APPLE__, macOS users receive an inapplicable error message; use platform-specific text here.
} else if (!rt_unavailable_warned.exchange(true)) {
logger_.warn("Could not apply SCHED_FIFO priority {} to task '{}' ({}); running without "
"realtime priority; grant CAP_SYS_NICE or configure RLIMIT_RTPRIO for RT "
"scheduling (e.g. PREEMPT_RT)",
param.sched_priority, config_.name, strerror(err));
…opriate RT warning - The worker's startup self-application could overwrite a newer concurrent set_priority(): startup reads 10, set_priority(3) stores+applies 3, startup applies the stale 10 (thread runs at 10, get_configured_priority() says 3). Both OS applications now hold a dedicated priority_apply_mutex_ and apply priority_.load() inside the lock, so every application converges on the last stored value. A dedicated mutex (not thread_mutex_) because notify_and_join() holds thread_mutex_ across join() - the worker taking it at startup could deadlock a stop() issued right after start(). - The SCHED_FIFO-unavailable warning no longer recommends Linux-only CAP_SYS_NICE/RLIMIT_RTPRIO remediation on macOS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the two new "suppressed" Copilot comments (again no review threads to resolve) in 17a4f0c:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
components/thread_pool/README.md:16
Statsdoes not expose per-band rejected counters; only submitted, executed, and aged are broken down by band. This currently promises a metric users cannot retrieve.
- Thread-safe stats for submitted / executed / rejected jobs (total and per band)
components/thread_pool/include/thread_pool.hpp:240
- The return documentation lists per-band rejected counts, but
Statshas noband_rejectedfield. Limit the per-band claim to the three arrays actually returned.
/// @return Stats struct with total and per-band submitted / executed / rejected / aged counts.
Application code writes Dscp::EF / Dscp::CS1 / Dscp::AF41 instead of magic numbers. New header socket/include/dscp.hpp with the IANA-registered DiffServ code points (CS0-CS7 class selectors, AF11-AF43 assured forwarding, EF, VoiceAdmit, LE - RFC 2474/2597/3246/5865/8622) plus constexpr dscp_to_tos(). UdpSocket::ReceiveConfig::dscp is now std::optional<espp::Dscp>; a custom code point remains expressible via static_cast<Dscp>(0-63) and out-of-range values are still rejected with a warning. - socket_reactor: applies dscp_to_tos(); pc test + esp32 example use the named values (invalid-path test uses static_cast<Dscp>(200)). - python: espp.Dscp enum bound (registered before its first default-arg use); ReceiveConfig.dscp and SocketReactor.add_udp_receiver take espp.Dscp; stub updated; dscp.hpp added to autogenerate_bindings.py and the Doxyfile (inc/dscp.inc referenced from socket_reactor.rst). - docs: socket README + socket_reactor.rst show the named code points. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The README and stats() docs promised per-band rejected counts that Stats did not provide. Add the missing counter rather than weakening the docs: every rejection path (null job, stopped/stopping, queue full, blocking submit woken by stop) attributes to the submitted band, and stop()'s dropped-queued-jobs accounting attributes each dropped job to the band it was queued in. Exposed in the fmt formatter, python bindings, and stub; pc test asserts both the band-less (Normal) and banded (Critical) attribution. Suite now 62/62. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the two latest "suppressed" Copilot comments (README/ |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
Previously missed (4) — in code that hasn't changed since the last review.
components/thread_pool/include/thread_pool.hpp:240
- This return description promises per-band rejected counts, but
Statsonly contains a totalrejectedcounter; only submitted, executed, and aged are per-band. Correct the public API documentation to match the struct.
/// @return Worker thread count.
lib/python_bindings/pybind_espp.cpp:2714
- The Python-facing stats documentation does not preserve the new start-time semantics:
band_executedsays only “executed,” while the adjacent total still says “successfully executed.” Since both counters are incremented beforejob()runs, Python users could incorrectly treat them as completion counters. Document that both mean execution has begun.
.def_readwrite("band_executed", &espp::ThreadPool::Stats::band_executed,
"/< Jobs executed per band (by the band they were popped from, i.e. "
"after any aging promotions).")
lib/python_bindings/espp/init.pyi:4175
- The regenerated stub likewise describes these as completed executions, but the implementation now increments both counters at job start. Keep the stub aligned with the runtime/C++ API so IDE documentation does not present these as completion counters.
band_executed: List[int] #/< Jobs executed per band (by the band they were popped from, i.e. after any aging promotions).
components/task/include/task.hpp:304
- The live-update contract is stated unconditionally here, but the host implementation only calls the OS API when
BaseConfig::host_realtimeis true; otherwise it stores the value and returns false. Make that opt-in explicit in this method's documentation so callers do not expect a live host change fromset_priority().
* currently running, the change is also applied to the live task
* immediately: via vTaskPrioritySet() on ESP, via
* pthread_setschedparam() (SCHED_FIFO for priority >= 1, default
* scheduling for priority 0) on Linux/macOS, and via
* SetThreadPriority() on Windows. See BaseConfig::priority for the
* per-platform semantics (including the unprivileged-Linux graceful
* fallback).
| $(PROJECT_PATH)/components/socket/include/socket.hpp \ | ||
| $(PROJECT_PATH)/components/socket/include/dscp.hpp \ | ||
| $(PROJECT_PATH)/components/socket/include/udp_socket.hpp \ | ||
| $(PROJECT_PATH)/components/socket/include/tcp_socket.hpp \ | ||
| $(PROJECT_PATH)/components/socket/include/socket_reactor.hpp \ |
… termios macros The esp32-timer-cam CI build failed because newlib's <sys/termios.h> defines CS5/CS6/CS7 as macros (character-size bits), which preprocessed into the all-caps Dscp enumerators and destroyed the enum in any TU including both headers (e.g. examples using the cli component). Renamed to espp-style CamelCase - Cs0..Cs7, Af11..Af43, Ef, Le, VoiceAdmit - which is macro-proof and consistent with QosBand::Critical; the doc comment explains the naming and keeps the RFC names in prose. All C++/python/stub/doc usages updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description
Phase 1 of priority-aware scheduling (foundation for prioritized RTPS endpoints, but useful to any espp user of these components). Design agreed up front: bucketed bands, not fine-grained — and everything defaults to today's behavior byte-for-byte when unused.
espp::QosBand(new,thread_pool/qos_band.hpp){ Critical=0, High, Normal (default), Low }— four buckets, shared by the pool and the reactor.espp::ThreadPool— Linux-scheduler-inspired band queuessubmit(job, band)/try_submit(job, band); per-band FIFO queues, workers drain most-urgent-first; existing overloads forward toNormal.0= strict): front-of-band promotion to the back of the next band with clock reset — O(bands) per pop. The threshold is an eligibility interval (evaluated when a worker next dequeues), not an at-most wait bound; promoted work beats all later arrivals, so progress toward Critical is guaranteed under sustained load and nothing starves.band_worker_counts+band_task_priorities): band-k workers run at band-k OS priority and service bands 0..k (the deepest configured band's workers service every band, so no band is unreachable) — with an otherwise-empty Critical queue, a new Critical job waits at most one in-flight job's remaining duration for a high-OS-priority worker, and that worker preempts lower ones via the OS scheduler (FreeRTOS on ESP; on host, RT scheduling is a further explicit opt-in — see below).Stats(submitted/executed/aged).espp::SocketReactor— priority-aware dispatchReceiveConfig, TCP listener/stream, raw fd) takes aQosBand; eachselect()batch is stable-sorted by band and submitted band-aware, so urgent sockets also win the remaining pool slots under saturation (the kernel-buffer backpressure for flooding sockets is unchanged — it was already the right behavior).ReceiveConfig::dscpappliesIP_TOS(e.g. 46/EF) for network-level marking.espp::Task— host priority application (explicit opt-in)Config::prioritywas silently ignored off-ESP. It can now be applied on host, but only whenBaseConfig::host_realtimeis set (defaultfalse— every existing caller keeps today's default host scheduling; ESP semantics unchanged). With the opt-in: Linux maps 1–25 →SCHED_FIFO[min,max]viapthread_setschedparam, applied by the worker thread itself before its first callback (true preemption under PREEMPT_RT; graceful fallback that explicitly resets toSCHED_OTHERwithoutCAP_SYS_NICE/RLIMIT_RTPRIO— start never fails), macOS best-effortSCHED_FIFO, WindowsSetThreadPriority. Liveset_priority()works on all platforms (stored priority is atomic); newget_configured_priority().ThreadPool::Config::band_workers_realtimeis the pool-level switch that sets it on per-band workers.Notable pre-existing bug found
Applying real RT priorities on macOS made workers deschedule between a job's completion-signal and the post-job
executed_++, deterministically failing three long-standing test assertions.Stats::executedis now counted at job start (documented) — the only observable stats-timing change.Testing
pc/tests/thread_pool.cpp: 59/59, repeated runs — RT-fallback round-trip, strict ordering (Critical overtakes queued Low), default-band FIFO equivalence, aging rescue under a continuous Normal stream, per-band stats, worker-band mixed-load smoke (no job lost), unreachable-band coverage fallback ({1,0,0,0} strict still executes Normal/Low).pc/tests/socket_reactor.cpp(new): 45/45, repeated runs — the flood scenario (Critical socket delivers all sparse messages with an assertedworst_latency < 500msbound while a Low socket floods), a deterministic queue-jump test (gated single-worker pool: 4 Low handlers queued first, Critical submitted last must run first — fails under band-less FIFO dispatch), and DSCP verification viagetsockopt(IP_TOS)including the out-of-range-rejection path.socketexample gains a "Reactor priority bands" scenario (Critical + DSCP EF vs Low under flood on real lwIP), and thetaskexample exercisesget_configured_priority()round-trips and the live FreeRTOS priority change.espp.QosBand, band-awaresubmit/try_submit,ThreadPool.Config/Statsband fields,Task.BaseConfig.host_realtime,Task.get_configured_priority,ReceiveConfig.band/.dscp, bandedSocketReactor.add_udp_receiver— all smoke-tested end-to-end; committed stub updated (andqos_band.hppadded toautogenerate_bindings.py).thread_pool,socket,task, andrtpsexamples build; cppcheck clean.Phase 2 (RTPS wiring: per-channel transport bands, endpoint
priority+ dedicated-port SEDP locators, facade config) follows separately.🤖 Generated with Claude Code