flexviz does not live update dashboard when data changes #161

Activity

1. jvdd commented on Oct 9, 2026

   

   jvdd

   on Oct 9, 2026

   Contributor

   More actions

   Hi @asteppke, thanks for the clear report and the runnable example. I can reproduce it.

   TLDR; this is how polars handles references to a frame.

   What happens

   register_source wraps your DataFrame with df.lazy(). In Polars that is a snapshot. Changes you make to df in place after that (extend, vstack(in_place=True)) are not seen. You can check it without FlexViz:

   lf = df.lazy()
   df.extend(point)
   lf.select(pl.len()).collect()  # still the old length

   cache=False only switches off FlexViz's response caches. It does not make FlexViz read the DataFrame again. Our docs do not say this (and I understand your confusion here), so we will fix the docs.

   The page also never polls the server. Even with fresh data on the server, an open page only shows it after you reload or interact with it (zoom, pan, clear a selection).

   What works today

   Re-registering on each update, as you found, is the supported way:

   df.extend(point)
   register_source(SOURCE, df, cache=False)

   About your question on sending all the data: re-registering does not send anything to the browser, and it copies nothing. It only swaps the snapshot inside the server process. The browser only ever gets the downsampled output: at most n_points per line and one count per bin for the histogram. Then reload the page or zoom to see the new data.

   -> so the browser is fully unaware of the changed / swapped source.
   -> there is also quite some recompute here (that theoretically can be avoided).

   Real live mode

   FlexViz has no live or streaming mode yet. We did build a prototype before the public release (but did not further pursue it since streaming had little traction when we were doing our outreach). It had:

   • an append handle (handle.append(batch)) with checks on time order
   • a WebSocket that tells the page new data is there, so the charts refresh by themselves
   • Live and Locked modes: Live follows the stream with a sliding window (30 s, 1 m, 5 m, ...), and Locked freezes the view so you can zoom and cross-filter

   Before we decide how (and if) to bring it back, it would help a lot to know your use case:

   1. What is the data, and how does it arrive? Rows per second, and from where: a Python loop, a message queue, files, a database?
   2. How much history do you keep in view? All of it, or a sliding window (last 5 minutes, last day)?
   3. Which charts do you plan to use? Only line and histogram, or also scatter, 2D histograms, maps, ...?
   4. Do you want to zoom or brush (cross-filter) while data comes in, or only watch it live and explore later?
   5. Will one person look at the dashboard, or several people at the same time?

   Thanks again, this is very useful input.
2. jvdd

   added

   enhancementNew feature or requestNew feature or request

   on Oct 9, 2026
3. asteppke commented on Oct 12, 2026

   

   asteppke

   on Oct 12, 2026

   Author

   More actions

   Thank you @jvdd for your insightful comments! I was not aware of the df.lazy() framing and it is good to know that re-registering does not lead to a full copy & send cycle.

   Regarding the streaming mode, that would be exactly what I was looking for, a live and locked mode that can be switched between.

   Our use case comes from machine data, both measurements and diagnostic information. For your questions specifically let me try to answer them briefly:

   1. The data are floating point values, representing intensities, positions, energy values and other physical quantities. The highest frequency data is generated at 100 Hz. We either subscribe to an ZMQ channel or query a database (QuestDB, or other via SQL) and update the data in 1 second intervals.
   2. For the history we look at in the equivalent of a locked representation up to a few weeks back. In the live representation up to 1 hour.
   3. I think line plots and histograms would cover pretty much all use cases for a live plot that I can think of.
   4. Zoom I would be quite useful, for brush I see less need for live data.
   5. This would need to work for several people at the same time. It is not necessarily so that several parallel interactions would occur but that it runs on several terminals and then if something out of the ordinary occurs people have the stream right there open and accessible.

   Thanks again for looking into this!
4. jvdd commented on Oct 13, 2026

   

   jvdd

   on Oct 13, 2026

   Contributor

   More actions

   Thanks @asteppke! We built a first version of streaming on the feat/streaming branch (draft PR #163, so the API can still change).

   What it does

   • register_stream(name, df, order_by="timestamp", window=...) registers data that grows.
     stream.append(batch) adds rows, from any thread (thus no re-registering needed anymore).
   • An open page asks the server once a second for new rows and refreshes all charts by itself.
   • Live: a time axis that is not zoomed shows the last window (for example 1 hour) and slides as rows arrive.
   • Locked: zoom, or click Lock All Axes, and the view stays put while rows keep arriving. Double-click to go back to live.
   • Selections stay active and also filter the new rows.
   • Several viewers can open the same dashboard. Each tab keeps its own zoom.

   import threading
   from datetime import timedelta
   from flexviz import Dashboard, register_stream
   
   stream = register_stream("machine", df, order_by="timestamp", window=timedelta(hours=1))
   
   def produce():
       while True:
           stream.append(read_batch())  # your ZMQ or QuestDB read, e.g. once a second
   
   threading.Thread(target=produce, daemon=True).start()
   
   dash = Dashboard()  # no data: it reads the stream by name
   dash.add_figure().add_line(x="timestamp", y="intensity")
   dash.add_figure().add_histogram(x="intensity", bins=80)
   dash.show(source_name="machine")  # blocks until Ctrl-C

   Limitations of this first version

   • No Live/Locked button and no window presets. Zoom (or Lock All Axes) and double-click switch between the two.
   • window applies only to the time axis. A histogram counts all rows, not the last hour.
   • No retention: all rows stay in memory. Adding a max age or a row limit is straightforward.
   • Each refresh recomputes every chart in full. A line over the time window only reads the rows in the window, but a histogram reads all rows, so its cost grows with the history (in our test: 6.5 ms at 1 day, 59 ms at 14 days of 100 Hz data)
   • Rows must arrive in time order. A late batch raises a ValueError.
   • The page polls once a second (no push), and the server runs as one process.
   • No database source for a stream: you read the rows from QuestDB or ZMQ and append them. We are working on database sources (feat: add SQLSource for live Postgres, DuckDB and ClickHouse sources #164, feat: add Trino support to SQLSource #165), but they do not stream yet, and QuestDB is not supported yet.

   => the retention + full recompute are still 2 things we should optimize / look into (before the PR can merge in main)

   Questions

   Some questions that help us decide what to build next:

   1. Live histograms: in live mode, should a histogram show the last hour (the same window as the line charts), or all the data? Currently a zoom on the time axis does not change the histogram. A brush (cross-filter) on the time axis does filter it, but the brush does not slide with live data.
   2. Live/Locked: is "zoom = locked, double-click = live" enough, or do you need a visible switch? Which window presets would you use?
   3. History: how many channels do you have, and how many weeks must the locked view reach? At 100 Hz, one week is about 60M rows. In our test, 14 days of 3 channels used 3.3 GiB. Is it better to drop rows older than N weeks, or to keep only recent rows in memory and read older history from QuestDB?
   4. Data shape: do you have one row per timestamp with a column per channel (as in the example), or one row per timestamp, channel and value? Do all channels have the same rate?
   5. Viewers: how many screens are open at the same time? Are they on the same network as the server?
   6. Late data: can ZMQ messages arrive out of time order? If yes, by how much?

   Thanks again, your input is very useful!
5. asteppke commented on Oct 15, 2026

   

   asteppke

   on Oct 15, 2026

   Author

   More actions

   Thank you @jvdd!

   I tested the draft PR #163 and it works very nice already with some test data.

   Concretely to your questions:

   1. Preferably the same window as the line charts. At least I would put this on the wishlist depending on how difficult this is.
   2. The double-click functionality works here well but a visible switch with an indicator prevents confusion and reduces mental load. So I would prefer a visible switch.
   3. We have potentially a high number of channels (hundreds) but only a handful where the combination of 100 Hz and going back for a week is interesting. I don't fully understand what you mean by "better to drop rows older than N weeks, or to keep only recent rows in memory" in this context?
   4. We have "one row per timestamp, channel and value" as timestamps are generated at the sources. Not all channels have the same rate, the majority runs at ~1 Hz, around ten interesting ones at 100 Hz.
   5. 3-4 screens would look at the data. All are on the same network as the server.
   6. While ZMQ messages could arrive out of order we would drop those.

   When using the interface in the current state one wish would be that when I change the time axis window, e.g. by zooming, that switching to live view slides as rows arrive without the need to zoom out.