truncate_corrupted_offsets() dropping valid samples #176

Activity

1. cbrnr commented on Sep 10, 2026

   

   cbrnr

   on Sep 10, 2026

   Contributor

   More actions

   Hmmm, thanks for the input and the tests. We should think about the current mechanism and how we could make it more robust. I had the impression that this was a fix for something very obviously wrong, but if this is affecting (i.e., dropping) perfectly valid samples, we should come up with a better solution ASAP. @mscheltienne WDYT?
2. jamieforth commented on Sep 10, 2026

   

   jamieforth

   on Sep 10, 2026

   ContributorAuthor

   More actions

   I should add the data I'm looking at was recorded prior to labstreaminglayer/App-LabRecorder#123, so all my footer sample counts are off-by-one - making this partly a backwards compatibility issue.

   As far as I can see #167 makes the existing _detect_clock_resets() handle corrupt offsets correctly, without dropping any samples. However, this doesn't guarantee the final sample is valid, beyond ensuring it's detected as part of a different clock segment if the sample time-stamp is closer to the start of the corrupt clock segment than the end of the valid clock segment.

   If your data requires continuing to handle corrupt offsets as a special case, one possibility to avoid adding two new parameters to load_xdf() could be to use _detect_clock_resets() instead of the dedicated _detect_corrupted_clock_offset() function. Or at least make the calculation of MAD consistent across both functions so they can both use the existing reset_threshold* parameters?

   FWIW I just tried replacing this:

           if len(stream.clock_times) >= 3:
               clock_corrupted = _detect_corrupted_clock_offset(
                   stream.clock_times, stream.clock_values
               )
   

   with:

           clock_segments = _detect_clock_resets(
               stream,
               reset_threshold_stds,
               reset_threshold_seconds,
               reset_threshold_offset_stds,
               reset_threshold_offset_seconds
           )
           if len(clock_segments) > 0:
               final_segment = clock_segments[-1]
               if final_segment[0] == final_segment[1]:
                   # Redundant (=corrupt) final clock segment.
                   clock_corrupted = True
   

   and the existing truncate_corrupted_offsets tests still passed.
3. mscheltienne commented on Sep 29, 2026

   

   mscheltienne

   on Sep 29, 2026

   Contributor

   More actions

   Hey, @jamieforth - can you give a shot to #178 ?
   On our file (I actually manage to find only one of the files I used in the past..) it seems to work while removing the truncation entirely thanks to a better fix in #167
4. cbrnr

   closed this as completedin #178on Oct 11, 2026