[jtniehof]

This is basically a sequel to #241 and #242 that does the same work for updating to ruff 0.16.10 (as suggested in #225). However I didn't do the black AST-checking here, assuming that the formatting is reasonably stable.

Some notes:

• Forgot to set the line length in Ruff formatting and linting auto-fixes #241, so set it here....which means the formatting diff is terrible again
• Almost all of these are by hand and probably should be checked (obnoxiously!) one commit at a time; the exceptions are "Run ruff check --fix with new ruff" and "ruff 0.16 auto-formatting"
• I added a git blame ignore-revs file to spotlight the formatting revisions and hopefully skip over them
• I tried to fix rather than noqa as much as possible. However, that wasn't always possible, and I think we can just accept this as technical debt to potentially address later. In some cases I'm pretty sure the linter issues can only be fixed by a substantial refactoring, which would probably be good but would need careful coverage checks and such as well.
• Similiarly, I am ignoring certain rules that may be worth addressing in the future:
  • Ignoring all the timezone checks (DTZ) because we heavily use naive datetimes assuming UTC. Probably worth thinking about on a project level at some point.
  • Also ignoring LOG015, calling methods on the root logger. This is worth investigating if we want to instantiate a logger or just use warnings.warn, but I wanted to preserve behavior out of caution for what might happen in our containerized environment
• I am inconsistent about when I added return value checking to subprocess.run. Basically if the run was the end of a chain of logic, I usually added a check, figuring that blowing up spectacularly is okay if there was no "next step" that was going to happen
• We had a lot of except Exception that made me very unhappy; some are in quite large try blocks. The chance that we might have real bugs being squashed there is real. If a real fix wasn't obvious, I tried to focus a little bit where I could. In other cases I just noqa'd it. This is real technical debt to address.

Because of the last two items in particular, there is a very good chance that any code without robust unit tests might wind up failing when deployed to production. I think that's a chance worth taking. Maybe we should try running a bunch of files through in dev before merging?

[leowerneck]

Started to review now.

[leowerneck]

LGTM.

It's hard to review every single change because of how many files are there, but I reviewed some of the key commits that seemed to be more than "just reformat" and it everything seems sensible, taking your PR description in mind.

I'd be fine with merging this and testing on dev, which would give us some reassurance that these formatting PRs didn't break anything.

[leowerneck]

Okay, here are some preliminary findings. I have the full paper-trail if you're interested in looking at it at some point (not very interesting, as you may imagine), but here's the summary:

• I picked a single-day, 2026-10-20, no particular reason for it.
• I re-processed locally all 18 L3+ files for that day, using the exact same command that Dagster used. Here is the list of job IDs I looked at:
  1. a92b9fbc: CoDICE L3a hi-direct-events
  2. 443d4bbe: CoDICE L3a lo-direct-events
  3. 9505f17f: CoDICE L3a lo-partial-densities
  4. 7cd53b69: CoDICE L3a lo-sw-charge-state-distributions
  5. f50b028d: CoDICE L3a lo-sw-ratios
  6. d6a1dd2e: CoDICE L3b hi-pitch-angle
  7. 2742b99b: CoDICE L3b lo-heplus-3d-distribution
  8. 1f48d216: CoDICE L3b lo-heplusplus-3d-distribution
  9. 40e450d8: CoDICE L3b lo-hplus-3d-distribution
  10. e15883d7: CoDICE L3b lo-oplus6-3d-distribution
  11. 0a58586f: GLOWS L3a hist
  12. 6e977f82: HIT L3 direct-events
  13. 85f9c23a: HIT L3 macropixel
  14. 58e93d5a: SWAPI L3a alpha-sw
  15. 26f5350c: SWAPI L3a proton-sw
  16. 1b2014d7: SWAPI L3a pui-he
  17. 3e404a4f: SWAPI L3b combined
  18. 87b46258: SWE L3 sci
• Most files (11/18) only differed in the attribute "Generation_date" and/or "ground_software_version" (42 vs. 41), so I marked those as "OK".
• Of the remaining 7 files, 6/7 (ha!) differ only by float32 noise, so I also marked those as "OK".
• 1 file had big changes: swe-l3-sci
  • Before, SWE L3 accepted two or more curvature peaks in the spike peak check (pitch_calculations.py).
  • After commit c362dd11, it now requires exactly two (len(peaks) == 2).
  • This causes a BACKUP_SPLINE_UNRESOLVED in swe_flags to be set for almost all records.
  • Because this data is set to NaN, any fit that falls back to the spline would have a NaN value.
  • To restore previous behavior, we can simply change it back to len(peaks) >= 2, unless the change was intentional.

I think resolving the last point before doing the full-month check for SWAPI/SWE is best.

[jtniehof]

Good catch. That was one of those cases of an IndexError from too few elements being caught by a broad except and I was too restrictive in fixing that. Fix pushed, and agreed, the full month check will be a lot better with this fixed.

[leowerneck]

GLOWS L3b+ have been validated, with the caveats that follow. L3b + L3c are identical. L3d has differences of order ~0.027% in 2 of 1068 lyman_alpha values for 1 file, everything else matches. L3e match exactly for repoints 355-368, but in 369-419 we have up to ~0.41% differences in surv_prob.

I believe the differences can be explained by the fact that the SDC files used an older SPICE kernels. Pending any alarm bells that this might have raised for you, I would say GLOWS is fine.

Still looking into SWAPI/SWE.

[jtniehof]

Yes, those differences are tiny, and easily explained by updated SPICE kernels or external dependencies. Any problems from my changes are far more likely to be disastrous failure or huge changes.

[leowerneck]

Okay, I just finished testing SWAPI & SWE and I think we're good to go. It took me a while because I was seeing some crazy differences in SWAPI's alpha-sw that took me down a rabbit hole, but I narrowed it down to tiny differences in the density caused by the fact I'm running these on a mac, which are then amplified in the temperature since the fit is somewhat degenerate.

To conclude these are benign, I picked a file with large differences between the SDC and my reprocess, reverted my repo back to the exact same commit that generated the SDC file, processed it again, and confirmed that this file is bit-identical to the file generated by your branch.

All of this to say: LGTM!