[jtniehof]

Performs auto-formatting and auto-linting fixes with ruff on the whole L3 codebase. Closes #240

There are threefour phases and threefour corresponding commits. The diffs are going to be terrible so please do verify process.

Versions used:

ruff --version  # 0.15.2
black --version  # 24.2.0

linting

In the imap_L3_processing checkout, ran:

ruff check --fix

I then manually checked the results with:

git difftool

The results were mostly f-strings that had no substitutions (e.g. f"foobar") and unused imports. I didn't actually verify that every removed import was unused, figuring unit tests would get that.

git add -u
git commit -m "Safe auto-fixes from ruff check" --no-verify

Commit e451191

formatting

The goal here was to use two characteristics of black: 1. it is idempotent; 2. it preserves the AST of the input code. So if we run black on the original code, and then on the code after running ruff, the result should be the same.

Operating in the directory above my checkout:

cp -a imap_L3_processing/ imap_L3_processing_pre_ruff
cd imap_L3_processing
ruff format
cd ..
cp -a imap_L3_processing/ imap_L3_processing_post_ruff
cd imap_L3_processing_pre_ruff
black .  # 5 failures here, notes below
cd ../imap_L3_processing_post_ruff
black .
cd ../

405 files were changed by ruff; of them, 85 were changed differently between the two runs of black. The differences between the black-ified versions were checked by hand:

find imap_L3_processing_pre_ruff imap_L3_processing_post_ruff -type f ! -name '*.py' -printf '%f\n' | diff -q -r imap_L3_processing_pre_ruff imap_L3_processing_post_ruff -X - | cut -d ' ' -f 2 | cut -d / -f 2- > suspicious_files.txt
while read -r line ; do xxdiff imap_L3_processing_pre_ruff/$line imap_L3_processing_post_ruff/$line; done < suspicious_files.txt

suspicious_files.txt

The post-ruff black-ified code, compared to the pre-ruff blackified code:

• had commas at end of sequences
• had more space around operators
• removed blank lines
• did some single-to-double-quotes
• had other random slight spacing differences
• more tuples explicitly in parens

I identified three files to hand-fix (next section, next commit) but did not do anything in this commit--i.e. this commit is entirely the auto-fixed code. Other than those three, nothing in this check changed behavior.

There were also five files that black failed to parse before ruff ran. For these, I hand-checked the changes which ruff made:

cd imap_L3_processing
for i in scripts/confirm_reprocessing.py scripts/hi/create_all_maps_from_inputs.py scripts/hi/stack_arrays_to_match_number_of_energies.py scripts/ultra/validation/create_l3_validation_maps.py tests/debug_utils/profile_wrappers.py; do git difftool -y $i; done

All looked fine.

git add -u
git commit -m "Auto-fixes from ruff format" --no-verify

Commit d4572b0

hand fixes

This is the commit that probably needs the closest review.

imap_l3_processing/maps/mpfit.py had a lot of whitespace in comments from the original Fortran. I restored these comments, changed tabs to spaces, and marked the comment blocks no-format.

scripts/ultra/create_example_ultra_l2_map_with_power_law.py and tests/codice/l3/lo/test_codice_lo_processor.py had lines that ended with a comma, so were turned into tuples that were never assigned to anything. I removed the tuple-wrapping

git add -u
git commit -m "Hand-fix formatting" --no-verify

Commit 9ecd008

Fix mocks of unused imports

e451191 removed unused imports. However, the tests still mocked on some of the unused. There were two categories:

• 5a8a31e moved the spice-related handling of get_parent_file_names from processor to utils, but not all the mocks were updated. With the removal of the unused spiceypy import from processor, these broke. Updated the mocks to utils.
• For some reason test_read_cdf_parents in test_utils.py mocked imap_data_access on glows.l3bc.utils instead of utils. 9f5210e removed the use of imap_data_access from glows.l3bc.utils, so the import was removed. Updated to use utils.

Commit 3d15512 -- like the hand fixes, this is a good one to check more closely

[jtniehof]

I'm on the test failures.

[leowerneck]

This LGTM. I reviewed the last two steps manually. All changes seem reasonable and the fact that tests are passing is reassuring. Looking forward to reviewing the remaining issues that couldn't be auto-fixed.