Repository navigation
Stop the HDF backend file from growing on every step - #576
Open
raashish1601 wants to merge 1 commit into
Open
raashish1601 wants to merge 1 commit into
raashish1601 wants to merge 1 commit into
Conversation
Writing the random state with attrs[...] = v recreates each attribute, which leaks a few KB per step. Use attrs.modify to update it in place, and create new files with persistent free-space tracking so rewritten compressed chunks reuse their old space. Fixes dfm#569
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #569.
HDFBackendalready takescompression=and passes it to every dataset, but the file still ends up much bigger than the data, and turning compression on makes it bigger rather than smaller. There are two causes:save_stepwrites the random state withg.attrs["random_state_i"] = von every step. Assigning an attribute that already exists deletes and recreates it, and HDF5 doesn't reuse that space, so each step leaks about 4 KB (roughly 4 GB per million steps) whether or not compression is on.attrs.modifyupdates the value in place.fs_strategy="fsm", fs_persist=True, so that space gets reused. Existing files open as before.File sizes for 32 walkers, 5 parameters and 2000 steps:
compression="gzip"The new test
test_hdf5_file_sizechecks that an uncompressed file stays close to the size of the data and that gzip makes it smaller. It fails onmainand passes here. The rest of the unit tests pass.