Conversation
UrlScraper now stores every response it fetches in tmp/cache/<slug> and serves subsequent runs from there, which makes iterating on a scraper's filters a lot faster. Only successful responses are stored, so transient failures aren't pinned forever. Cached responses are collected by the Requester and handed over iteratively rather than from within Hydra#add, because delivering them right away would nest one request's callbacks inside the previous one's and overflow the stack on large documentations. thor docs:clean deletes the caches. It recognizes them by a marker file so that it leaves the assets cache alone.
The cache files were Marshal dumps, which are opaque when you open one to find out what a scraper actually got back. They're now JSON, in the entry schema of the HTTP Archive format: http://www.softwareishard.com/blog/har-12-spec/ An archive is a log of many entries; keeping one entry per file instead means the cache stays incremental, at the cost of the files not being valid archives on their own. Bodies that aren't valid UTF-8 fall back to the spec's base64 encoding, and are handed back to the scrapers as binary either way, so a warm run sees exactly what a cold one does. Cache files gained a .json extension, so the entries written by the previous format are ignored; run thor docs:clean to drop them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.