title	Shared HTTP Cache
description	Node.js utility for fetching multiple HTTP resources with browser-like cache management.

Shared HTTP Cache

shared-http-cache fetches HTTP resources through a shared, content-addressed cache. It follows the shared-cache semantics defined by RFC 9111, with explicit assumptions and controlled implementation decisions. It provides:

• concurrent fetching for multiple resources;
• browser-like freshness, staleness, and revalidation behavior;
• request and response Cache-Control handling;
• conditional requests through ETag and Last-Modified;
• subresource integrity checks during fetch and storage;
• direct access to the underlying cacache store.

Downloaded content is stored with its HTTP response headers in cache metadata. cacache provides lockless, high-concurrency, content-addressed storage. The package is CommonJS and depends on cacache.

Install

npm install shared-http-cache

API

Create a shared cache

Construct an independent cache instance with optional storage and request behavior.

new SharedHttpCache(options?) -> SharedHttpCache

const SharedHttpCache = require('shared-http-cache');
const sharedHttpCache = new SharedHttpCache();

Options and example

new SharedHttpCache({
    cacheDir?: string,
    requestTimeoutMs?: number,
    awaitStorage?: boolean,
    onStorageError?: function,
    deferGarbageCollection?: boolean
}) -> SharedHttpCache

Option	Default	Responsibility
cacheDir	.cache	Cache storage directory
requestTimeoutMs	5000	Base request timeout in milliseconds
awaitStorage	false	Await cache writes before completing fetch
onStorageError	process warning	Handle a detached cache-write failure
deferGarbageCollection	true	Defer replacement cleanup to a later action

When deferGarbageCollection is false, the existing content index entry is removed before replacement, producing a clean entry at a performance cost.

const sharedHttpCache = new SharedHttpCache({
    cacheDir: '/tmp/http-cache',
    awaitStorage: false,
    onStorageError: ({ url, error }) => console.error('Cache write failed:', url, error),
    requestTimeoutMs: 1000,
});

With awaitStorage: false, fetched content is available without waiting for cache storage. A later write failure is passed to onStorageError as { url, headers, error, index }; without a handler, it is emitted as a process warning. If the handler throws or returns a rejected promise, that failure is also emitted as a process warning. With awaitStorage: true, storage remains part of the current request and rejects fetch() through its normal indexed error array.

Fetch resources

fetch(requests) processes all supplied requests concurrently. It resolves to the same cache instance when every request succeeds, enabling chained workflows, and rejects with an array of indexed errors when one or more requests fail.

sharedHttpCache.fetch(requests) -> Promise<this>

fetch([{
    url: string,
    integrity?: string,
    options?: RequestInit,
    callback?: function
}]) -> Promise<this>

await sharedHttpCache.fetch([
    {
        url: 'https://example.com/data.txt',
        callback: ({ buffer }) => console.log(buffer.toString()),
    },
]);

Callback and error behavior

The response is converted to a Buffer and passed to the callback before newly fetched content is stored. This allows callers to inspect, transform, or validate content before storage.

interface ResponseHeaders {
    readonly [name: string]: string | undefined;
    readonly age?: string;
    readonly 'cache-control'?: string;
    readonly 'content-type'?: string;
    readonly etag?: string;
    readonly expires?: string;
    readonly 'last-modified'?: string;
    readonly vary?: string;
}

callback({
    buffer: Buffer,
    headers: ResponseHeaders,
    fromCache: boolean,
    index: number
}) -> void

headers is a plain response-header object produced from Fetch Headers entries or cache metadata. It supports property access such as headers['content-type'], but it is not a Headers instance and does not provide methods such as get(), has(), or entries(). The named properties above provide editor suggestions for common headers; other response-header names remain available through the string index.

Callback errors and fetch errors are collected in the rejected error array.

await sharedHttpCache
    .fetch([
        {
            url: 'https://example.com/data.txt',
            callback: ({ buffer, headers, fromCache, index }) => {
                console.log(buffer.toString());
                console.log(headers);
                console.log(index, fromCache);
            },
        },
    ])
    .catch((errors) =>
        errors.forEach((entry) =>
            console.error(entry.index, entry.url, entry.error.message)
        )
    );

Pass fetch options

Each request's options are passed directly to the Node.js global fetch. They follow standard RequestInit semantics, including method, credentials, headers, mode, and cache mode.

request.options -> RequestInit

Request option examples

Request JSON

await sharedHttpCache.fetch([
    {
        url: 'https://api.example.com/list',
        options: { headers: { Accept: 'application/json' } },
        callback: ({ buffer }) => console.log(buffer.toString()),
    },
]);

Require revalidation with no-cache

await sharedHttpCache.fetch([
    {
        url: 'https://example.com/data',
        options: { headers: { 'Cache-Control': 'no-cache' } },
        callback: ({ fromCache }) => console.log(fromCache),
    },
]);

Permit bounded staleness with max-stale

await sharedHttpCache.fetch([
    {
        url: 'https://example.com/data',
        options: { headers: { 'Cache-Control': 'max-stale=3600' } },
        callback: ({ fromCache }) => console.log(fromCache),
    },
]);

Use a non-stored HEAD request

await sharedHttpCache.fetch([
    {
        url: 'https://example.com/resource',
        options: { method: 'HEAD' },
        callback: ({ headers }) => console.log(headers),
    },
]);

Use subresource integrity

An optional integrity value is passed to fetch for a new resource and to cacache for cache retrieval and storage.

await sharedHttpCache.fetch([
    {
        url: 'https://example.com/file.bin',
        integrity: 'sha256-abcdef...',
        callback: ({ buffer }) => console.log(buffer.length),
    },
]);

The complete integrity modes and verification timing are described in the processing model.

Access the cache store

The underlying cacache implementation is exposed directly.

sharedHttpCache.store -> cacache

Available operations include:

• sharedHttpCache.store.put(...)
• sharedHttpCache.store.get(...)
• sharedHttpCache.store.get.info(...)
• sharedHttpCache.store.rm.entry(...)
• sharedHttpCache.store.rm.content(...)

See the complete cacache API.

Listing, verification, and cleanup examples

List entries after fetching

sharedHttpCache
    .fetch(requests)
    .then((sharedHttpCache) => sharedHttpCache.store.ls(sharedHttpCache.cacheDir))
    .then(console.log)
    .catch((errors) => console.error('Errors:', errors));

Verify and compact the cache

sharedHttpCache.store.verify(cacheDir) -> Promise<Object>

// deadbeef is collected because of an invalid checksum.
sharedHttpCache.store.verify(sharedHttpCache.cacheDir).then((stats) => {
    console.log('cache is much nicer now! stats:', stats);
});

Clean entries that cannot be served from cache

const SharedHttpCache = require('shared-http-cache');

// only-if-cached also means the entry must exist and must be acceptable as cached.
(async () => {
    const cache = new SharedHttpCache({ cacheDir: '.cache', awaitStorage: true });
    const entries = await cache.store.ls(cache.cacheDir);
    const requests = Object.keys(entries).map((url) => ({
        url,
        options: { headers: { 'cache-control': 'only-if-cached' } },
    }));

    await cache.fetch(requests).catch(async (errors) => {
        for (const { url } of errors) {
            const file = url && await cache.store.get.info(cache.cacheDir, url);
            if (file) {
                await cache.store.rm.entry(cache.cacheDir, url, { removeFully: true });
                await cache.store.rm.content(cache.cacheDir, file.integrity);
            }
        }
    });
})();

This RFC 9111-based strategy removes resources that can be determined as unusable from their stored response headers. For a more flexible policy, combine only-if-cached with max-stale=<acceptedStaleness>. Empirical cleanup policies such as least-recently-used eviction are not recommended by this package.

Processing model

The cache processes each request through an explicit shared-cache decision flow:

1. Normalize request method and headers and look up the URL in cacache.
2. Parse request and stored-response Cache-Control directives.
3. Determine whether the stored entry is fresh, stale but acceptable, or requires origin access.
4. Apply no-cache, only-if-cached, must-revalidate, and proxy-revalidate constraints.
5. Serve acceptable cached content or add conditional request headers and fetch from the origin.
6. Handle successful, revalidated, gone, and error responses.
7. Invoke the callback before storing newly fetched or revalidated content.
8. Apply request, response, method, authorization, Vary: *, and integrity storage rules; non-* variant selection remains the caller's responsibility.
9. Await storage and collect request-level errors when configured; otherwise detach the write and report a later failure through onStorageError or a process warning.

Shared-cache scope and cache lookup

Shared-cache scope and assumptions

The cache is shared rather than private and applies shared-cache rules:

• Request methods other than GET are served but not stored.
• Requests containing Authorization are served but their responses are not stored.
• Responses with Cache-Control: private or Set-Cookie are served but not stored.
• Responses with Vary: * are served but not stored.
• Partial responses carrying Content-Range are served but not stored.
• Time calculations rely exclusively on locally recorded timestamps rather than server-provided Date.
• No heuristic freshness is used.
• Storage and eviction are deterministic; no background or implicit cleanup is assumed.

Vary boundary

Cache entries are indexed only by URL. A response containing Vary: * is served but not stored.

Other Vary values are preserved in response metadata, but they do not create separate cache entries and are not compared with later request headers. The cache does not independently store or select representations based on fields such as Accept, Accept-Language, Accept-Encoding, or User-Agent.

Callers must ensure that every request sharing a URL and cache directory expects the same representation. When representation selection matters, use one of these approaches:

• send request Cache-Control: no-cache, no-store to bypass reuse and prevent storage;
• use distinct resource URLs;
• isolate variants in separate cache instances or cache directories.

Do not rely on a non-* Vary response header to prevent reuse of an incompatible cached representation. General variant indexing and selection require an independent cache-key policy rather than a change to cacache storage behavior.

Cache lookup and exclusions

Each request begins by determining whether a cached response exists. If no entry exists, only-if-cached produces a 504 error; otherwise the request is sent to the origin.

Two directives may short-circuit normal cache use:

• no-cache, on either the request or stored response, requires revalidation before cached data can be used;
• no-store, on either the request or new response, serves the response without storing it.

Freshness, staleness, and revalidation

Freshness evaluation

Strict freshness is evaluated before max-stale.

The response freshness lifetime is selected from s-maxage, then max-age, and then Expires when no cache-control lifetime is present. A request max-age may further limit that lifetime.

currentAge = now − storedTime + incomingAge

incomingAge is taken from the stored response Age header when present.

remainingFreshness = freshnessLifetime − currentAge

A request min-fresh value reduces the freshness available to the request:

remainingFreshness = remainingFreshness − minimumFreshness

When remainingFreshness ≥ 0, the response is served as fresh.

Stale handling and revalidation

If strict freshness fails, a request max-stale directive is considered. An unspecified value accepts any staleness; a numeric value accepts the entry when:

currentAge ≤ freshnessLifetime + maximumStaleness

If the entry exceeds the permitted staleness, the request proceeds toward revalidation or an origin fetch.

Even when max-stale permits stale content, response must-revalidate or proxy-revalidate forbids serving it. If only-if-cached also applies, the request produces a 504 error instead of contacting the origin. Otherwise, acceptable stale content may be served.

When a cached response provides ETag or Last-Modified, the cache automatically adds If-None-Match or If-Modified-Since, respectively. Revalidated entries are explicitly replaced during successful fetches to avoid unbounded index growth.

Integrity, storage paths, and origin outcomes

Integrity modes and verification timing

Use one integrity policy consistently for each URL.

Trust the source and reuse the generated digest

When a request does not supply integrity, the callback receives a newly fetched body before storage. cacache.put() then calculates a digest with its default sha512 algorithm and stores the content under that digest. Later URL-based cache reads use cacache.get(), which validates the content against the digest recorded in the cache entry.

After storage completes, callers can obtain the generated digest from the exposed store:

const { integrity } = await sharedHttpCache.store.get.info(sharedHttpCache.cacheDir, url);

Use awaitStorage: true when that digest is required immediately after fetch(); with awaitStorage: false, storage may still be in progress when fetch() completes.

Require a known digest

When a request supplies a trusted integrity value:

• on a cache miss, Node.js fetch verifies the incoming body before the callback, and cacache.put() verifies it again when storage is permitted;
• on a fresh cache hit, cacache.get.byDigest() performs a strict digest lookup and validates the stored content against the supplied integrity;
• for an accepted stale entry or a 304 Not Modified response, cacache.get() validates content against the digest already recorded for the URL;
• when revalidation returns a new 2xx body, the callback runs before cacache.put() validates that body against the supplied integrity.

In the last case, an integrity mismatch rejects the current fetch() when awaitStorage: true. With awaitStorage: false, the mismatch occurs after content delivery and is reported through onStorageError or a process warning.

A strict digest lookup that cannot find the requested content rejects instead of automatically retrying the origin. Accepted-stale and 304 paths validate the stored digest rather than comparing a newly supplied value with the cache entry, so callers should not change integrity identities for an existing URL and expect automatic reconciliation.

Origin request outcomes

When the cache contacts the origin:

• 2xx: the body is read and passed to the callback, then storage is attempted unless a restriction applies;
• 304 Not Modified: cached content is integrity-checked, reused, and passed to the callback, then storage with updated metadata is attempted unless a restriction applies;
• 410 Gone: the stale cache entry and its content are removed;
• any other response: an error is collected and returned through the rejected error array.

State diagram and legend

State diagram

The diagram summarizes the complete decision flow:

stateDiagram-v2
    state "Request Init" as request_init
    state "cached?" as cached
    state "only-if-cached?" as only_if_cached
    state "no-cache?" as no_cache
    state "is fresh?" as is_fresh
    state "max-stale?" as max_stale
    state "must-revalidate? / proxy-revalidate?" as must_revalidate
    state "Return 504" as return_504
    state "Send Request" as send_request
    state "no-store? see (2)" as no_store
    state "Store Response" as store_response
    state "Serve Response" as serve_response
    state "Serve Fresh" as serve_fresh
    state "Serve Stale" as serve_stale
    state "Update Metadata" as update_metadata
    state "Remove Stale" as remove_stale
    state "Return Error" as return_error

    [*] --> request_init
    request_init --> cached

    cached --> only_if_cached: no
    cached --> no_cache: yes

    no_cache --> only_if_cached: yes, see (1)
    no_cache --> is_fresh: no

    is_fresh --> serve_fresh: yes
    is_fresh --> max_stale: no, see (3)

    max_stale --> only_if_cached: no
    max_stale --> must_revalidate: yes

    must_revalidate --> only_if_cached: yes
    must_revalidate --> serve_stale: no

    only_if_cached --> return_504: yes
    only_if_cached --> send_request: no

    send_request --> no_store: 2xx OK
    send_request --> update_metadata: 304 Not Modified
    send_request --> remove_stale: 410 Gone, see (4)
    send_request --> return_error: other HTTP response

    no_store --> serve_response: yes
    no_store --> store_response: no
    store_response --> serve_response

    update_metadata --> serve_fresh
    remove_stale --> return_error

    return_504 --> [*]
    serve_response --> [*]
    serve_fresh --> [*]
    serve_stale --> [*]
    return_error --> [*]

Loading

Legend:

1. no-cache may appear on the request or response and always requires revalidation.
2. no-store may appear on the request or response; the shared-cache scope adds further storage restrictions.
3. Strict freshness excludes max-stale, which is evaluated only after freshness fails.
4. Removing content after 410 Gone is an explicit coherence decision; no heuristic eviction is used.

Real-world use cases

Fetch multiple resources concurrently

Example and context

Build one request list and process every resource through the same cache instance.

const urls = ['https://example.com/file1', 'https://example.com/file2'];
const parser = ({ url, buffer, headers, fromCache, index }) => {
    console.log(index, fromCache, url);
    console.log(headers);
    console.log(buffer.toString());
};

const requests = urls.map((url) => ({
    url,
    callback: (response) => parser({ ...response, url }),
}));

sharedHttpCache
    .fetch(requests)
    .catch((errors) =>
        errors.forEach((entry) =>
            console.error(entry.index, entry.url, entry.error.message)
        )
    );

Control tolerated staleness

Guidance

Many servers publish max-age=0, while a client may know that a bounded stale response remains useful. Supplying max-stale—commonly up to 24 hours for tolerant workflows—can reduce origin requests while preserving an explicit caller policy.

Accelerate verified cache access

Guidance

Providing an integrity value lets the cache address matching content directly by digest. This can speed cache reads and verifies fetched or stored content against the caller's expected digest.

Coordinate fetch and store operations

Guidance

Use awaitStorage: true when a workflow calls fetch and immediately continues with store operations. This ensures pending writes complete before listing, verification, or removal begins. Keep awaitStorage: false when fetched content should be usable before storage completes, and use onStorageError when the caller needs programmatic notification of a later write failure.

Intentional behavior and limitations

Cache policy and operational boundaries

• The cache is shared and does not implement private-cache semantics.
• Private or sensitive responses may be served, but the documented authorization and response restrictions prevent them from being stored.
• Cache validation does not establish that a response is trustworthy, authorized, safe to execute, or appropriate for a caller.
• No heuristic freshness or automatic stale-entry eviction is performed.
• A 410 Gone response explicitly removes the corresponding entry.
• Cleanup and eviction policy remain the consumer's responsibility.
• Least-recently-used and other empirical eviction policies are not recommended by this package; use stored response semantics and an explicitly selected staleness policy.
• max-stale is intentionally available because callers may know how much staleness their workload can tolerate.
• Callers should use one integrity policy consistently for each URL; changing integrity identities does not trigger automatic cache reconciliation.
• awaitStorage: false allows fetch completion before asynchronous storage completes; later write failures use onStorageError or a process warning and cannot reject an already completed fetch() call.
• Non-* Vary indexing and selection are outside this implementation; callers must keep URL representations stable or isolate their cache keys.
• The package exposes cacache directly, so callers remain responsible for using its lower-level operations consistently with their cache policy.

Tests

The behavioral suite contains 10 tests covering fresh and stale reuse, revalidation, integrity, storage failures, and response storage restrictions. GitHub Actions runs the suite and syntax validation on Node.js 20, 22, and 24 across Ubuntu, Windows, and macOS.

Materialize and run the tests

The test fixtures are maintained separately as public workspace data, so they are not included in the package or canonical repository. Users and contributors who need them can materialize them into a cloned repository with gh-workspace-data.

Install the GitHub CLI extension once:

gh extension install SorinGFS/gh-workspace-data

Then run the workspace-data commands from the repository:

gh workspace-data init
gh workspace-data load

The tests are materialized as ordinary local files under #/public/tests/ and remain excluded from the canonical Git repository. Run the behavioral suite and syntax validation with:

npm test
npm run check

Authoritative references

• RFC 9111 — HTTP Caching
• Fetch API and RequestInit
• Subresource Integrity
• Node.js global fetch
• cacache API

Disclaimer

This package implements a controlled shared-cache policy; it does not determine whether a resource is trustworthy, confidential, authorized, current enough for a particular application, or safe to consume. Callers remain responsible for request policy, integrity expectations, cache-directory protection, cleanup strategy, and the consequences of serving stale content.