data-sources-getting-started.adoc

Data sources and Hugging Face

SKaiNET separates artifact resolution from dataset parsing and preprocessing. Use skainet-data-source when a dataset, tokenizer, model sidecar, or fixture can live either on disk or behind a remote URI.

URI form	Meaning
file:///path/to/file	Read a local file.
https://host/path/file	Download and cache a generic remote artifact.
hf+https://huggingface.co/org/repo/resolve/main/file	Treat a Hugging Face resolve URL as a Hugging Face artifact.
hf://org/repo@main/path/file	Expand to a Hugging Face model repository resolve URL.
hf://datasets/org/repo@main/path/file	Expand to a Hugging Face dataset repository resolve URL.

Add the modules

For JVM consumers, add the source module beside the data loaders you use:

dependencies {
    implementation(platform("sk.ainet:skainet-bom:{skainet_version}"))

    implementation("sk.ainet.core:skainet-data-source-jvm")
    implementation("sk.ainet.core:skainet-data-simple-jvm")
}

The source module also publishes Kotlin/JS and Kotlin/Wasm variants (skainet-data-source-js, skainet-data-source-wasm-js; a multiplatform consumer depends on sk.ainet.core:skainet-data-source and Gradle picks the variant). See In a browser for what changes there.

Resolve one artifact

JvmDataSourceResolver materializes remote artifacts into a cache and returns a DataSourceArtifact that opens a kotlinx.io.Source. Public Hugging Face files do not need credentials. Private files should pass an explicit DataSourceAuthToken on the request or resolver. Existing Authorization headers still take precedence. On JVM, the resolver can also read HF_TOKEN / HUGGING_FACE_HUB_TOKEN from the environment as an opt-in convenience fallback.

import sk.ainet.data.source.DataSourceAuthToken
import sk.ainet.data.source.DataSourceRequest
import sk.ainet.data.source.JvmDataSourceResolver

val resolver = JvmDataSourceResolver(
    huggingFaceToken = DataSourceAuthToken.from("hf_...")
)
val artifact = resolver.resolve(
    DataSourceRequest(
        uri = "hf+https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/resolve/main/tokenizer.json"
    )
)

println(artifact.filename)
println(artifact.localPath)

val source = artifact.openSource()
try {
    // Pass the source to a parser/loader for model-sized artifacts.
} finally {
    source.close()
}

// Convenience for small sidecars and tests.
val bytes = artifact.readBytes()

For per-request credentials, pass the token directly on DataSourceRequest. This is useful when one resolver works with more than one private repository:

val privateArtifact = resolver.resolve(
    DataSourceRequest(
        uri = "hf://datasets/your-org/private-dataset@main/data/train.bin",
        huggingFaceToken = DataSourceAuthToken.from("hf_...")
    )
)

To opt into JVM environment fallback:

val resolver = JvmDataSourceResolver(
    useEnvironmentHuggingFaceToken = true
)

In a browser

WebDataSourceResolver is the Kotlin/JS and Kotlin/Wasm resolver: the same URIs, cache policies, checksum verification and token handling, over the browser’s fetch (the Ktor JS engine). Three things a tab imposes:

• No file system. Remote artifacts are cached in memory for the life of the resolver (InMemoryDataSourceArtifactStore), so a second resolve of the same URI fetches nothing, and a resolved artifact has no localPath. A file: URI resolves only to bytes the page registered under that path — what it received from a file picker or a drop — and fails otherwise. For a persistent cache, pass your own DataSourceArtifactStore over the Cache API, IndexedDB or OPFS as store.

• No environment. A Hugging Face token comes from the request or from the resolver’s huggingFaceToken; there is no HF_TOKEN to read.

• CORS. The host must allow the page’s origin. Hugging Face does; a server of your own has to send the Access-Control-Allow-Origin header.

import sk.ainet.data.source.DataSourceAuthToken
import sk.ainet.data.source.DataSourceRequest
import sk.ainet.data.source.WebDataSourceResolver

val resolver = WebDataSourceResolver(
    huggingFaceToken = DataSourceAuthToken.from("hf_..."),
    localArtifacts = mapOf("/picked/tokenizer.json" to pickedBytes)
)
val remote = resolver.resolve(
    DataSourceRequest("hf://Qwen/Qwen2.5-0.5B-Instruct@main/tokenizer.json")
)
val local = resolver.resolve(DataSourceRequest("file:///picked/tokenizer.json"))
val bytes = remote.readBytes()

The checksum is computed by Sha256DataSourceChecksum, a SHA-256 in common Kotlin that every target may use (crypto.subtle is asynchronous and cannot serve the resolver’s incremental hash). The response body is read whole: fetch hands the page the bytes, and the store keeps them in memory anyway.

Use sources with built-in loaders

MNIST and Fashion-MNIST expose per-file URI overrides. CIFAR-10 exposes an archive URI override. Defaults still point to the historical public dataset locations, so existing code keeps working.

import sk.ainet.data.mnist.MNIST
import sk.ainet.data.mnist.MNISTLoaderConfig

val token = "hf_..."
val train = MNIST.loadTrain(
    MNISTLoaderConfig(
        trainImagesUri = "file:///datasets/mnist/train-images-idx3-ubyte",
        trainLabelsUri = "hf+https://huggingface.co/your-org/mnist-idx/resolve/main/train-labels-idx1-ubyte.gz",
        huggingFaceTokenProvider = { token }
    )
)

val batches = train.batchIterator<sk.ainet.lang.types.Int8, Byte>(batchSize = 64)

The Iris provider is different: the 150-row dataset ships embedded inside the library itself, so loading it needs no network access, no cache directory and works identically on every platform target.

import kotlinx.coroutines.runBlocking
import sk.ainet.data.iris.Iris

val (train, test) = runBlocking {
    Iris.load().split(0.8, seed = 42L, stratified = true)
}

Cache behavior

Use CachePolicy.Use for normal operation, Refresh to re-download, Offline to require a cached copy, and Bypass to avoid writing the cache. Built-in JVM loaders map useCache = true to Use and useCache = false to Refresh. In a browser the cache is the resolver’s memory: Use and Offline see what this resolver fetched earlier, nothing survives the page.

import sk.ainet.data.source.CachePolicy
import sk.ainet.data.source.DataSourceRequest

val refreshed = resolver.resolve(
    DataSourceRequest(
        uri = "hf://datasets/your-org/your-dataset@main/data/train-00000.parquet",
        cachePolicy = CachePolicy.Refresh
    )
)

Keep preprocessing separate

After bytes are parsed into a dataset, continue using the existing transform DSL for image/tensor preprocessing:

import sk.ainet.data.transform.mnistPreprocessing

val preprocessing = mnistPreprocessing(ctx)