Turn Wikidata revision history into historical graphs you can compare.
Wikidata-Flashback reconstructs dated knowledge graph snapshots from Wikidata's
pages-meta-history XML dumps. Extract the lifetime of each supported triple,
keep a reusable DuckDB event index, and export the graph and its net changes
at the timestamps you choose.
Use it to study graph evolution, compare historical knowledge, or prepare dated inputs for temporal knowledge graph experiments.
Quick start · Local setup · Outputs · Examples · Troubleshooting · Citation
- Choose your timeline. Generate yearly snapshots or supply custom timestamps.
- See what changed. Export added and removed triples between consecutive snapshots.
- Keep the history behind the graph. Interval CSVs retain creation/deletion dates and users.
- Index once, export again. Reuse the persistent DuckDB database when your dates change.
- Keep files in your workspace. Intervals, the database, and exports live under
output/. - Start with Docker or Python. Extraction uses the standard library; DuckDB is the only additional Python dependency.
Install Docker with the Compose plugin, then clone the repository:
git clone https://github.com/RYFoR/Wikidata-Flashback.git
cd Wikidata-Flashback
cp env.example .env
mkdir -p data/inputCheck that Docker and Compose are available:
docker --version
docker compose versionChoose a dump release from the Wikimedia Wikidata dump directory
and download the pages-meta-history XML files you want to process.
The Wikidata download guide
provides background on XML dumps.
Place the files directly inside data/input/. Supported extensions are .xml,
.xml.gz, and .xml.bz2; compressed files do not need to be decompressed.
The pipeline needs revision history, rather than a current-state JSON or RDF dump.
data/input/
├── history-part-001.xml.bz2
├── history-part-002.xml.bz2
└── ...
For batch discovery, filenames must end in .xml, .xml.gz, or .xml.bz2.
Wikimedia part files such as .xml-p123p456.bz2 are also supported after renaming
that suffix to .xml.bz2 (for example, part-p123p456.xml.bz2).
Keep a complete page history together; see scope and limitations.
Start with a small input before processing a large collection.
Edit config/snapshot_dates.txt. It accepts one date
or timestamp per line, with blank lines and # comments allowed:
# Compare two historical graphs
2020-12-31T23:59:59Z
2021-12-31T23:59:59Z
The supplied file contains end-of-year timestamps for 2013–2024.
Date-only values such as 2020-12-31 mean midnight UTC at the start of that day.
Provide at least two distinct timestamps to get delta files.
Configure host folders and extraction concurrency in .env:
| Variable | Default | Purpose |
|---|---|---|
INPUT_DIR |
./data/input |
Folder containing history dumps; mounted read-only |
OUTPUT_DIR |
./output |
Folder for interval CSVs, the database, and exports |
JOBS |
4 |
Number of parallel extraction processes |
Relative paths are resolved from the repository directory. You can use absolute
host paths to keep large inputs or outputs on another disk. Choose JOBS to fit
your available memory and CPUs; there is no fixed full-dump resource estimate.
docker compose up --buildThe services run in order and exit after completing their work. A successful run
shows exit code 0 for all three services:
docker compose ps -aFor a background run:
docker compose up --build -d
docker compose logs -fResults appear in output/, or the folder set by OUTPUT_DIR. With the default
12 dates and --emit both, the pipeline writes 12 snapshots and 22 delta files.
Remove the completed containers when finished:
docker compose downYour output files remain on the host. output/, default input dumps, and .env
are excluded from Git by .gitignore.
flowchart LR
A["XML revision history"] --> B["Triple lifetimes"]
B --> C[("DuckDB event index")]
C --> D["Snapshots and added/removed triples"]
| Service | Role | Persistent output |
|---|---|---|
events-processing |
Compares successive entity revisions and records triple lifetimes | output/intervals/ |
duckdb |
Converts creation/deletion intervals into indexed temporal events | output/duckdb/wikidata.duckdb |
snapshots-processing |
Sweeps those events through the requested timestamps | output/snapshots/ |
DuckDB runs embedded in the Python jobs. The duckdb service builds the index
and exits; it does not expose a database network port. The snapshot service opens
the same persistent database file. Compose waits for each preceding service to
complete successfully, keeping database writers sequential.
output/
├── intervals/
│ └── <input-filename>.csv
├── duckdb/
│ └── wikidata.duckdb
└── snapshots/
├── 2020-12-31_23-59-59_snapshot.csv
├── 2021-12-31_23-59-59_snapshot.csv
├── 2020-12-31_23-59-59_to_2021-12-31_23-59-59_added.csv
└── 2020-12-31_23-59-59_to_2021-12-31_23-59-59_removed.csv
These files have a header and one row per continuous lifetime of a triple:
| Column | Meaning |
|---|---|
s, p, o |
Subject, predicate, and object in the tool's RDF-like representation |
cdate, cuser |
Revision timestamp and user when the triple appeared |
ddate, duser |
Revision timestamp and user when it disappeared; empty while open-ended |
Illustrative row:
s,p,o,cdate,cuser,ddate,duser
wd:Q100,wdt:P31,wd:Q200,2020-01-01T00:00:00Z,EditorA,2021-01-01T00:00:00Z,EditorBA triple is present at time t when cdate <= t < ddate. An empty deletion
date is open-ended. Removing and later re-adding a triple creates separate lifetimes.
Exports contain three columns, s,p,o, without a header. They use standard
CSV quoting and export each present triple once. Use a CSV reader when values
contain commas or quotes.
- Snapshot: triples present at the requested timestamp.
- Added: triples present at the later endpoint and absent at the earlier endpoint.
- Removed: triples present at the earlier endpoint and absent at the later endpoint.
Deltas describe net graph changes. A triple added and removed entirely between two endpoints does not appear in their delta files.
Requires Python 3.10+ and Bash. Run these commands from the repository root:
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
mkdir -p data/inputPlace history dumps in data/input/, then extract their triple lifetimes:
bash src/events/run_wd_history_batch_parallel.sh \
--input-dir data/input \
--out-dir output/intervals \
--jobs 4Build the database and generate snapshots and deltas:
python3 src/snaphots/create_snapshots_event_sweep.py \
--input-dir output/intervals \
--output-dir output/snapshots \
--work-db output/duckdb/wikidata.duckdb \
--dates-file config/snapshot_dates.txt \
--keep-work-db --sort-outputThe current source directory is spelled src/snaphots/.
For the default yearly dates, the wrapper provides the same paths:
bash src/snaphots/createAllYearlySnapshots.shLocal exports are protected against accidental replacement. Pass --overwrite
when you intend to regenerate files at the same paths.
Edit config/snapshot_dates.txt, then reuse the existing database:
docker compose run --rm --no-deps snapshots-processingThis skips extraction and indexing. The Compose snapshot job overwrites matching exports; older files with different date labels remain in the output folder.
python3 src/snaphots/create_snapshots_event_sweep.py \
--input-dir output/intervals \
--output-dir output/duckdb \
--work-db output/duckdb/wikidata.duckdb \
--build-only--build-only keeps the database and does not require dates.
python3 src/snaphots/create_snapshots_event_sweep.py \
--output-dir output/custom \
--work-db output/duckdb/wikidata.duckdb \
--reuse-work-db \
--dates 2020-01-01 2021-01-01 \
--emit both --sort-outputAn existing event index is sufficient; the interval CSV folder is not needed
for this command. Use --emit snapshots or --emit deltas to export only one kind.
Append these options to a local snapshot command as needed:
| Option | Example | Effect |
|---|---|---|
--threads |
--threads 4 |
Sets DuckDB's worker thread count |
--memory-limit |
--memory-limit 8GB |
Sets DuckDB's configured memory limit |
--temp-dir |
--temp-dir output/tmp |
Chooses a directory for spill files |
--sort-output |
--sort-output |
Sorts rows by s,p,o for stable output order |
--delimiter |
`--delimiter ' | '` |
--input-delimiter |
`--input-delimiter ' | '` |
These resource values are examples, not minimum system requirements. Extraction
concurrency is controlled separately by the batch script's --jobs option.
To inspect all available options:
bash src/events/run_wd_history_batch_parallel.sh --help
python3 src/events/wd_history_rdf_csv.py --help
python3 src/snaphots/create_snapshots_event_sweep.py --help- Completed nonempty interval CSVs are skipped, allowing extraction to resume at file boundaries. Failed batch workers remove temporary files and report failure.
- If input contents or extraction code change, remove the corresponding generated interval CSVs before extracting them again.
- A full Compose run rebuilds the event database from the available interval CSVs. To change only snapshot dates, use the independent snapshot command above.
- Removing containers with
docker compose downpreserves the host output folders. - Run one pipeline at a time against a given output folder.
| Symptom | What to check |
|---|---|
| Docker cannot connect to its daemon | Start Docker Desktop or your Docker Engine service. |
| No input files found | Check INPUT_DIR, filename endings, and that files are directly inside the input folder. |
| A dependent service does not start | Run docker compose logs events-processing duckdb and resolve the preceding job's error. |
| An early snapshot is empty | Confirm that the input covers entity history at that timestamp. |
| Work database not found | Build the index first and check the --work-db path or OUTPUT_DIR. |
| Local exports already exist | Use a new output folder or explicitly pass --overwrite. |
| Memory pressure during extraction | Lower JOBS / --jobs; each worker holds one page's revision history. |
| Snapshot processing needs resource controls | Use --memory-limit, --threads, and --temp-dir with the Python CLI. |
| Updated dump data is not reflected | Regenerate its interval CSV before rebuilding the index. |
- RDF coverage: extracts labels, descriptions, statement links and values, ranks, qualifiers, and selected direct claims. References, aliases, and sitelinks are omitted. Quantities, times, and other structured values become JSON literals. Output is an RDF-like subset, not a replica of the official Wikidata RDF export.
- History assumptions: revisions must be chronological, have UTC timestamps, and contain entity JSON directly in revision text. Invalid JSON or suppressed revision text fails extraction. Non-entity wiki pages are skipped.
- Input completeness: use complete page histories. Processing independent fragments of the same page cannot reconstruct lifetimes across file boundaries. A snapshot reflects the supplied input, not entities absent from that input.
- Attribution: user metadata is retained in interval CSVs, not in snapshot exports or the DuckDB event index.
- Scale: XML is processed page by page, with an entire page's revisions held in memory. Full-dump resource use and performance have not been benchmarked here.
- Upstream formats: historical XML payload formats may vary; see the Wikidata XML dump documentation.
The regression suite creates small synthetic histories and checks extraction, compression, namespace handling, worker failures, database persistence, snapshot boundaries, CSV quoting, and overlapping triple lifetimes. No external dump is needed.
After installing requirements:
python3 -m unittest discover -s tests -vOr run inside the container after building its image:
docker compose build snapshots-processing
docker run --rm -e PYTHONDONTWRITEBYTECODE=1 -v "$PWD:/app" -w /app \
--entrypoint python wikidata-flashback-snapshots:local \
-m unittest discover -s tests -vWithout DuckDB installed, database integration checks are skipped. The Docker image includes DuckDB and runs the complete suite.
Bug reports, documentation improvements, additional RDF mappings, and performance measurements are welcome. Read CONTRIBUTING.md for development and respectful collaboration guidelines.
For questions or problems, open an issue with your command, software versions, relevant logs, and a small reproducible input when possible. Research contact details are in CITATION.cff.
If you use Wikidata-Flashback in research, cite the software and record the commit or version, dump release, snapshot dates, and relevant processing options. Author and version metadata are provided in CITATION.cff.
Read the paper: Wikidata-Flashback - Characterising the Temporal Evolution of Wikidata (PDF).
@inproceedings{Rincon-Yanez2026,
author = {Rincon-Yanez, Diego and Keisling, Elmar and Ferranti, Nicolas},
booktitle = {Wikidata@ISWC26},
pages = {12},
publisher = {CEUR-WS},
title = {{Wikidata-Flashback - Characterising the Temporal Evolution of Wikidata}},
year = {2026},
url = {https://github.com/RYFoR/Wikidata-Flashback}
}
The software is available under the MIT License. For source data licensing, consult Wikidata's download and licensing information.
