Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

⏪ Wikidata-Flashback

Turn Wikidata revision history into historical graphs you can compare.

License: MIT Python 3.10+ Docker Compose DuckDB Contributions welcome

Wikidata-Flashback reconstructs dated knowledge graph snapshots from Wikidata's pages-meta-history XML dumps. Extract the lifetime of each supported triple, keep a reusable DuckDB event index, and export the graph and its net changes at the timestamps you choose.

Use it to study graph evolution, compare historical knowledge, or prepare dated inputs for temporal knowledge graph experiments.

Wikidata RDF example showing Lionel Messi's multilingual labels, descriptions, citizenship statements, and start-time qualifiers

Quick start · Local setup · Outputs · Examples · Troubleshooting · Citation

Why Wikidata-Flashback?

  • Choose your timeline. Generate yearly snapshots or supply custom timestamps.
  • See what changed. Export added and removed triples between consecutive snapshots.
  • Keep the history behind the graph. Interval CSVs retain creation/deletion dates and users.
  • Index once, export again. Reuse the persistent DuckDB database when your dates change.
  • Keep files in your workspace. Intervals, the database, and exports live under output/.
  • Start with Docker or Python. Extraction uses the standard library; DuckDB is the only additional Python dependency.

Quick start with Docker

1. Get the project

Install Docker with the Compose plugin, then clone the repository:

git clone https://github.com/RYFoR/Wikidata-Flashback.git
cd Wikidata-Flashback
cp env.example .env
mkdir -p data/input

Check that Docker and Compose are available:

docker --version
docker compose version

2. Add your history dumps

Choose a dump release from the Wikimedia Wikidata dump directory and download the pages-meta-history XML files you want to process. The Wikidata download guide provides background on XML dumps.

Place the files directly inside data/input/. Supported extensions are .xml, .xml.gz, and .xml.bz2; compressed files do not need to be decompressed. The pipeline needs revision history, rather than a current-state JSON or RDF dump.

data/input/
├── history-part-001.xml.bz2
├── history-part-002.xml.bz2
└── ...

For batch discovery, filenames must end in .xml, .xml.gz, or .xml.bz2. Wikimedia part files such as .xml-p123p456.bz2 are also supported after renaming that suffix to .xml.bz2 (for example, part-p123p456.xml.bz2). Keep a complete page history together; see scope and limitations. Start with a small input before processing a large collection.

3. Choose dates and folders

Edit config/snapshot_dates.txt. It accepts one date or timestamp per line, with blank lines and # comments allowed:

# Compare two historical graphs
2020-12-31T23:59:59Z
2021-12-31T23:59:59Z

The supplied file contains end-of-year timestamps for 2013–2024. Date-only values such as 2020-12-31 mean midnight UTC at the start of that day. Provide at least two distinct timestamps to get delta files.

Configure host folders and extraction concurrency in .env:

Variable Default Purpose
INPUT_DIR ./data/input Folder containing history dumps; mounted read-only
OUTPUT_DIR ./output Folder for interval CSVs, the database, and exports
JOBS 4 Number of parallel extraction processes

Relative paths are resolved from the repository directory. You can use absolute host paths to keep large inputs or outputs on another disk. Choose JOBS to fit your available memory and CPUs; there is no fixed full-dump resource estimate.

4. Run the pipeline

docker compose up --build

The services run in order and exit after completing their work. A successful run shows exit code 0 for all three services:

docker compose ps -a

For a background run:

docker compose up --build -d
docker compose logs -f

Results appear in output/, or the folder set by OUTPUT_DIR. With the default 12 dates and --emit both, the pipeline writes 12 snapshots and 22 delta files. Remove the completed containers when finished:

docker compose down

Your output files remain on the host. output/, default input dumps, and .env are excluded from Git by .gitignore.

How it works

flowchart LR
    A["XML revision history"] --> B["Triple lifetimes"]
    B --> C[("DuckDB event index")]
    C --> D["Snapshots and added/removed triples"]
Loading
Service Role Persistent output
events-processing Compares successive entity revisions and records triple lifetimes output/intervals/
duckdb Converts creation/deletion intervals into indexed temporal events output/duckdb/wikidata.duckdb
snapshots-processing Sweeps those events through the requested timestamps output/snapshots/

DuckDB runs embedded in the Python jobs. The duckdb service builds the index and exits; it does not expose a database network port. The snapshot service opens the same persistent database file. Compose waits for each preceding service to complete successfully, keeping database writers sequential.

Understanding the outputs

output/
├── intervals/
│   └── <input-filename>.csv
├── duckdb/
│   └── wikidata.duckdb
└── snapshots/
    ├── 2020-12-31_23-59-59_snapshot.csv
    ├── 2021-12-31_23-59-59_snapshot.csv
    ├── 2020-12-31_23-59-59_to_2021-12-31_23-59-59_added.csv
    └── 2020-12-31_23-59-59_to_2021-12-31_23-59-59_removed.csv

Interval CSVs

These files have a header and one row per continuous lifetime of a triple:

Column Meaning
s, p, o Subject, predicate, and object in the tool's RDF-like representation
cdate, cuser Revision timestamp and user when the triple appeared
ddate, duser Revision timestamp and user when it disappeared; empty while open-ended

Illustrative row:

s,p,o,cdate,cuser,ddate,duser
wd:Q100,wdt:P31,wd:Q200,2020-01-01T00:00:00Z,EditorA,2021-01-01T00:00:00Z,EditorB

A triple is present at time t when cdate <= t < ddate. An empty deletion date is open-ended. Removing and later re-adding a triple creates separate lifetimes.

Snapshot and delta CSVs

Exports contain three columns, s,p,o, without a header. They use standard CSV quoting and export each present triple once. Use a CSV reader when values contain commas or quotes.

  • Snapshot: triples present at the requested timestamp.
  • Added: triples present at the later endpoint and absent at the earlier endpoint.
  • Removed: triples present at the earlier endpoint and absent at the later endpoint.

Deltas describe net graph changes. A triple added and removed entirely between two endpoints does not appear in their delta files.

Run locally

Requires Python 3.10+ and Bash. Run these commands from the repository root:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
mkdir -p data/input

Place history dumps in data/input/, then extract their triple lifetimes:

bash src/events/run_wd_history_batch_parallel.sh \
  --input-dir data/input \
  --out-dir output/intervals \
  --jobs 4

Build the database and generate snapshots and deltas:

python3 src/snaphots/create_snapshots_event_sweep.py \
  --input-dir output/intervals \
  --output-dir output/snapshots \
  --work-db output/duckdb/wikidata.duckdb \
  --dates-file config/snapshot_dates.txt \
  --keep-work-db --sort-output

The current source directory is spelled src/snaphots/. For the default yearly dates, the wrapper provides the same paths:

bash src/snaphots/createAllYearlySnapshots.sh

Local exports are protected against accidental replacement. Pass --overwrite when you intend to regenerate files at the same paths.

Usage examples

Regenerate Docker exports for new dates

Edit config/snapshot_dates.txt, then reuse the existing database:

docker compose run --rm --no-deps snapshots-processing

This skips extraction and indexing. The Compose snapshot job overwrites matching exports; older files with different date labels remain in the output folder.

Build an index without exporting snapshots

python3 src/snaphots/create_snapshots_event_sweep.py \
  --input-dir output/intervals \
  --output-dir output/duckdb \
  --work-db output/duckdb/wikidata.duckdb \
  --build-only

--build-only keeps the database and does not require dates.

Export a custom comparison from an existing index

python3 src/snaphots/create_snapshots_event_sweep.py \
  --output-dir output/custom \
  --work-db output/duckdb/wikidata.duckdb \
  --reuse-work-db \
  --dates 2020-01-01 2021-01-01 \
  --emit both --sort-output

An existing event index is sufficient; the interval CSV folder is not needed for this command. Use --emit snapshots or --emit deltas to export only one kind.

Tune resource use and formats

Append these options to a local snapshot command as needed:

Option Example Effect
--threads --threads 4 Sets DuckDB's worker thread count
--memory-limit --memory-limit 8GB Sets DuckDB's configured memory limit
--temp-dir --temp-dir output/tmp Chooses a directory for spill files
--sort-output --sort-output Sorts rows by s,p,o for stable output order
--delimiter `--delimiter ' '`
--input-delimiter `--input-delimiter ' '`

These resource values are examples, not minimum system requirements. Extraction concurrency is controlled separately by the batch script's --jobs option.

To inspect all available options:

bash src/events/run_wd_history_batch_parallel.sh --help
python3 src/events/wd_history_rdf_csv.py --help
python3 src/snaphots/create_snapshots_event_sweep.py --help

Reruns and persistence

  • Completed nonempty interval CSVs are skipped, allowing extraction to resume at file boundaries. Failed batch workers remove temporary files and report failure.
  • If input contents or extraction code change, remove the corresponding generated interval CSVs before extracting them again.
  • A full Compose run rebuilds the event database from the available interval CSVs. To change only snapshot dates, use the independent snapshot command above.
  • Removing containers with docker compose down preserves the host output folders.
  • Run one pipeline at a time against a given output folder.

Troubleshooting

Symptom What to check
Docker cannot connect to its daemon Start Docker Desktop or your Docker Engine service.
No input files found Check INPUT_DIR, filename endings, and that files are directly inside the input folder.
A dependent service does not start Run docker compose logs events-processing duckdb and resolve the preceding job's error.
An early snapshot is empty Confirm that the input covers entity history at that timestamp.
Work database not found Build the index first and check the --work-db path or OUTPUT_DIR.
Local exports already exist Use a new output folder or explicitly pass --overwrite.
Memory pressure during extraction Lower JOBS / --jobs; each worker holds one page's revision history.
Snapshot processing needs resource controls Use --memory-limit, --threads, and --temp-dir with the Python CLI.
Updated dump data is not reflected Regenerate its interval CSV before rebuilding the index.

Scope and limitations

  • RDF coverage: extracts labels, descriptions, statement links and values, ranks, qualifiers, and selected direct claims. References, aliases, and sitelinks are omitted. Quantities, times, and other structured values become JSON literals. Output is an RDF-like subset, not a replica of the official Wikidata RDF export.
  • History assumptions: revisions must be chronological, have UTC timestamps, and contain entity JSON directly in revision text. Invalid JSON or suppressed revision text fails extraction. Non-entity wiki pages are skipped.
  • Input completeness: use complete page histories. Processing independent fragments of the same page cannot reconstruct lifetimes across file boundaries. A snapshot reflects the supplied input, not entities absent from that input.
  • Attribution: user metadata is retained in interval CSVs, not in snapshot exports or the DuckDB event index.
  • Scale: XML is processed page by page, with an entire page's revisions held in memory. Full-dump resource use and performance have not been benchmarked here.
  • Upstream formats: historical XML payload formats may vary; see the Wikidata XML dump documentation.

Tests

The regression suite creates small synthetic histories and checks extraction, compression, namespace handling, worker failures, database persistence, snapshot boundaries, CSV quoting, and overlapping triple lifetimes. No external dump is needed.

After installing requirements:

python3 -m unittest discover -s tests -v

Or run inside the container after building its image:

docker compose build snapshots-processing
docker run --rm -e PYTHONDONTWRITEBYTECODE=1 -v "$PWD:/app" -w /app \
  --entrypoint python wikidata-flashback-snapshots:local \
  -m unittest discover -s tests -v

Without DuckDB installed, database integration checks are skipped. The Docker image includes DuckDB and runs the complete suite.

Contributing and support

Bug reports, documentation improvements, additional RDF mappings, and performance measurements are welcome. Read CONTRIBUTING.md for development and respectful collaboration guidelines.

For questions or problems, open an issue with your command, software versions, relevant logs, and a small reproducible input when possible. Research contact details are in CITATION.cff.

Citation

If you use Wikidata-Flashback in research, cite the software and record the commit or version, dump release, snapshot dates, and relevant processing options. Author and version metadata are provided in CITATION.cff.

Read the paper: Wikidata-Flashback - Characterising the Temporal Evolution of Wikidata (PDF).

@inproceedings{Rincon-Yanez2026,
  author = {Rincon-Yanez, Diego and Keisling, Elmar and Ferranti, Nicolas},
  booktitle = {Wikidata@ISWC26},
  pages = {12},
  publisher = {CEUR-WS},
  title = {{Wikidata-Flashback - Characterising the Temporal Evolution of Wikidata}},
  year = {2026},
  url = {https://github.com/RYFoR/Wikidata-Flashback}
}

License

The software is available under the MIT License. For source data licensing, consult Wikidata's download and licensing information.

About

No description, website, or topics provided.

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages