Skip to content

glaas 0.3.0: new WHO pull, stored snapshot, rebuildable data - #12

Merged
larnsce merged 20 commits into
mainfrom
dev
Nov 13, 2026
Merged

larnsce merged 20 commits into
mainfrom
dev

Conversation

@larnsce

@larnsce larnsce commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

What changes for users

  • The data come from a new pull of the WHO source on 13 November 2026. The dataset has 259,546 rows (0.2.0: 259,313) and the same 121 variables with the same names and types.
  • The data are a plain tibble. The import details that readr had attached to the object in 0.2.0 are gone.
  • The CSV and XLSX files of the data are attached to the release and archived on Zenodo. The README links them. They are too large for the repository (272 MB and 105 MB).
  • Lars Schöbitz is the maintainer. Adriana Clavijo Daza, Colin Walder and Nicolò Massari are authors.
  • The package has a data dictionary with one description per variable, a shorter title, keywords, and the place and period the data cover. The vignette is now an article on the website.

What changes in the repository

  • The WHO pull is stored as data-raw/2026-11-13-glaas-who-pull.csv.xz. data-raw/data_processing.R replaces DATASET.R, rebuilds the data from the snapshot without a connection, and gives the same file on each run.
  • A workflow checks the WHO source once a month (third day, 06:17 UTC) and opens one issue when the row count or the newest load time differs from the stored state in data-raw/source-state.json (Monthly check of the WHO source, with an issue when it changes #11).
  • R CMD check runs on main and dev.
  • Version 0.2.0 cannot be rebuilt: its source pull was not kept. NEWS says so.

Closes #8. Closes #11. Part of #10, #9 and #7.

Data diff from 0.2.0

  • Same survey cycles (2013 to 2024), same 151 countries, territories and areas, same 333 indicators; 233 rows more.
  • WHO loaded the whole source table again between 20 and 23 July 2026, so every record identifier and load time is new.
  • 19 answers of the 2024 cycle have another value (Niger 8, Ethiopia 3, Saudi Arabia 8), and 9 rows have another data year.
  • WHO recoded some dimension values (for example SURVEIL_A became SURVEIL); filters written against 0.2.0 that use them can break.
  • About 5,400 rows have another flag, comment or colour code (is_latest, published_country_highlights, the comparability flags).

Open points

Test plan

  • R CMD check passes on GitHub for dev (first run of the workflow)
  • devtools::check() locally: 0 errors, 0 warnings, 0 notes
  • Rscript data-raw/data_processing.R leaves data/glaas.rda unchanged
  • washr::check_publication_readiness(): only readme_extdata_links fails
  • The site builds after the merge and shows the article and the new row count
  • Release v0.3.0 created, with glaas.csv and glaas.xlsx attached under exactly these names
  • The two download links in the README work
  • Zenodo deposit as dataset in the openwashdata community
  • washr::update_citation(doi = ...) with the new DOI
  • Sync dev with main and push origin/dev
  • Run the monthly check once by hand

rainbow-train and others added 20 commits November 11, 2026 07:36
Create data-raw/dictionary.csv with washr::setup_dictionary() and write
a description for each of the 121 variables of the glaas data. WHO
publishes no column documentation for the source table, so the
descriptions rest on the source column names in data-raw/DATASET.R, the
values in the data and the notes on the GLAAS data portal.

Add an exception to .gitignore so that the dictionary is tracked.

Refs #10

Assisted-by: Claude Fable 5.2
Run washr::setup_roxygen() and devtools::document() so that R/glaas.R
and man/glaas.Rd carry the descriptions of data-raw/dictionary.csv, one
entry per variable. Title, description and source are unchanged.

Refs #10

Assisted-by: Claude Fable 5.2
Written by washr::update_metadata(). The roxygen stamp in DESCRIPTION
comes from devtools::document().

Refs #10

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154SimK7rT4Xy3fsA5rnGSg
Two conflicts, both from lines added at the end of a file on each side.
Both sides are kept: .gitignore has the exception for the dictionary
(dev) and docs/ (main); DESCRIPTION has the roxygen version field (dev)
and the Config/washr fields (main).

Assisted-by: Claude claude-fable-5-2
Lars Schöbitz gets the roles cre and aut. Adriana Clavijo Daza, Colin
Walder and Nicolò Massari stay as authors. The citation files follow in
a later commit.

Assisted-by: Claude claude-fable-5-2
DATASET.R becomes data_processing.R, the name the review standard asks
for. The script reads a dated, xz compressed CSV snapshot of the WHO
table in data-raw/ when it is present and pulls from the source only
when it is not, so a version rebuilds without a connection.

The script now holds every step that shapes the data. Version 0.2.0 was
read from a CSV file with guessed column types, a step the old script
did not contain. The types are now set by column: 35 numeric, 12
logical, 2 date-time, the rest text, the same classes as in 0.2.0. The
object is a plain tibble without the import details of readr. The
script stops when a value does not fit its type or when the dictionary
does not match the variables, writes data-raw/source-state.json, and
writes CSV and XLSX exports to a folder outside the package on request.

Run twice from the snapshot, it gives the same object and the same
file.

Assisted-by: Claude claude-fable-5-2
…shot

All 259,546 rows and 121 columns of the table V_MSTR_GLS_EN_66_VIEW as
the WHO data platform gave them on 13 November 2026, under the WHO
column names, as one xz compressed CSV (12 MB). data-raw is left out of
the built package.

Assisted-by: Claude claude-fable-5-2
The dataset now has 259,546 rows (0.2.0: 259,313) and the same 121
variables with the same names and classes. WHO loaded the source table
again between 20 and 23 July 2026, so values, flags and the system
columns differ from 0.2.0. The source pull of 0.2.0 was not kept, so
that version cannot be rebuilt.

data-raw/source-state.json records the row count and the newest load
time of the source at the pull. The version number is not changed here;
it is set at the release.

Assisted-by: Claude claude-fable-5-2
The title has 61 characters (limit 65) and waits for confirmation. The
keywords carry the two discovery keywords of openwashdata, the survey,
its publishers and the five thematic areas in the data. The spatial
coverage is global (151 countries, territories and areas). The temporal
coverage runs from the 2013/2014 cycle to the 2024/2025 cycle, as the
README and the WHO pages name the cycles.

update_description() of washr 1.3.0 records its version and the parent
organisations of the publisher, and the footer of the site follows.

Assisted-by: Claude claude-fable-5-2
The review standard keeps all vignettes in vignettes/articles, where
pkgdown builds them for the site and R CMD check leaves them alone.
VignetteBuilder is no longer needed in DESCRIPTION.

Assisted-by: Claude claude-fable-5-2
Written by washr::setup_ci(). It runs on main and on dev.

Assisted-by: Claude claude-fable-5-2
On the third day of each month at 06:17 UTC the workflow asks the WHO
data platform, with one small request, for the row count and the newest
load time of the GLAAS table and compares them with
data-raw/source-state.json. When they differ it opens one issue, and no
second one while that issue is open. It uses only the token of the
workflow run.

Assisted-by: Claude claude-fable-5-2
The download section links the CSV and XLSX files of the latest release
on GitHub. The files are too large for the repository; the links work
once a release carries the two files. The size note reads the number of
rows from the data, the package citation is printed from the citation
file, and the figures are drawn from the new data.

Assisted-by: Claude claude-fable-5-2
washr::update_citation(build = FALSE) writes CITATION.cff, inst/CITATION
and .zenodo.json from DESCRIPTION, with the new maintainer, title and
keywords. The Zenodo metadata type the upload as a dataset in the
openwashdata community. inst/CITATION no longer holds the hand-written
second entry for the WHO dataset; the README still names it.

Assisted-by: Claude claude-fable-5-2
washr::update_metadata() after the changes to DESCRIPTION and the data.

Assisted-by: Claude claude-fable-5-2
format() cuts a time off after the third decimal, so a load time could
come out one millisecond early in the exports and in the source state
file. The data object is not affected.

Assisted-by: Claude claude-fable-5-2
Four descriptions state facts about the values that changed with the
new pull: feature_id is now unique, sys_version also holds 3, and the
two load times fall between 20 and 23 July 2026. The help page and the
site metadata follow, and the help page names the new row count. No
description of meaning is changed; the inferred descriptions still wait
for WHO.

Assisted-by: Claude claude-fable-5-2
Version 0.3.0 with the date of 13 November 2026 in DESCRIPTION. The NEWS
entry of the development version becomes the entry of 0.3.0 and says
that recoded dimension values can break filters written against 0.2.0.
The citation files, .zenodo.json and the site metadata follow.

Assisted-by: Claude claude-fable-5-2
The description of one variable names the units "US$" and
"US$ millions". pkgdown passes the header to pandoc as part of a
template when it renders an article, and pandoc reads "$" as the start
of a template variable, so the article failed with "Error compiling
template" on GitHub.

The two signs are written as the JSON escape $. The JSON is still
valid and reads back as "US$". washr::update_metadata() will write the
plain sign again on its next run, see
openwashdata/washr#173 for the related
readiness work; a washr fix is needed for the escape.

Assisted-by: Claude claude-fable-5-2
@larnsce
larnsce merged commit 02c5369 into main Nov 13, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants