Where the data comes from

The DBIE portal exposes two data surfaces. Its Reports section is auth-walled — guests are redirected to a login page. Its SDMX Data Query wizard is guest-accessible and can export any series as SDMX CSV. That wizard is what we scrape.

The scraper drives the wizard with Playwright — a real browser, not raw HTTP — because the portal encrypts its query payloads in the browser and only offers the CSV download from the rendered output view. For each of the ~250 elements it selects the element in the sector tree, fills the date range, selects all dimensions, runs the query and downloads the CSV. About 30 seconds per element.

Running it

git clone https://github.com/Reserve-Bank-Innovation-Hub/dbie.rbihub.in.git
cd dbie.rbihub.in
pnpm install     # includes Playwright

pnpm scrape:tree # (re)build sdmx-tree.json — the master list of elements
pnpm scrape      # scrape everything (~2 hours; resumable)

Progress lands in data/scrape-manifest.json — one entry per element, marked ok, no-record or error. Re-running skips what already succeeded, so an interrupted scrape just resumes; delete an element’s entry to force a re-scrape. To scrape selectively:

node scripts/scrape-sdmx.mjs --sector "External Sector"
node scripts/scrape-sdmx.mjs --sub "External Debt"
node scripts/scrape-sdmx.mjs --dsd EXT_DBT_RT_RN
node scripts/scrape-sdmx.mjs --limit 10
node scripts/scrape-sdmx.mjs --headful   # watch the browser while debugging

Scraped CSVs land outside git. pnpm data:ingest then files them into the committed data/sdmx/ tree under human-readable names — that commit is what the rest of the pipeline builds from.

Processors — source files to page JSON

A processor is a small, dependency-light Node script in data/processors/ — one per dataset — that turns committed source files (SDMX CSVs, publication spreadsheets) into exactly the JSON its page renders. There are ~96 of them, producing all 345 JSON payloads the site serves.

Every processor is self-checking: it asserts hard-coded anchor values — a known figure at a known date, expected row counts, expected column sets — and exits non-zero if the source’s shape has drifted. A silent format change upstream becomes a loud build failure here, not a wrong number on a page.

Oracles — verified outputs

An oracleis a committed, known-good copy of a processor’s output, kept in data/processors/oracles/. pnpm data:verify re-runs the comparison for every dataset in one of two modes:

• exact — the source is frozen, so the output must match the oracle byte-for-byte. Any difference fails the check.
• fresh — the source has been re-scraped since the oracle was captured, so the oracle serves as a shape-and-history reference: the structure must match and the historical observations must still agree, while newer observations are allowed to extend the series.

Together they make a data refresh reviewable: sources, processors and verified outputs travel through the same commit, and nothing ships that the checks haven’t passed.

The pipeline, end to end

data/sdmx/, data/publications/       committed source files
        │
        ▼   pnpm data:build
data/processors/*.mjs                one self-checking script per dataset
        │
        ▼   pnpm data:verify
data/processors/oracles/*.json       byte-exact comparison — must ALL PASS
        │
        ▼   pnpm data:sync
public/data/*.json                   what the pages render

Deployments run exactly this: the CI build executes data:build, data:verify and data:sync before next build, so an environment serves precisely what its branch’s processors produce from its branch’s committed sources — and a failed oracle check fails the deploy.

Refreshing the data, start to finish

pnpm scrape          # refresh the raw CSVs (staged outside git)
pnpm data:ingest     # file them into the committed data/sdmx/
pnpm data:build      # regenerate the site's JSON
pnpm data:verify     # every processor and oracle check must pass
pnpm data:sync       # copy into public/data/ for a local look
pnpm dev             # eyeball the affected pages, then commit