Download, cache and read files

Used to fetch input files that are needed for the package's functions and that were built in external sources and are too large to include directly. This is a public function for transparency purposes, so that users can inspect the original inputs of this package that were not directly processed here.

If the requested file doesn't exist locally, it is downloaded from a public link and cached before reading it. This is all implemented using the pins package. It supports multiple file formats and file versioning.

Other packages and projects can use the same reader for their own pinned inputs by passing a registry: a table with the same alias, board_url and version columns as whep_inputs, usually loaded and validated from a CSV with whep_registry().

Usage

whep_read_file(
  file_alias,
  type = "parquet",
  version = NULL,
  years = NULL,
  year_col = "year",
  registry = NULL
)

Arguments

file_alias

Internal name of the requested file. You can find the possible values in the alias column of the whep_inputs dataset.

type

The extension of the file that must be read. Possible values:

parquet: This is the default value for code efficiency reasons.
csv: Mainly available for those who want a more human-readable option. If the parquet version is available, this is useless because this function already returns the dataset in an R object, so the origin is irrelevant, and parquet is read faster.
nc / nc4: Returns the path to the downloaded NetCDF instead of its contents, because these grids are read lazily by ncdf4/terra and are far too large to materialise as a tibble.
raw: Returns the file path without any processing.

Saving each file in both formats is for transparency and accessibility purposes, e.g., having to share the data with non-programmers who can easily import a CSV into a spreadsheet. You will most likely never have to set this option manually unless for some reason a file could not be supplied in e.g. parquet format but was in another one.

version

The version of the file that must be read. Possible values:

NULL: This is the default value. A frozen version is chosen to make the code reproducible when the file has a registry version. Each release will have its own frozen versions. The version is the string that can be found in whep_inputs in the version column. A blank registry version requests the latest board version.
"latest": This overrides the frozen version and instead fetches the latest one that is available. This might or might not match the frozen version.
Other: A specific version can also be used. For more details read the version column information from whep_inputs.

years

Optional integer vector of years to keep. For parquet the filter is pushed into the file, so only the row groups whose statistics overlap the requested range are read from disk and the exact set is applied afterwards. This is what makes a single-year read of a large monthly pin affordable: lpjml-soc-hydrology holds 193,317,960 rows over 1901-2022, and one year of it is 1.3 seconds and 34 MB instead of ~12 GB materialised. For csv the filter is applied after reading. The formats returned as a path (nc, nc4, raw, archives) cannot honour it and abort rather than ignore it. NULL, the default, reads the whole file.

year_col

Name of the year column years filters on.

registry

The table that maps file_alias to a board and a frozen version. NULL, the default, is whep_inputs, this package's own inputs. Another package or project passes its own registry, a table with the columns alias, board_url and version, usually read with whep_registry(); it is validated the same way before it is used. See the section Reading from another registry.

Value

A tibble with the dataset. Some information about each dataset can be found in the code where it's used as input for further processing.

Reading from another registry

A registry other than whep_inputs is authoritative for its aliases. The file is downloaded from that registry's board, cached, and read back from the cache when the board is unreachable, exactly as for whep_inputs. Two behaviours belong to whep_inputs alone, so with another registry the example board bundled with this package is not consulted, so an alias such as read_example in another registry resolves to that registry's board. The predecessor-pipeline warnings below are not raised, since they describe this package's pins.

reg <- whep::whep_registry("inst/extdata/input_registry.csv")
whep::whep_read_file("crop_yields", registry = reg)
whep::whep_list_file_versions("crop_yields", registry = reg)

Frozen predecessor-pipeline references

Four aliases in whep_inputs are not outputs of this package: primary_prod, commodity_balance_sheet, processing_coefs and feed_intake. They are the 2025-07-14 snapshot of the predecessor R-script pipeline this package replaced, kept frozen so inst/scripts/compare_global_whep.R can benchmark whep against it. No package function reads them, their schema is the old one (CamelCase, name-keyed, no polity columns) and they stop in 2021, so reading one warns.

Their numbers are not what current code produces, so they are references, not substitutes. primary_prod (2,443,516 rows, 1961–2021) against a fresh build_primary_production() over the same 2015–2021 window:

The pin has no slaughtered_heads row in any of its years; the build emits 12,195 over 2015–2021. Three item_prod codes are pin-only (378, 773, 1163; 1,332 rows) and three appear in no pin year at all (1051, and the modelled grassland items 3001 and 3002).
317,587 rows against the build's 328,242 for the window, a 3.2% shortfall.
Its 2020–2021 fodder harvested area is carried forward from 2019 — 85.93 Mha in each year, 468 country-item series over 96 areas, equal to 2019 to the last digit — where the build emits no fodder row at all after 2019, because eu-agridb-fodder stops in 2019, faostat-production-old in 2013, and faostat-production carries none of the 16 fodder items.
Even in a year both cover they disagree: 2019 fodder harvested area is 85.93 Mha in the pin against 90.42 Mha from the build.

Build the current series with the matching build_*() or get_*() function instead.

The batch pin on the build path

Two further aliases were published in that same 2025-07-14 batch — crop_residues and bilateral_trade, alongside the four above, between 12:33:43Z and 12:33:50Z. Unlike the four, these two are read on the default build path, which is why no warning is attached to them: every commodity balance build reads them, so a warning at the read is noise rather than information. bilateral_trade has since been regenerated by this package (#1122), so crop_residues is the only one still registered at its 2025-07-14 version.

They did not share a provenance despite sharing a timestamp. Each was established separately (#1054), because the timestamp alone establishes nothing.

`crop_residues` is predecessor output

Read by get_primary_residues(), and from there by build_commodity_balances(). All 475,688 of its Product rows equal the primary_prod pin's tonnes values exactly — no key unmatched on either side, no value differing at a relative tolerance of 1e-6 — so it is a downstream artifact of the same predecessor run, carrying that run's production series into the commodity balance.

Its residue quantities are those production numbers times a residue-to-product ratio that varies by year, and times a recovery rate: the pin holds the residue the predecessor recovered from the field, not the residue produced (#1195). The predecessor's code (Global/R/crop_npp.r, then afsetools::residue_use()) writes each row as

product * kg_residue_kg_product_FM * HI_changes(region, year) * residue_dm_product_dm(region, category) / residue_dm_product_dm("West Europe", category) * recovery_rates(region, category),

with region and category keyed as in residue_recovery.csv. Rebuilt from the predecessor's own coefficient tables, that formula reproduces all 425,934 pin rows whose region it can key, at a relative tolerance of 1e-6. The year variation is HI_changes alone: a harvest-index change factor per region, given at eight anchor years from 1910 to 2000, linear in between and flat after 2000 (East Europe 1.10 in 1962 to 1.00 in 2000), which the predecessor attributes to Krausmann et al. (2013), PNAS 110:10324, Table M2 (assumed, unverified). It is a regional historical adjustment, not a yield response, and after 2000 the ratio of every region and category is constant. That is why biomass_coefs$kg_residue_kg_product_FM alone reproduces only 3,189 of the 472,790 comparable keys. Of the 33,411 keys that carry a residue of exactly 0, 30,620 are cells whose recovery rate is 0 (fodder crops in seven of the eight regions; roots and tubers, cassava, sugar beet and dry beans in West Europe and in North America and Oceania) and 1,919 are crops whose biomass_coefs ratio is 0.

WHEP ships the recovery rate and the regional ratio the predecessor used (both equal in all 160 cells), but not the HI_changes table. So get_primary_residues() undoes the recovery exactly, dividing by the same rate, but cannot recompute the predecessor's residue from production. It then re-keys the regional ratio on Wirsenius's own region membership, and estimates cereal residue (#1448), and that of the non-cereal crops with their own IPCC Table 11.2 line (#1518), from the pin's production and harvested area instead of reading it.

It is not a small input. The pin holds 7.63 Gt of recovered residue for 2010 (Straw 3.60 Gt, Other crop residues 2.49 Gt, Firewood 1.54 Gt) and 327.7 Gt over 1961–2021, which get_primary_residues() turns into 7.64 Gt (Straw 3.92 Gt, Other crop residues 2.82 Gt, Firewood 0.90 Gt) and 366 Gt of residue produced. 3,998 of its 249,095 output rows carry NA polity columns because the pin is name-keyed.

Driving the same residue model off a fresh get_primary_production() would move those numbers. Measured for 2010 at (area_code, item_prod): 1.958 Gt of current production sits on 6,425 keys the pin never sees (809 Mt of it primary crops, including the modelled temporary-grassland item), 64.8 Mt on 296 keys is pin-only, and of the 8,010 shared keys 538 disagree, putting 1.102 Gt — 11.01% of the pin's shared-key mass — more than 1% apart. The largest class is rice, 673.9 Mt in the pin against 463.3 Mt fresh, a ratio of 0.6876: current code puts rice on a milled-equivalent basis while the pin's is paddy, which is a basis difference rather than an error, since straw scales with the field crop. The second is fodder, where the current build is close to twice the pin on every forage and silage item, the pin predating that work. Replacing the pin therefore means choosing a residue model, which is a science decision and not a refresh. For cereals that model is chosen by get_primary_residues(cereal_residue = ), and for the non-cereals with an IPCC line by non_cereal_residue =; both still run on the pin's production.

`bilateral_trade` is built by `build_detailed_trade()`

Read by get_bilateral_trade() and by the live-animal branch of build_commodity_balances(). The registered version, 20261001T135651Z-ca7b3, is build_detailed_trade()'s output verbatim, with its defaults and extend_time = FALSE, run on the faostat-trade-bilateral pin 20260407T095142Z-b3f81 (#1122): 10,038,220 rows over 1986–2021, polity-keyed, carrying 64,450,133,319 tonnes and 87,840,869,640 head.

The 2025-07-14 version it replaces, 20250714T123347Z-2c392, held the same FAOSTAT Detailed Trade Matrix values in the predecessor pipeline's name-keyed schema — against the raw pin over 2010, all 296,642 shared tonnes keys and all 5,102 shared Head keys agree exactly — but without FAOSTAT's 1000 Head rows: 76,141,882 thousand head of live broiler chicken, turkey, duck, rabbit and goose trade over 1986–2021, which PR #1113 recovered in the producer. The regenerated pin carries 7.50x its 11,707,083,640 head and 22.5% less than its 83,147,095,412 tonnes, all of that tonnage difference in CBS items 5001-5003, which no commodity balance carries: item 1293's unbacked tonnage, screened by method_unbacked_quantity (#1023), and three CBS names with no item_cbs_code. On the tonnes that do reach a trade matrix the two versions differ by 0.056%. .clean_bilateral_trade() reads either schema.

Examples

whep_read_file("read_example")
#> ℹ Fetching files for read_example...
#> # A tibble: 1 × 2
#>   col_1                col_2         
#>   <chr>                <chr>         
#> 1 I'm a sample dataset second version
whep_read_file("read_example", type = "parquet", version = "latest")
#> ℹ Fetching files for read_example...
#> # A tibble: 1 × 2
#>   col_1                col_2        
#>   <chr>                <chr>        
#> 1 I'm a sample dataset third version
whep_read_file(
  "read_example",
  type = "csv",
  version = "20250721T152646Z-ce61b"
)
#> ℹ Fetching files for read_example...
#> # A tibble: 1 × 2
#>   col_1                col_2        
#>   <chr>                <chr>        
#> 1 I'm a sample dataset first version