Fieldprint Platform API v5 — Practitioner Notes
Lessons learned from a large-scale sensitivity analysis project
Introduction
These notes document practical lessons learned while building a sensitivity analysis (SA) pipeline that submitted roughly 150,000 payloads across three FPP v5 sub-endpoints. The goal is to give other API data partners and programmers a head start — covering authentication, endpoint structure, payload construction, throttling behavior, error patterns, and response parsing, with concrete R code that shows how each piece fits together.
All code uses the httr2 + jsonlite + tidyverse stack. The patterns translate directly to any other language or HTTP library.
1 Authentication
The API uses Bearer token authentication. The token is passed in an Authorization header on every request. There is no session management or refresh flow — the same static token is reused for all calls.
# Recommended: store in OS keychain, never hardcode
Sys.setenv("FPP_AUTH_TOKEN" = keyring::key_get(service = "ftm_api"))
# Alternative: .env file in project root
# FPP_AUTH_TOKEN="your_token_here"
readRenviron(".env")
# Usage
auth_token <- Sys.getenv("FPP_AUTH_TOKEN")When building requests with httr2:
library(httr2)
req <- request("https://api.fieldtomarket.org/v5/...") |>
req_headers(
"Accept" = "application/json",
"Content-Type" = "application/json",
"Authorization" = paste("Bearer", auth_token)
)Never hardcode tokens in scripts or notebooks. Store secrets in .Renviron, a .env file (git-ignored), or an OS keychain (keyring::key_set()).
2 Endpoints
The API exposes several endpoints under the v5 base URL https://api.fieldtomarket.org/v5/. The complete reference is available at the FTM Endpoints Reference. This document covers only the endpoints used in the SA pipeline. Other notable endpoints not covered here include:
/FieldData/PrefillCropIntervals— generate a single crop interval with management defaults (alternative toPrefillFieldHistorywhen you don’t need full history)/FieldData/SSURGO— retrieve USDA soil survey data for a field geometry/Calculator— combined endpoint returning all eight sustainability metrics plus scaled 0–100 benchmark scores/ScaledBenchmarks— retrieve benchmark scores separately after a/Calculatorcall- 40+ Reference Data GET endpoints (
/crops,/operations,/nutrients, etc.) that return valid IDs for constructing payloads
2.1 PrefillFieldHistory
POST /FieldData/PrefillFieldHistoryTakes a field boundary geometry and optional broad management assumptions like:
| Management | Argument | Possible values |
|---|---|---|
| Tillage regimes | tillage_class_id |
1L (No-Till), 2L (Strip-Till), 4L (Reduced Till), 6L (Conventional tillage) |
| Irrigation | is_irrigated |
TRUE (Irrigated), FALSE (Non-Irrigated) |
| Cover crops | cover_crop |
TRUE (With cover crop), FALSE (Without cover crop) |
Returns the crop history inferred from the Cropland Data Layer (CDL), together with prefilled management activity data for the most recent crop intervals.
This is the starting point of any workflow: query this endpoint first to obtain a valid base payload structure that can then be submitted to the metric endpoints.
2.2 Metric sub-endpoints
POST /calc/GHG
POST /calc/EnergyUse
POST /calc/SoilCarbonEach sub-endpoint accepts the same Calculator payload structure but returns a different sustainability metric in both metric and imperial units. The three metrics are:
| Endpoint | Output metric | Notes |
|---|---|---|
/calc/GHG |
GHG Emissions (kg CO₂eq/kg and lbs CO₂eq/{functional unit}) | Broken down by system boundary and gas species |
/calc/EnergyUse |
Energy Use (MJ/kg and BTU/{functional unit}) | Broken down by upstream/on-farm/post-harvest |
/calc/SoilCarbon |
Soil Carbon (kg CO₂eq/ha/yr and lbs CO₂eq/acre/yr) | Includes year-by-year SOC flux history |
/Calculator
/Calculator is the right default for most integrations: a single call returns all eight sustainability metrics plus scaled 0–100 benchmark scores, along with additional metadata. For this reason it is the endpoint we recommend as the main entry point for QMDPs consuming this API.
For our sensitivity analysis, sub-endpoints were the better fit. We only needed three of the eight metrics, and /Calculator’s response time is dominated by the soil conservation metric, whose model (e.g. wind erosion) is considerably more computationally expensive than GHG, Energy Use, or Soil Carbon on their own. Since our exercise submitted roughly 150,000 payloads, dispatching directly to the three relevant sub-endpoints in parallel avoided paying that overhead on every call.
If you need the full metric set or the benchmark scores, /Calculator remains the simpler and more maintainable choice.
3 Field Boundary Requirements
The PrefillFieldHistory endpoint accepts any valid GeoJSON geometry (Polygon or MultiPolygon). The common errors include malformed polygons, for instance: open rings, self-intersections, or duplicate vertices. The API does not validate geometry correctness and will return a 400 error if the geometry is invalid.
In the case of using CLU data, selecting the right field size strongly affects response quality:
| Size range | Behavior |
|---|---|
| < 10 ha | Might return unreliable or empty crop history |
| 10 – 200 ha | Good coverage — typical of individual farm fields |
| > 200 ha | May include non-agricultural areas or spatially aggregated parcels |
Pre-filter and quality check your field polygons before querying the API to avoid wasting calls on fields that will return no usable data like sliver polygons or non-representative small fields.
3.1 Boundary quality — correct vs. incorrect examples
Beyond geometric validity, the boundary must represent the cultivated area only. The API infers crop history from CDL pixels inside the polygon, so any non-agricultural area included in the boundary dilutes the crop signal and degrades the returned field history. Two common digitization errors are shown below.
Example 1 — Non-crop area included inside the boundary. The incorrect polygon (8.5 ha) was simplified to speed up mapping and includes a wooded area (~0.7 ha) that the correct polygon (7.8 ha) excludes. Shortcuts like this contaminate the CDL sample with forest pixels and degrade the quality of the inferred field history.
Example 2 — Square drawn over a center pivot field. The incorrect polygon is the square parcel outline (62.2 ha); the actual cultivated area is the pivot circle (44.5 ha). The four corners (~18 ha, 28% of the submitted area) are not cropped, which skews both the CDL-derived crop history and plantable_acres.
In both cases the API will accept the geometry without error — the degradation is silent. Check boundaries against recent imagery before submission.
The geometry must be passed as a parsed GeoJSON object (R list), not a raw JSON string:
library(geojsonsf)
library(jsonlite)
# Convert sf geometry to GeoJSON string, then parse into a list
geojson_str <- geojsonsf::sf_geojson(my_sf_object)
geometry_list <- jsonlite::fromJSON(geojson_str)
req_body <- list(
field = list(geometry = geometry_list),
options = list(
is_irrigated = FALSE,
cover_crop = FALSE,
tillage_class_id = 4L # see Section 5
)
)4 PrefillFieldHistory — Request and Response
4.1 Request body
{
"field": {
"geometry": { "type": "Polygon", "coordinates": [...] }
},
"options": {
"is_irrigated": false,
"cover_crop": false,
"tillage_class_id": 4
}
}All options keys are optional. Omitting a key lets the server apply its default. Passing NULL from R (via compact()) omits the key cleanly:
req_body <- list(
field = list(geometry = fromJSON(geojson)),
options = compact(list(
is_irrigated = is_irrigated, # NULL → omitted
cover_crop = cover_crop,
tillage_class_id = tillage_class_id
))
)4.2 Response envelope
The API wraps every response in a {status, response, error} envelope. The actual payload is nested inside response:
{
"status": "success",
"response": {
"field": { ... },
"crop_intervals": [ ... ]
},
"error": null
}response.field carries the field metadata; response.crop_intervals carries the crop history ordered most-recent-first.
4.3 Field object
"field": {
"geometry": {
"type": "Polygon",
"coordinates": [ [ [-91.7556, 34.2274], ... ] ]
},
"plantable_acres": { "value": 50.83, "unit": "acre" },
"state": "AR",
"county": "Jefferson County"
}Note that the geometry is echoed back in the response. plantable_acres is the API’s estimate of the cultivable area within the polygon, which may differ from the total polygon area. Always use this value — not the raw polygon area — as the field size for yield and rate calculations.
The official PrefillFieldHistory documentation notes that “field history returned may not be accurate and should be reviewed” by qualified professionals. Default operation dates are estimates and may produce conflicts at year boundaries — date validation and fixing (see Section 6) is always required before submitting to the metric endpoints. When CDL detects multiple crops within a field polygon, the API selects the dominant crop by acreage.
4.4 Crop interval structure
Each crop interval has a harvest_year, crop_id, tillage_class_id, and an activities list. Below is a real example for Soybeans (2024) with reduced tillage, irrigation, cover crop, commercial fertilizer, and multiple tillage passes:
{
"harvest_year": 2024,
"crop_id": "11101",
"tillage_class_id": "4",
"activities": [
{
"type": "planting_cash",
"date": "2024-05-25",
"inputs": {
"planting_operation_id": "22917",
"seeding_rate": { "value": 167, "unit": "lb / acre" },
"diesel_use": { "value": 0.64, "unit": "gal_diesel / acre" },
"seed_treatment": true,
"inoculant_applied": false
}
},
{
"type": "planting_cover",
"date": "2023-11-15",
"inputs": {
"cover_crop_id": "30770",
"planting_operation_id": "22952",
"diesel_use": { "value": 0.13, "unit": "gal_diesel / acre" },
"termination_date": "2024-05-20",
"termination_operation_id": "23045"
}
},
{
"type": "irrigation",
"date": "2024-10-13",
"inputs": {
"water_sources": [
{
"water_source_id": "1",
"irrigation_volume": { "value": 8.8, "unit": "acre_inch / acre" },
"application_method_id": "2"
}
]
}
},
{
"type": "harvest",
"date": "2024-10-18",
"inputs": {
"harvest_operation_id": "22819",
"yield": { "value": 44, "unit": "bushel / acre" },
"diesel_use": { "value": 1.53, "unit": "gal_diesel / acre" },
"transport_fuel_id": "1"
}
},
{
"type": "tillage",
"date": "2024-05-10",
"inputs": {
"tillage_operation_id": "22975",
"diesel_use": { "value": 0.48, "unit": "gal_diesel / acre" }
}
},
{
"type": "nutrient_commercial",
"date": "2024-05-09",
"inputs": {
"fertilizer_operation_id": "22693",
"diesel_use": { "value": 0.16, "unit": "gal_diesel / acre" },
"fertilizers": [
{
"product_type_id": 21,
"nitrogen": { "value": 21, "unit": "lb / acre" },
"phosphorus": { "value": 119, "unit": "lb / acre" },
"potassium": { "value": 94, "unit": "lb / acre" },
"sulfur": { "value": 15, "unit": "lb / acre" },
"slow_release": false,
"inhibitor": false
}
]
}
}
]
}Activity types returned by the API:
type |
Description |
|---|---|
planting_cash |
Primary cash crop planting |
planting_cover |
Cover crop planting; includes termination_date and termination_operation_id |
tillage |
Tillage pass; multiple passes per interval are common |
harvest |
Harvest event with yield and transport details |
nutrient_commercial |
Commercial fertilizer application (see Section 13.1 for product ID structure) |
irrigation |
Irrigation event; water_sources is always a list, even for a single source |
tillage_class_id at the interval level is a string (e.g. "4"), not an integer. The options request body accepts an integer (4L in R), but the response echoes it back as a string. Cast accordingly when reading responses.
4.5 Crop IDs for FTM-supported crops
This is a table of FTM-supported crops used for the sensitivity analysis. The full list of Fieldprint Platform supported crops can be found here.
| Crop | crop_id |
|---|---|
| Corn (grain) | "10401" |
| Soybeans | "11101" |
| Wheat (winter) | "11303" |
| Cotton | "10501" |
| Peanuts | "10701" |
The API infers crop history from CDL and will return many non-FTM crops (sorghum, sunflowers, vegetables, etc.), fallow or other non-agricultural land use. Filter crop_intervals by the crop_id values you care about before building Calculator payloads.
5 Building a Calculator Payload
The prefill response is not directly usable as a Calculator payload. You must supplement it with several required fields that the prefill omits. The minimal additions are:
5.1 Top-level options
payload$options <- list(
report = list(
grower = NA,
generated_for = "my project name",
generate = TRUE
),
data_status = NA
)5.2 Field details
payload$field$field_name <- "my_field_id"
payload$field$location <- NA
payload$field$name <- NA
payload$field$farm_serial_number <- NA
payload$field$tract_number <- NA
payload$field$field_number <- NA
payload$field$field_detail_inputs <- list(
tile_drainage = list(tile_density_id = "1"),
landuse_change = list(has_landuse_change = FALSE)
)5.3 Additional data (crop interval level)
Each crop interval that will be scored must have additional_data populated:
payload$crop_intervals[[1]]$additional_data <- list(
habitat = list(wildlife_habitat_ids = list()),
pest_management = list(integrated_pest_management_id = "7"),
nutrient_management = list(
management_plan = FALSE,
nutrient_setbacks = FALSE,
precision_application = FALSE,
p_multi_crop_application_id = "0"
)
)5.4 Post-harvest drying inputs
These are required on the harvest activity’s inputs object. Defaults vary by crop:
# Corn (grain), Soybeans, Wheat (winter)
harvest_inputs$crop_dried <- TRUE
harvest_inputs$moisture_removed <- list(value = 5, unit = "%")
harvest_inputs$transport_distance <- list(value = 10, unit = "miles")
harvest_inputs$drying_location_id <- "2"
# Cotton
harvest_inputs$crop_dried <- TRUE
harvest_inputs$ginning_moisture_id <- "2"
harvest_inputs$transport_distance <- list(value = 10, unit = "miles")
# Peanuts
harvest_inputs$crop_dried <- TRUE
harvest_inputs$peanut_moisture <- list(value = 15, unit = "%")
harvest_inputs$transport_distance <- list(value = 10, unit = "miles")5.5 Irrigation inputs
When is_irrigated = TRUE, the irrigation water source entry must carry additional energy details:
payload$crop_intervals[[1]]$activities[[irrig_idx]]$inputs$water_sources[[1]] <- list_modify(
payload$crop_intervals[[1]]$activities[[irrig_idx]]$inputs$water_sources[[1]],
has_irrigation_energy = TRUE,
energy_source_id = "5",
pump_pressure = list(value = 50, unit = "psi"),
pump_depth = list(value = 250, unit = "ft")
)Additionally, irrigated crop intervals require a non_irrigated_yield on the harvest inputs:
harvest_inputs$non_irrigated_yield <- list(
value = harvest_inputs$yield$value * 0.5, # This is a conservative assumption
unit = harvest_inputs$yield$unit
)5.6 Cover crops — is_legume and growth_id
Cover crop intervals (type == "planting_cover") need two extra fields on their inputs:
# is_legume: TRUE/FALSE depending on species
# growth_id: "2" (standard default)
activity$inputs$is_legume <- TRUE # or FALSE
activity$inputs$growth_id <- "2"These are not provided by the prefill and must be looked up from the cover crop species reference table.
5.7 Perennial crops and cover crop compatibility
Perennial crops (e.g., alfalfa) have a different payload structure from annual crops — they carry multiple cutting intervals rather than a single planting–harvest pair, and their planting_cover activity is not meaningful in the same agronomic sense. When building payloads for perennial crop intervals, the planting_cover activity should be removed regardless of the cover_crop flag passed to the prefill:
# Strip planting_cover from perennial crop intervals
payload$crop_intervals <- map(payload$crop_intervals, \(ci) {
if (ci$crop_id %in% perennial_crop_ids) {
modify_at(ci, "activities", \(acts) discard(acts, \(a) a$type == "planting_cover"))
} else {
ci
}
})There is no API-level error for submitting a planting_cover activity on a perennial crop interval — the payload will be accepted but the cover crop will be silently ignored in the calculation. The removal must be done client-side before submission.
6 Date Overlap Issues — fix_rotations()
Prefill responses can contain date conflicts between consecutive crop intervals, particularly when cover crops and tight rotation sequences are combined. Two overlap classes occur:
Within-interval overlap: the cash crop planting date falls before the cover crop termination date in the same interval. The Calculator rejects these payloads.
Between-interval overlap: the earliest planting or tillage event in one interval occurs before the latest harvest or cover crop termination of the chronologically preceding interval.
Both must be resolved before submitting to the metric endpoints. The approach is to shift conflicting dates earlier, working from the most recent interval outward, and bounding shifts by the harvest date of the next-older interval to avoid cascading conflicts.
fix_rotations() (see R/fix_rotations.R, full source in Appendix A.1) implements this pass. Run it on every payload right after prefill, and again after any client-side date edit (Section 13.2):
source(here::here("R/fix_rotations.R"))
# payload$crop_intervals as returned by PrefillFieldHistory, ordered
# most-recent-first
payload <- fix_rotations(payload)The function walks crop_intervals from newest to oldest with purrr::reduce(), shifting only the interval(s) that conflict — it leaves already-consistent dates untouched, so it is safe to call unconditionally rather than checking first for overlaps.
7 Serializing the Payload
The payload must be serialized with auto_unbox = TRUE so that scalar R values become JSON scalars (not single-element arrays):
library(jsonlite)
json_body <- toJSON(payload, auto_unbox = TRUE)When using httr2, pass the raw JSON string directly to avoid double-serialization:
req <- request(endpoint_url) |>
req_body_raw(charToRaw(json_body), type = "application/json")
# Alternatively, let httr2 serialize a list:
req <- request(endpoint_url) |>
req_body_json(payload_list, auto_unbox = TRUE, type = "application/json")8 Rate Limiting and Throttling
The API imposes a rate limit. The limit encountered in practice was approximately 30 requests per 60 seconds for metric endpoints. Exceeding it returns HTTP 429 Too Many Requests with a Retry-After header indicating how long to wait.
httr2 handles this transparently with req_throttle() and req_retry():
req <- request(url) |>
# Token-bucket: at most `capacity` requests per `fill_time_s` seconds.
# The bucket is process-shared, so it works correctly with req_perform_parallel().
req_throttle(capacity = 30, fill_time_s = 60) |>
# Retry on transient errors (429, 500, 503); honor Retry-After when present.
req_retry(
max_tries = 3,
is_transient = \(resp) resp_status(resp) %in% c(429L, 500L, 503L),
after = \(resp) {
ra <- resp_retry_after(resp)
if (is.null(ra) || is.na(ra)) NULL else ra
}
)When dispatching requests with req_perform_parallel(), the token-bucket throttle is shared across all concurrent connections within the same R process. This is the correct behavior for async dispatch — unlike spawning separate furrr workers (each worker gets its own independent bucket and collectively they can exceed the global limit).
The PrefillFieldHistory endpoint was found to tolerate a higher rate (~300 req/60 s) in testing, but be conservative until you confirm this with your API team.
9 Parallel Dispatch
For large batches, req_perform_parallel() (async I/O within a single R process) is far more efficient than furrr::future_map() for I/O-bound requests:
# Build a list of httr2 request objects
reqs <- pmap(df, \(payload_json, endpoint, ...) {
request(url_map[[endpoint]]) |>
req_headers("Authorization" = paste("Bearer", token),
"Content-Type" = "application/json") |>
req_body_raw(charToRaw(payload_json), type = "application/json") |>
req_error(is_error = \(resp) FALSE) # catch per-response, not globally
})
# Dispatch up to 50 concurrent connections
raw_resps <- req_perform_parallel(
reqs,
on_error = "continue", # don't abort the batch on one failure
max_active = 50L,
progress = TRUE
)Chunking strategy: When dispatching hundreds of thousands of payloads, chunk into batches of ~200 rows. Parse each batch fully and write responses to disk before moving to the next chunk. This bounds peak memory to chunk_size × avg_response_size rather than loading all responses at once.
chunks <- split(seq_len(nrow(df)), ceiling(seq_len(nrow(df)) / 200))
for (k in seq_along(chunks)) {
idx <- chunks[[k]]
batch <- df[idx, ]
reqs <- ... # build requests for this batch
resps <- req_perform_parallel(reqs, on_error = "continue", max_active = 50)
# Parse and write to disk immediately
walk2(resps, batch$json_out, \(resp, path) {
body <- resp_body_json(resp)
write_json(body, path, pretty = FALSE, auto_unbox = TRUE)
})
rm(resps); gc()
}10 Error Handling — Two Error Channels
The FPP API surfaces errors through two separate channels. For the full error reference, see the Calculator usage notes.
10.1 HTTP-level errors (4xx / 5xx)
Standard HTTP status codes signal transport and authentication failures:
| Status | Meaning |
|---|---|
400 |
Malformed request (bad JSON, missing required field) |
401 / 403 |
Authentication failure |
429 |
Rate limit exceeded |
500 |
Internal server error |
503 |
Service temporarily unavailable |
10.2 Application-level errors (HTTP 200 + errorType in body)
The API also returns a valid HTTP 200 response body that contains an error:
{
"errorType": 400,
"error": "Invalid input: ..."
}or for model-level failures:
{
"errorType": 500,
"error": "Simulation failed"
}You must always inspect the parsed body for errorType, regardless of HTTP status code.
The Calculator endpoint supports a validate_only flag that runs structural validation without executing the environmental models. This is useful for catching payload issues quickly during development without incurring the full computation cost.
{ "options": { "validate_only": true }, "field": { ... }, "crop_intervals": [ ... ] }10.3 Complete response handler
parse_response <- function(raw_resp, scenario_id, output_path) {
# Connection / timeout errors from req_perform_parallel on_error = "continue"
if (inherits(raw_resp, "error")) {
return(list(status = "error", error = conditionMessage(raw_resp)))
}
# HTTP errors
if (resp_status(raw_resp) >= 400L) {
msg <- tryCatch(
paste0("HTTP ", resp_status(raw_resp), ": ", resp_body_json(raw_resp)$detail),
error = \(e) paste0("HTTP ", resp_status(raw_resp))
)
return(list(status = "error", error = msg))
}
# Parse body
body <- tryCatch(resp_body_json(raw_resp), error = \(e) NULL)
if (is.null(body)) {
return(list(status = "error", error = "Failed to parse response body"))
}
# Application-level error (HTTP 200 but errorType present)
if (!is.null(body$errorType)) {
error_msg <- body$error %||% paste0("API error (errorType: ", body$errorType, ")")
# Save to a failed/ sub-directory for later inspection
failed_path <- file.path(dirname(output_path), "failed", basename(output_path))
dir.create(dirname(failed_path), showWarnings = FALSE, recursive = TRUE)
write_json(body, failed_path, auto_unbox = TRUE)
return(list(status = "error", error = error_msg))
}
# Success
write_json(body, output_path, pretty = FALSE, auto_unbox = TRUE)
list(status = "success", response = body, error = NA_character_)
}11 Response Structure — Metric Endpoints
11.1 GHG Emissions response
The GHG response is organized by system boundary → source category → source detail. Each node carries a greenhouseGasMetric: {value, unit} leaf and a gases object with per-gas breakdowns:
{
"upstream": {
"greenhouseGasMetric": { "value": 120.5, "unit": "kg CO2e / ha" },
"gases": {
"co2Fossil": { "value": 80.0, "unit": "kg CO2e / ha" },
"ch4Fossil": { "value": 10.0, "unit": "kg CO2e / ha" },
"n2o": { "value": 30.5, "unit": "kg CO2e / ha" }
},
"electricityProd": { ... },
"fuelProd": { ... }
},
"onFarmMechanical": { ... },
"postHarvest": { ... },
"onFarmSourcesSinks": { ... }
}Gas names used by the API (camelCase): co2Fossil, co2Biogenic, ch4Fossil, ch4Biogenic, n2o, nf3, sf6.
11.2 Energy Use response
Same hierarchical structure. On-farm and post-harvest detail nodes have a different shape from upstream nodes:
// Upstream EU detail: flat leaf
{ "value": 5.2, "unit": "MJ / ha" }
// On-farm / post-harvest EU detail: has fuel and electricity children
{
"energyMetric": { "value": 12.0, "unit": "MJ / ha" },
"fuel": { "value": 8.0, "unit": "MJ / ha" },
"electricity": { "value": 4.0, "unit": "MJ / ha" }
}11.3 Soil Carbon response
Returns a scalar fieldprint value and a time series of annual SOC fluxes:
{
"fieldprint": { "value": -0.42, "unit": "kg CO2e / ha / yr" },
"annual_fluxes": [
{ "year": 2014, "value": -1.2, "unit": "kg CO2e / ha / yr" },
{ "year": 2015, "value": -0.9, "unit": "kg CO2e / ha / yr" }
]
}The SOC endpoint is the most prone to server-side errors (HTTP 500 / model failures). Query it last after verifying EU and GHG succeeded for the same payloads, since the same payload issue will typically affect all three endpoints.
11.4 N₂O special case
The API stores direct and indirect N₂O emissions as raw nitrogen inputs (kg N from synthetic fertilizer, organic fertilizer, crop residue, etc.) that must be summed and converted to CO₂e using the appropriate GWP. These are not pre-aggregated in the response and require a post-processing step if you need to attribute N₂O by source.
11.5 Extracting component-level detail — extract_eu_ghg_details()
Rather than hand-walking the nested GHG/EU JSON shown above, R/extract_eu_ghg_details.R (full source in Appendix A.2) provides a path-table-driven extractor that flattens both responses into a single tidy tibble — one row per system boundary × source category × source detail × basis (Field, Functional unit, Area):
source(here::here("R/extract_eu_ghg_details.R"))
ghg_details <- extract_eu_ghg_details(
ghg_resp = ghg_response_body, # parsed body from /calc/GHG
eu_resp = eu_response_body, # parsed body from /calc/EnergyUse
gwp_ref = gwp_lookup, # tibble: ghg, gwp
total_production = harvest_yield_bu,
total_production_unit = "bu",
field_area = field_area_ha,
use_metric_units = TRUE,
raw_ghg = FALSE # convert gas masses to CO2e via gwp_ref
)
ghg_details |>
dplyr::filter(basis == "Functional unit", system_boundary == "Upstream") |>
dplyr::select(source_category, source_detail, value, unit)The function is schema-driven: traversal paths into the JSON are read once from R/eu_ghg_path_schema.csv rather than hardcoded, so adding a new node the API exposes is a CSV edit, not a code change. N₂O detail rows and landuseChange carbon-stock columns are handled as special cases (see Section 11.4) and appended after the schema-driven pass.
12 Idempotent Resumption Pattern
For large-scale batch jobs, implement an idempotent resume pattern so interrupted runs restart cleanly without re-querying already-successful rows:
- Track progress in a database (we used DuckDB). Write a timestamp column (
dispatched_at) only after a successful API response and successful disk write. - Cache responses to disk as JSON files with deterministic names derived from identifiers.
- At startup, reconcile the DB against files actually on disk: if a file exists but the DB row has no timestamp, backfill the timestamp so the row is skipped.
- On retry, filter to rows with
is.na(dispatched_at)before building requests.
# Deterministic filename
json_out <- file.path(
payloads_dir,
glue("{field_id} - {metric} - {crop} - {year} - {trt_code}.json")
)
# Skip if already on disk
if (file.exists(json_out)) {
return(read_json(json_out))
}This pattern makes it safe to kill and restart the process at any point — it will pick up exactly where it left off.
13 Modifying Payload Activities
When you need to test alternative management scenarios, modify individual activities in the crop_intervals[[1]]$activities list rather than rebuilding the payload from scratch. Key patterns:
library(purrr)
# Find an activity by type
idx <- detect_index(
payload$crop_intervals[[1]]$activities,
\(x) x$type == "nutrient_commercial"
)
# Modify a nested field
payload <- modify_in(
payload,
list("crop_intervals", 1, "activities", idx, "inputs", "fertilizers", 1, "slow_release"),
\(x) TRUE
)
# Remove an activity entirely (e.g., when fertilizer rate = 0)
payload$crop_intervals[[1]]$activities <- discard(
payload$crop_intervals[[1]]$activities,
\(a) a$type == "nutrient_commercial"
)13.1 Fertilizer product type IDs
Product type IDs 21–24 have a different input structure (separate N/P/K/S rate fields) compared to all other products (single product_amount field):
product_type_id |
Nutrient | Rate fields |
|---|---|---|
21 |
K₂O | potassium: {value, unit} |
22 |
Sulfur | sulfur: {value, unit} |
23 |
N fertilizer | nitrogen: {value, unit} |
24 |
P fertilizer | phosphorus: {value, unit} |
| Other | Product blend | product_amount: {value, unit} |
13.2 Always re-run fix_rotations() after date changes
Any modification that shifts a planting, tillage, harvest, or cover crop termination date must be followed by a rotation-fixing pass to ensure no interval overlaps are introduced. This is especially important when testing planting date offsets or cover crop termination month variants.
14 Performance Summary
Observed timings from the full SA run (≈150K payloads across 3 endpoints):
| Task | Approx time |
|---|---|
PrefillFieldHistory — 30 req/min, 10 concurrent |
~2 min / 1000 fields |
Payload build + fix_rotations() — 4 CPU workers |
~3200 s / 100 K payloads |
| Metric endpoints — 50 concurrent async connections | ~1 s / 100 payloads |
The metric endpoints are fast and I/O-bound — async parallel dispatch with 50 concurrent connections is the right approach. Payload building is CPU-bound — parallelize with furrr::future_pmap() across multiple cores, processing in chunks to bound RAM.
15 Order of Analysis for Multi-Year Submissions
The API always generates a Fieldprint Analysis for the most recent crop year in the request body. Prior crop intervals provide rotation context for the environmental models (particularly the SWAT+ soil carbon model) but do not themselves receive metric outputs. See the Order of Analysis documentation for the official guidance.
To obtain results for multiple harvest years for the same field, you must submit a separate request per target year, each including the full crop history up to that year:
Request 1 → crop_intervals 2008–2022 → produces metrics for 2022
Request 2 → crop_intervals 2008–2023 → produces metrics for 2023
Request 3 → crop_intervals 2008–2024 → produces metrics for 2024Submit in chronological order — oldest target year first. The server caches intermediate model state (soil carbon spin-up, rotation history) from earlier submissions, so later requests in the sequence execute faster. Submitting in reverse order forfeits this benefit and significantly increases total compute time for large field sets.
For prior intervals that are not the analysis target, the minimum required fields are: crop type, yield, nitrogen application, tillage class, cover crop flag, and irrigation flag. Full activity detail is only needed for the interval being analyzed.
Quick Reference
| Item | Value / Pattern |
|---|---|
| Base URL | https://api.fieldtomarket.org/v5/ |
| Auth | Authorization: Bearer <token> |
| Content-Type | application/json |
| Rate limit | ~30 req / 60 s (metric); ~300 req / 60 s (prefill, unconfirmed) |
| Retry statuses | 429, 500, 503 |
| Error channel 1 | HTTP 4xx / 5xx |
| Error channel 2 | HTTP 200 + errorType in body |
| Field size filter | 10–200 ha |
| Coordinate system | WGS 84 (EPSG:4326) |
| Max parallel | 50 concurrent (metric), 10 (prefill) |
| Most fragile endpoint | /calc/SoilCarbon |
| Fastest sub-endpoint | /calc/EnergyUse |
| Geometry format | GeoJSON object (not string) embedded in request body |
Appendix — Full Source Listings
A.1 fix_rotations()
Full source for the rotation date-fixing pass introduced in Section 6.
#' Fix date overlaps in crop rotation intervals
#'
#' Detects and resolves two classes of date conflicts in a crop history payload:
#' \itemize{
#' \item \strong{Within-interval overlap}: a cash crop planting date falls
#' before the cover crop termination date in the same interval. The cover
#' crop termination date is shifted earlier to precede planting.
#' \item \strong{Between-interval overlap}: the earliest planting or tillage
#' event in interval \emph{i} falls before the latest harvest or cover crop
#' termination in interval \emph{i+1} (the chronologically prior interval).
#' Interval \emph{i+1} is shifted earlier, bounded by the harvest date of
#' interval \emph{i+2} to avoid cascading conflicts.
#' }
#' Intervals are processed sequentially from the most recent to the oldest
#' using \code{\link[purrr]{reduce}}.
#'
#' @param payload A named list representing a single FPP API payload. Must
#' contain a \code{crop_intervals} element — a list of crop interval objects,
#' each with an \code{activities} sub-list. Activities are expected to have at
#' minimum a \code{type} field and a \code{date} field. Cover crop activities
#' (\code{type == "planting_cover"}) must additionally carry
#' \code{inputs$termination_date}.
#'
#' @return The input \code{payload} with \code{crop_intervals} replaced by the
#' conflict-free version. All other top-level payload fields are preserved
#' unchanged.
#'
#' @seealso \code{\link{create_base_payload}} which constructs the payload
#' passed to this function.
#'
#' @examples
#' \dontrun{
#' payload_fixed <- fix_rotations(payload)
#' }
fix_rotations <- function(payload) {
crop_intervals <- payload |> pluck("crop_intervals")
# Build activities tibble for one interval (includes cc_termination row)
build_acts_tbl <- function(activities) {
tbl <- tibble(
idx = seq_along(activities),
type = map_chr(activities, "type"),
date = map_chr(activities, "date") |> as_date()
)
if ("planting_cover" %in% tbl$type) {
cc_idx <- filter(tbl, type == "planting_cover") |> pull(idx)
tbl <- tbl |>
add_row(
idx = cc_idx,
type = "cc_termination",
date = pluck(activities, cc_idx, "inputs", "termination_date") |> as_date()
) |>
arrange(date)
}
tbl
}
check_within_overlap <- function(acts_cache, x) {
tbl <- acts_cache[[x]]
plt_rows <- filter(tbl, type == "planting_cash")
term_rows <- filter(tbl, type == "cc_termination")
till_rows <- filter(tbl, type == "tillage")
if (nrow(plt_rows) == 0 || nrow(term_rows) == 0) return(0)
plt_date <- slice_min(plt_rows, date, n = 1) |> pull(date)
term_date <- pull(term_rows, date)
till_date <- if (nrow(till_rows) > 0) {
slice_min(till_rows, date, n = 1) |> pull(date)
} else {
NA_Date_
}
delta <- as.integer(min(term_date, till_date, na.rm = TRUE) - plt_date)
if (delta > 0) delta else 0
}
check_between_overlap <- function(acts_cache, x) {
cur_tbl <- acts_cache[[x]]
cash_rows <- filter(cur_tbl, type == "planting_cash")
till_rows <- filter(cur_tbl, type == "tillage")
cash_date <- if (nrow(cash_rows) > 0) slice_min(cash_rows, date, n = 1) |> pull(date) else NA_Date_
till_date <- if (nrow(till_rows) > 0) slice_min(till_rows, date, n = 1) |> pull(date) else NA_Date_
cur_date <- min(cash_date, till_date, na.rm = TRUE)
if (is.infinite(cur_date)) return(0)
prev_tbl <- acts_cache[[x + 1]]
harv_rows <- filter(prev_tbl, type == "harvest")
term_rows <- filter(prev_tbl, type == "cc_termination")
harv_date <- if (nrow(harv_rows) > 0) slice_max(harv_rows, date, n = 1) |> pull(date) else NA_Date_
term_date <- if (nrow(term_rows) > 0) pull(term_rows, date) else NA_Date_
prev_date <- max(harv_date, term_date, na.rm = TRUE)
if (is.infinite(prev_date)) return(0)
delta <- as.integer(prev_date - cur_date)
if (delta > 0) delta else 0
}
shift_b_interval <- function(crop_interval, delta) {
crop_interval$activities <- map(
crop_interval$activities,
\(act) {
if (act$type == "planting_cover" && !is.null(act$inputs$termination_date)) {
act$inputs$termination_date <- as.character(
as_date(act$inputs$termination_date) - delta
)
} else {
act$date <- as.character(as_date(act$date) - delta)
}
act
}
)
crop_interval
}
shift_w_interval <- function(crop_interval, delta) {
crop_interval$activities <- map(
crop_interval$activities,
\(act) {
if (act$type == "planting_cover" && !is.null(act$inputs$termination_date)) {
act$inputs$termination_date <- as.character(
as_date(act$inputs$termination_date) - delta
)
}
act
}
)
crop_interval
}
fixed_intervals <- reduce(
seq_len(length(crop_intervals) - 1L),
\(state, i) {
intervals <- state$intervals
acts_cache <- state$acts_cache
# Fix within overlap
delta_w <- check_within_overlap(acts_cache, i)
if (delta_w > 0) {
intervals[[i]] <- shift_w_interval(intervals[[i]], delta_w + 1L)
acts_cache[[i]] <- build_acts_tbl(intervals[[i]]$activities) # invalidate cache
}
# Fix between overlap
# Always shift by delta_b + 1 — do not cap to avoid creating a conflict
# with i+2. Any downstream conflict introduced at i+1 vs i+2 will be
# caught and resolved in the next reduce iteration.
delta_b <- check_between_overlap(acts_cache, i)
if (delta_b > 0) {
intervals[[i + 1L]] <- shift_b_interval(intervals[[i + 1L]], delta_b + 1L)
acts_cache[[i + 1L]] <- build_acts_tbl(intervals[[i + 1L]]$activities) # invalidate cache
}
list(intervals = intervals, acts_cache = acts_cache)
},
.init = list(
intervals = crop_intervals,
acts_cache = map(crop_intervals, \(ci) build_acts_tbl(ci$activities))
)
)$intervals
payload$crop_intervals <- fixed_intervals
return(payload)
}A.2 extract_eu_ghg_details()
Full source for the path-table-driven GHG/EU extractor introduced in Section 11.5, including the schema construction, JSON-navigation helpers, and special-case handlers (N₂O, land-use change, GWP conversion, cotton lint adjustment).
# =============================================================================
# extract_eu_ghg_details_v2.R
# =============================================================================
#
# JSON-first, path-table-driven replacement for extract_eu_ghg_details.R.
#
# DESIGN PHILOSOPHY
# -----------------
# The original v1 function (extract_eu_ghg_details.R) follows a tidyverse
# pattern: it coerces the parsed JSON list into a tibble early and then pipes
# it through dplyr/tidyr verbs. That works well when the JSON structure maps
# cleanly onto a rectangular shape, but the FPP API response is a deeply nested
# list with heterogeneous node types, so the conversion step is fragile and
# expensive.
#
# v2 keeps the response as a raw R list (as returned by jsonlite::read_json or
# httr2::resp_body_json) and navigates it directly with purrr::pluck. Rows are
# built one at a time as tibble::tibble_row() calls and assembled at the end
# with dplyr::bind_rows(). No intermediate wide-to-long pivots or unnesting.
#
# PATH-TABLE SCHEMA (eu_ghg_path_schema.csv)
# ------------------------------------------
# All traversal logic is encoded in a CSV read once at package load time. Each
# row describes one node to extract. Columns:
#
# summary_level — "metric", "boundary", "category", or "detail"
# system_boundary — camelCase API key (e.g. "upstream"), NA at metric level
# source_category — camelCase API key (e.g. "electricityProd"), NA above
# source_detail — camelCase API key (e.g. "cropDrying"), NA above detail
# metric_col — sub-key to descend into after navigating to the parent
# node, in order to reach the {value, unit} leaf.
# NA when the parent node IS the leaf (e.g. upstream EU
# detail nodes are flat {value, unit} with no wrapper key).
# For metric-level rows this is the root key; make_node_path
# uses it to build the path and then blanks it out so
# pluck_metric does not try to descend further.
# is_special — TRUE to exclude a row from schema-driven traversal.
# Used for nodes whose value must be computed (n2o detail
# rows) rather than plucked directly from the JSON.
# The is_special rows are handled by dedicated helpers
# (process_n2o) whose output is appended after the loop.
# has_gases — TRUE when the node has a "gases" child with per-gas
# breakdowns (co2Fossil, ch4Fossil, n2o, …).
# has_energy — TRUE when the node has "fuel" and "electricity" children
# (on-farm/post-harvest EU detail nodes only).
#
# NODE STRUCTURE NOTES
# --------------------
# Not all detail nodes have the same shape:
#
# Upstream EU detail (electricityProd/fuelProd components):
# { value, unit } — flat leaf; metric_col = NA
#
# On-farm / post-harvest EU detail (mobile/stationary components):
# { energyMetric: {value, unit}, fuel: {…}, electricity: {…} }
# metric_col = "energyMetric" to reach the main value;
# has_energy = TRUE so pluck_energy() also extracts fuel/electricity.
#
# GHG nodes (all levels):
# { greenhouseGasMetric: {value, unit}, gases: { co2Fossil: …, … } }
# metric_col = "greenhouseGasMetric"; has_gases = TRUE.
#
# EXTRA COLUMNS PATTERN
# ---------------------
# Some nodes carry supplemental data that does not follow the standard
# {value, unit} leaf structure but should appear as extra columns on the same
# row rather than as separate rows:
#
# N2O detail rows: the API stores direct/indirect N emissions as raw kg-N
# inputs (syntheticFert, organicFert, …) that are summed and converted to
# CO2e. These raw values are retained as extra columns on the detail row.
#
# landuseChange rows: the API stores previousCarbonStock and currentCarbonStock
# as area-normalised values (kg_co2_c / ha). They are added as extra columns
# and are NOT scaled during basis expansion because they are already per-area.
#
# =============================================================================
# -----------------------------------------------------------------------------
# Package-level constants
# -----------------------------------------------------------------------------
gas_names_camel <- c("co2Fossil", "co2Biogenic", "ch4Fossil", "ch4Biogenic", "n2o", "nf3", "sf6")
gas_names_snake <- c("co2_fossil", "co2_biogenic", "ch4_fossil", "ch4_biogenic", "n2o", "nf3", "sf6")
# Label lookup vectors — camelCase API key → human-readable string.
# coalesce(labels[key], "All") returns "All" for keys not in the table
# (e.g. NA system_boundary at the metric level).
sys_bnd_labels <- c(
"upstream" = "Upstream",
"onFarmMechanical" = "On-Farm Mechanical",
"postHarvest" = "Post-Harvest",
"onFarmSourcesSinks" = "On-Farm Non-Mechanical Sources and Sinks"
)
src_cat_labels <- c(
"electricityProd" = "electricity generation and distribution",
"fuelProd" = "production of fuels",
"inputsTransport" = "transportation of agricultural inputs",
"fertilizers" = "production of fertilizers",
"protectants" = "production of pesticides",
"seed" = "production of seed",
"mobile" = "mobile machinery",
"stationary" = "stationary machinery",
"ch4NonFlooded" = "CH4 flux from non-flooded soils",
"lime" = "CO2 from carbonate lime applications to soils",
"urea" = "CO2 from urea fertilizer applications",
"burning" = "Non-CO2 emissions from biomass burning",
"landuseChange" = "Direct land use change emissions",
"n2o" = "Soil N2O",
"ch4Rice" = "CH4 emissions from flooded rice cultivation",
"carbonStock" = "Soil carbon stock changes"
)
src_det_labels <- c(
"irrigationOps" = "Irrigation Operations",
"cropDrying" = "Crop Drying",
"fieldOps" = "Field Operations",
"manureTransport" = "Manure Transportation",
"cropTransport" = "Crop Transportation",
"direct" = "Direct emissions",
"indirect" = "Indirect emissions"
)
# -----------------------------------------------------------------------------
# Path table construction
# -----------------------------------------------------------------------------
# Build the purrr::pluck() path for one schema row.
#
# For GHG and EU metric-level rows the path IS the metric_col key (e.g.
# "totalGreenhouseMetric"), so metric_col doubles as the first element of the
# path and is then blanked out in the schema mutate below so pluck_metric()
# does not try to descend a second time.
#
# Imperial API nodes drop the trailing "Metric" suffix
# (e.g. "greenhouseGasMetric" → "greenhouseGas"). When use_metric_units = FALSE,
# str_remove strips that suffix before the path is built.
#
# Returns NULL for is_special rows; those rows are excluded from the pmap loop
# in Step 3 and handled by dedicated helpers instead.
make_node_path <- function(
metric_col,
summary_level,
system_boundary,
source_category,
source_detail,
is_special,
use_metric_units = TRUE
) {
if (is_special) return(NULL)
if (!use_metric_units) metric_col <- stringr::str_remove(metric_col, "Metric$")
switch(
summary_level,
"metric" = metric_col,
"boundary" = system_boundary,
"category" = c(system_boundary, source_category),
"detail" = c(system_boundary, source_category, "components", source_detail)
)
}
# Read the schema CSV once at source time. The "metric" column is derived from
# metric_col (greenhouse → "GHG", energy → "EU") and then filled downward
# because detail rows that are flat leaves have metric_col = NA and would
# otherwise lose their family assignment.
eu_ghg_schema <- readr::read_csv(
here::here("R/eu_ghg_path_schema.csv"),
col_types = readr::cols(
summary_level = readr::col_character(),
system_boundary = readr::col_character(),
source_category = readr::col_character(),
source_detail = readr::col_character(),
metric_col = readr::col_character(),
is_special = readr::col_logical(),
has_gases = readr::col_logical(),
has_energy = readr::col_logical()
),
na = "NA",
show_col_types = FALSE
) |>
dplyr::mutate(
metric = dplyr::case_when(
str_detect(metric_col, "[gG]reenhouse") ~ "GHG",
str_detect(metric_col, "[Ee]nergy") ~ "EU"
)
) |>
tidyr::fill(metric)
# Pre-compute the static (metric-unit) path table at load time.
# Not used directly by the exported function — each call rebuilds via
# active_schema to honour the use_metric_units argument — but useful for
# inspection and testing.
full_path_table <- eu_ghg_schema |>
dplyr::mutate(
metric_path = purrr::pmap(
list(
metric_col,
summary_level,
system_boundary,
source_category,
source_detail,
is_special
),
make_node_path
),
# Blank metric_col for metric-level rows: the key was already consumed by
# make_node_path() to build the path; pluck_metric() must NOT descend again.
metric_col = dplyr::if_else(
summary_level == "metric",
NA_character_,
metric_col
),
gases_path = purrr::map2(
metric_path,
has_gases,
~ if (.y && !is.null(.x)) c(.x, "gases") else NULL
),
energy_path = purrr::map2(
metric_path,
has_energy,
~ if (.y && !is.null(.x)) .x else NULL
)
)
# -----------------------------------------------------------------------------
# Helper functions — JSON navigation
# -----------------------------------------------------------------------------
# Navigate to a {value, unit} node and return its contents as a named list.
#
# path — character vector passed to purrr::pluck via !!!as.list()
# metric_col — optional sub-key to descend into after reaching the parent node.
# NA (or NULL) when the path already terminates at the leaf, e.g.
# upstream EU detail nodes are flat {value, unit}.
# Non-NA for nodes with a wrapper key, e.g. metric_col =
# "greenhouseGasMetric" for GHG nodes, "energyMetric" for on-farm
# EU detail nodes.
pluck_metric <- function(resp, path, metric_col = NULL) {
node <- purrr::pluck(resp, !!!as.list(path), .default = NULL)
if (is.null(node)) {
return(list(value = NA_real_, unit = NA_character_))
}
if (!is.null(metric_col) && !is.na(metric_col)) {
node <- node[[metric_col]]
}
list(value = node[["value"]], unit = node[["unit"]])
}
# Extract all 7 gas values from a "gases" child node.
# Gas nodes are sparse — not every node has all gases — so missing keys return
# NA via .default rather than an error.
pluck_gases <- function(resp, gases_path) {
node <- purrr::pluck(resp, !!!as.list(gases_path), .default = NULL)
if (is.null(node)) {
return(purrr::set_names(rep(NA_real_, 7L), gas_names_snake))
}
purrr::map_dbl(
gas_names_camel,
~ purrr::pluck(node, .x, "value", .default = NA_real_)
) |>
purrr::set_names(gas_names_snake)
}
# Extract fuel and electricity sub-components from an on-farm / post-harvest
# EU detail node. These sit at the same level as "energyMetric" (not nested
# inside it), so energy_path points to the detail node itself.
pluck_energy <- function(resp, energy_path) {
node <- purrr::pluck(resp, !!!as.list(energy_path), .default = NULL)
if (is.null(node)) {
return(tibble::tibble(
fuel_value = NA_real_,
fuel_unit = NA_character_,
electricity_value = NA_real_,
electricity_unit = NA_character_
))
}
tibble::tibble(
fuel_value = purrr::pluck(node, "fuel", "value", .default = NA_real_),
fuel_unit = purrr::pluck(node, "fuel", "unit", .default = NA_character_),
electricity_value = purrr::pluck(node, "electricity", "value", .default = NA_real_),
electricity_unit = purrr::pluck(node, "electricity", "unit", .default = NA_character_)
)
}
# -----------------------------------------------------------------------------
# Special-node helpers
# -----------------------------------------------------------------------------
# Produce two detail rows (direct, indirect) for Soil N2O.
#
# Why not schema-driven? The n2o category row IS handled by the schema
# (is_special = FALSE at category level), but the direct/indirect detail nodes
# do not contain a pre-computed {value, unit} leaf — they store raw kg-N inputs
# that must be summed and converted:
#
# N (kg) → N2O (kg) via 44/28 molecular weight ratio
# N2O (kg) → CO2e via GWP
#
# The raw kg-N inputs are retained as extra columns on the direct/indirect rows
# (syntheticFert, organicFert, …, volitized, leached). These values are NOT
# scaled during basis expansion because they are field-total inputs, not
# per-area or per-unit emissions.
process_n2o <- function(resp, use_metric_units, gwp_n2o, raw_ghg = TRUE) {
n2o_node <- purrr::pluck(resp, "onFarmSourcesSinks", "n2o")
mass_conv <- if (use_metric_units) 1 else kg_lb
unit_raw <- if (use_metric_units) "kg" else "lb"
unit_co2e <- if (use_metric_units) "kg_co2e" else "lb_co2e"
# Direct emissions: sum N sources then apply management factors
direct <- n2o_node[["direct"]]
tl_fac <- direct[["tillageFactor"]]
bc_fac <- direct[["biocharFactor"]]
n_kg_direct <- purrr::map_dbl(
c("syntheticFert", "organicFert", "aboveGroundResidue", "belowGroundResidue", "prp"),
~ purrr::pluck(direct, .x, "value", .default = 0)
) |>
sum(na.rm = TRUE)
n2o_kg_direct <- n_kg_direct * (44 / 28) * tl_fac * bc_fac * mass_conv
# Indirect emissions: leaching + volatilization
indirect <- n2o_node[["indirect"]]
n_kg_indirect <- purrr::map_dbl(
c("volitized", "leached"),
~ purrr::pluck(indirect, .x, "value", .default = 0)
) |>
sum(na.rm = TRUE)
n2o_kg_indirect <- n_kg_indirect * (44 / 28) * mass_conv
# Raw input values (kg-N) kept as extra columns on the detail rows
pluck_val <- function(node, key) purrr::pluck(node, key, "value", .default = NA_real_)
make_n2o_row <- function(det, n2o_raw, extra_cols) {
n2o_gas <- if (raw_ghg) n2o_raw else n2o_raw * gwp_n2o
ghg_unit <- if (raw_ghg) unit_raw else unit_co2e
base <- tibble::tibble_row(
summary_level = "detail",
system_boundary = "onFarmSourcesSinks",
source_category = "n2o",
source_detail = det,
metric = "GHG Emissions",
basis = "Field",
value = n2o_raw * gwp_n2o, # main value always CO2e
unit = unit_co2e,
ghg_unit = ghg_unit,
co2_fossil = NA_real_,
co2_biogenic = NA_real_,
ch4_fossil = NA_real_,
ch4_biogenic = NA_real_,
n2o = n2o_gas,
nf3 = NA_real_,
sf6 = NA_real_
)
dplyr::bind_cols(base, extra_cols)
}
direct_extras <- tibble::tibble_row(
syntheticFert = pluck_val(direct, "syntheticFert"),
organicFert = pluck_val(direct, "organicFert"),
aboveGroundResidue = pluck_val(direct, "aboveGroundResidue"),
belowGroundResidue = pluck_val(direct, "belowGroundResidue"),
prp = pluck_val(direct, "prp"),
tillageFactor = purrr::pluck(direct, "tillageFactor", .default = NA_real_),
biocharFactor = purrr::pluck(direct, "biocharFactor", .default = NA_real_),
volitized = NA_real_,
leached = NA_real_
)
indirect_extras <- tibble::tibble_row(
syntheticFert = NA_real_,
organicFert = NA_real_,
aboveGroundResidue = NA_real_,
belowGroundResidue = NA_real_,
prp = NA_real_,
tillageFactor = NA_real_,
biocharFactor = NA_real_,
volitized = pluck_val(indirect, "volitized"),
leached = pluck_val(indirect, "leached")
)
dplyr::bind_rows(
make_n2o_row("direct", n2o_kg_direct, direct_extras),
make_n2o_row("indirect", n2o_kg_indirect, indirect_extras)
)
}
# -----------------------------------------------------------------------------
# Post-processing helpers
# -----------------------------------------------------------------------------
# Multiply each gas column by its GWP factor (in-place on the assembled tibble).
# Only called when raw_ghg = FALSE; gas values are stored as raw kg otherwise.
# Also updates ghg_unit from "kg" → "kg_co2e" (or "lb" → "lb_co2e").
apply_gwp <- function(tbl, gwp_ref) {
gwp_vec <- purrr::set_names(gwp_ref$gwp, gwp_ref$ghg)
gas_cols <- c("co2_fossil", "co2_biogenic", "ch4_fossil", "ch4_biogenic", "n2o", "nf3", "sf6")
present <- intersect(gas_cols, names(tbl))
for (g in present) {
if (!is.na(gwp_vec[g])) tbl[[g]] <- tbl[[g]] * gwp_vec[g]
}
tbl$ghg_unit <- tbl$ghg_unit |>
stringr::str_replace("^kg$", "kg_co2e") |>
stringr::str_replace("^lb$", "lb_co2e")
tbl
}
# Apply the cotton lint factor (0.83) to all gas columns.
# Cotton GHG is reported per kg lint; the API returns per kg seed cotton,
# so all gas values are scaled by the lint fraction.
apply_cotton <- function(tbl) {
gas_cols <- c("co2_fossil", "co2_biogenic", "ch4_fossil", "ch4_biogenic", "n2o", "nf3", "sf6")
present <- intersect(gas_cols, names(tbl))
dplyr::mutate(tbl, dplyr::across(dplyr::all_of(present), ~ .x * 0.83))
}
# -----------------------------------------------------------------------------
# Main exported function
# -----------------------------------------------------------------------------
#' Extract Detailed Energy Use and GHG Emissions Results
#'
#' Extracts component-level breakdowns for Energy Use and GHG Emissions from a
#' raw FPP API response list. Uses a JSON-first, path-table-driven approach:
#' the response is navigated directly with [purrr::pluck()] guided by
#' `eu_ghg_path_schema.csv`, avoiding fragile unnesting of heterogeneous nodes.
#'
#' @param api_resp Full API response list (used when `eu_resp`/`ghg_resp` are
#' not supplied separately). The function plucks `cropyears$energyUse` and
#' `cropyears$greenhouseGas` from it.
#' @param gwp_ref Data frame with columns `ghg` (snake_case gas name) and `gwp`
#' (global warming potential factor). Used to convert raw gas masses to CO2e
#' when `raw_ghg = FALSE`, and to extract the N2O GWP for the N2O detail rows.
#' @param total_production Numeric. Total crop production for the field (in the
#' unit implied by `total_production_unit`). Used to derive the Functional
#' unit basis rows.
#' @param total_production_unit Character. Unit string for `total_production`,
#' e.g. `"bu"`, `"kg / ha"`. The `" / acre"` or `" / ha"` suffix is stripped
#' before appending to output unit strings.
#' @param field_area Numeric. Field area in hectares (metric) or acres
#' (imperial). Used to derive the Area basis rows.
#' @param use_metric_units Logical (default `TRUE`). When `FALSE`, the function
#' targets the imperial API node names (no trailing "Metric" suffix) and
#' applies unit conversions (MJ → BTU, kg → lb, kg/ha → lb/acre).
#' @param crop Character or `NULL`. Pass `"Cotton"` to apply the 0.83 lint
#' fraction to all gas columns via [apply_cotton()].
#' @param raw_ghg Logical (default `TRUE`). When `TRUE`, gas columns store raw
#' mass values (kg or lb per gas). When `FALSE`, [apply_gwp()] is called to
#' convert to CO2-equivalent.
#' @param eu_resp Parsed energy-use response list. Overrides the `cropyears$
#' energyUse` path from `api_resp` when supplied.
#' @param ghg_resp Parsed GHG response list. Overrides the `cropyears$
#' greenhouseGas` path from `api_resp` when supplied.
#' @param ... Ignored. Allows forward-compatible calls with extra arguments.
#'
#' @return A tibble with one row per (summary_level × basis) combination.
#' Columns: `summary_level`, `system_boundary`, `source_category`,
#' `source_detail`, `metric`, `basis`, `value`, `unit`, plus optional columns
#' depending on the metric — `fuel_value`/`fuel_unit`/`electricity_value`/
#' `electricity_unit` (EU detail), `ghg_unit`/`co2_fossil`/…/`sf6` (GHG),
#' `syntheticFert`/…/`leached` (N2O detail), `previousCarbonStock`/
#' `currentCarbonStock`/`unit_carbon_stock` (landuseChange).
#'
#' @export
extract_eu_ghg_details <- function(
api_resp = NULL,
gwp_ref,
total_production,
total_production_unit,
field_area,
use_metric_units = TRUE,
crop = NULL,
raw_ghg = TRUE,
eu_resp = NULL,
ghg_resp = NULL,
...
) {
# Step 1 — Resolve response objects -----------------------------------------
if (is.null(eu_resp)) {
eu_resp <- purrr::pluck(api_resp, "cropyears", "energyUse")
}
if (is.null(ghg_resp)) {
ghg_resp <- purrr::pluck(api_resp, "cropyears", "greenhouseGas")
}
# Step 2 — Build unit-adjusted path tables ----------------------------------
# Rebuilt per call so use_metric_units is honoured. make_node_path() strips
# the "Metric$" suffix from metric_col when use_metric_units = FALSE, making
# the paths target the imperial API node names.
active_schema <- eu_ghg_schema |>
dplyr::mutate(
metric_path = purrr::pmap(
list(metric_col, summary_level, system_boundary, source_category, source_detail, is_special),
make_node_path,
use_metric_units = use_metric_units
),
# Blank metric_col for metric-level rows (key was consumed by
# make_node_path to form the path; pluck_metric must not descend again).
# For imperial calls, strip "Metric$" so pluck_metric uses the right key.
metric_col = dplyr::case_when(
summary_level == "metric" ~ NA_character_,
!use_metric_units ~ stringr::str_remove(metric_col, "Metric$"),
TRUE ~ metric_col
),
gases_path = purrr::map2(
metric_path, has_gases,
~ if (.y && !is.null(.x)) c(.x, "gases") else NULL
),
energy_path = purrr::map2(
metric_path, has_energy,
~ if (.y && !is.null(.x)) .x else NULL
)
)
active_ghg_table <- dplyr::filter(active_schema, metric == "GHG")
active_eu_table <- dplyr::filter(active_schema, metric == "EU")
gwp_n2o <- gwp_ref$gwp[gwp_ref$ghg == "n2o"]
ghg_unit <- dplyr::case_when(
use_metric_units & raw_ghg ~ "kg",
use_metric_units & !raw_ghg ~ "kg_co2e",
!use_metric_units & raw_ghg ~ "lb",
TRUE ~ "lb_co2e"
)
# Step 3 — Traverse GHG path table ------------------------------------------
# Each non-special schema row produces exactly one Field-basis tibble_row.
# The n2o detail rows (is_special = TRUE) are appended separately via
# process_n2o(); the n2o category row is handled here like any other category.
if (!is.null(ghg_resp)) {
regular <- dplyr::filter(active_ghg_table, !is_special)
ghg_rows <- purrr::pmap(
list(
summary_level = regular$summary_level,
system_boundary = regular$system_boundary,
source_category = regular$source_category,
source_detail = regular$source_detail,
metric_path = regular$metric_path,
metric_col = regular$metric_col,
gases_path = regular$gases_path
),
function(
summary_level, system_boundary, source_category, source_detail,
metric_path, metric_col, gases_path
) {
met <- pluck_metric(ghg_resp, metric_path, metric_col)
gases <- if (!is.null(gases_path)) {
pluck_gases(ghg_resp, gases_path)
} else {
purrr::set_names(rep(NA_real_, 7L), gas_names_snake)
}
if (!use_metric_units) gases <- gases * kg_lb
tibble::tibble_row(
summary_level = summary_level,
system_boundary = system_boundary,
source_category = source_category,
source_detail = source_detail,
metric = "GHG Emissions",
basis = "Field",
value = met$value,
unit = met$unit,
ghg_unit = ghg_unit,
co2_fossil = gases["co2_fossil"],
co2_biogenic = gases["co2_biogenic"],
ch4_fossil = gases["ch4_fossil"],
ch4_biogenic = gases["ch4_biogenic"],
n2o = gases["n2o"],
nf3 = gases["nf3"],
sf6 = gases["sf6"]
)
}
)
# Append n2o detail rows (direct/indirect with raw kg-N input columns)
ghg_rows <- c(
ghg_rows,
list(process_n2o(ghg_resp, use_metric_units, gwp_n2o, raw_ghg))
)
# Add carbon stock extra columns to landuseChange rows.
# previousCarbonStock / currentCarbonStock are already area-normalised
# (kg_co2_c / ha) and must NOT be divided during basis expansion.
luc_node <- purrr::pluck(ghg_resp, "onFarmSourcesSinks", "landuseChange")
luc_prev <- purrr::pluck(luc_node, "previousCarbonStock", "value", .default = NA_real_)
luc_curr <- purrr::pluck(luc_node, "currentCarbonStock", "value", .default = NA_real_)
luc_unit <- purrr::pluck(luc_node, "previousCarbonStock", "unit", .default = NA_character_)
if (!use_metric_units) {
luc_prev <- luc_prev * kg_lb / ha_ac
luc_curr <- luc_curr * kg_lb / ha_ac
luc_unit <- "lbs_co2_c / acre"
}
res_GHG <- dplyr::bind_rows(ghg_rows) |>
dplyr::mutate(
previousCarbonStock = dplyr::if_else(source_category == "landuseChange", luc_prev, NA_real_),
currentCarbonStock = dplyr::if_else(source_category == "landuseChange", luc_curr, NA_real_),
unit_carbon_stock = dplyr::if_else(source_category == "landuseChange", luc_unit, NA_character_)
)
if (!raw_ghg) res_GHG <- apply_gwp(res_GHG, gwp_ref)
if (identical(crop, "Cotton")) res_GHG <- apply_cotton(res_GHG)
} else {
res_GHG <- tibble::tibble()
}
# Step 4 — Traverse EU path table -------------------------------------------
# EU rows have no gas columns; on-farm/post-harvest detail rows additionally
# get fuel and electricity sub-components via pluck_energy().
if (!is.null(eu_resp)) {
eu_rows <- purrr::pmap(
list(
summary_level = active_eu_table$summary_level,
system_boundary = active_eu_table$system_boundary,
source_category = active_eu_table$source_category,
source_detail = active_eu_table$source_detail,
metric_path = active_eu_table$metric_path,
metric_col = active_eu_table$metric_col,
energy_path = active_eu_table$energy_path
),
function(
summary_level, system_boundary, source_category, source_detail,
metric_path, metric_col, energy_path
) {
met <- pluck_metric(eu_resp, metric_path, metric_col)
val <- met$value
unit <- met$unit
if (!use_metric_units) {
val <- val * mj_btu
unit <- stringr::str_replace(unit, "MJ", "btu")
}
base_row <- tibble::tibble_row(
summary_level = summary_level,
system_boundary = system_boundary,
source_category = source_category,
source_detail = source_detail,
metric = "Energy Use",
basis = "Field",
value = val,
unit = unit
)
if (!is.null(energy_path)) {
base_row <- dplyr::bind_cols(base_row, pluck_energy(eu_resp, energy_path))
}
base_row
}
)
res_EU <- dplyr::bind_rows(eu_rows)
} else {
res_EU <- tibble::tibble()
}
# Step 5 — Combine and apply label lookups ----------------------------------
all_rows <- dplyr::bind_rows(res_EU, res_GHG) |>
dplyr::mutate(
system_boundary = dplyr::coalesce(sys_bnd_labels[system_boundary], "All"),
source_category = dplyr::coalesce(src_cat_labels[source_category], "All"),
source_detail = dplyr::coalesce(src_det_labels[source_detail], "All"),
# For mechanical/upstream boundaries, prepend the metric family to the
# source_category label (e.g. "mobile machinery" → "GHG emissions
# associated with mobile machinery"). On-Farm Non-Mechanical and All
# keep their raw labels.
source_category = dplyr::if_else(
!system_boundary %in% c("On-Farm Non-Mechanical Sources and Sinks", "All") &
source_category != "All",
stringr::str_glue("{metric} associated with {source_category}") |>
stringr::str_replace("Energy Use", "Energy use") |>
stringr::str_replace("GHG Emissions", "GHG emissions"),
source_category
)
)
# Step 6 — Basis expansion --------------------------------------------------
# Only Field-basis rows are expanded; any row already on a non-Field basis
# (none expected in normal flow) is passed through unchanged.
# gas columns are scaled proportionally; extra columns (N2O raw inputs,
# carbon stock) are NOT in ghg_gas_cols and are therefore carried over
# unchanged to all three basis rows.
res_field <- dplyr::filter(all_rows, basis == "Field")
res_special <- dplyr::filter(all_rows, basis != "Field")
ghg_gas_cols <- c("co2_fossil", "co2_biogenic", "ch4_fossil", "ch4_biogenic", "n2o", "nf3", "sf6")
prod_unit <- stringr::str_remove(total_production_unit, " / acre| / ha")
area_label <- if (use_metric_units) "ha" else "acre"
res_prod <- res_field |>
dplyr::mutate(
value = value / total_production,
basis = "Functional unit",
unit = stringr::str_glue("{unit} / {prod_unit}"),
dplyr::across(dplyr::any_of(ghg_gas_cols), ~ .x / total_production)
)
res_area <- res_field |>
dplyr::mutate(
value = value / field_area,
basis = "Area",
unit = stringr::str_glue("{unit} / {area_label}"),
dplyr::across(dplyr::any_of(ghg_gas_cols), ~ .x / field_area)
)
# Step 7 — Final bind and sort ----------------------------------------------
dplyr::bind_rows(res_prod, res_field, res_area, res_special) |>
dplyr::arrange(metric, basis, system_boundary, source_category, source_detail)
}