Weather & Agronomic Data Integration for Field Models

Vegetation indices tell you a field is behind; weather tells you whether that is a problem. A canopy that looks thin on 12 June is normal if the field has accumulated 380 growing degree days and alarming if it has accumulated 620, and no amount of spectral processing will distinguish the two. The output of this topic is a per-field-season daily table of agronomic covariates — accumulated thermal time, rainfall, reference evapotranspiration and a simple water balance — computed from a gridded weather product and joined on the same field identity used everywhere else in farm data platform engineering.

It is the cheapest data in the platform and the most under-used. It arrives on a schedule, needs no authorisation per grower, and turns a set of index maps into something a model can reason about.

Prerequisites

  • Python 3.11+, xarray 2024., rioxarray 0.15., rasterstats 0.19.* or exactextract 0.2., geopandas 1.0., numpy 1.26., pandas 2.2.
  • A gridded weather source: hourly or daily temperature, precipitation and reference evapotranspiration, in NetCDF or Zarr
  • Field boundaries and planting dates from PostGIS schema design for farm data — thermal time accumulates from planting, not from 1 January
  • A metre-based CRS per field for area weighting, chosen as in understanding CRS in precision agriculture
  • Optionally an on-farm rain gauge or nearby station record for validation

1. Concept: Coarse in Space, Dense in Time

Weather is the mirror image of imagery. A satellite gives 10 m pixels on 8–15 usable days a season; a weather grid gives 1–10 km cells every single day. That asymmetry drives every design choice here: there is no point resampling a 4 km grid to field resolution — the value is not spatial detail but an uninterrupted daily series.

The aggregation is therefore simple and the accumulation is where the agronomy lives. Growing degree days convert daily temperature into thermal time, the clock crops actually run on. Rainfall accumulation and reference evapotranspiration combine into a running water balance that explains most within-season variation that indices show but cannot date.

Area weighting matters more than it looks

A 60-hectare field inside a 4 km grid cell takes that cell’s value entire. A field straddling four cells does not: the correct value is the area-weighted mean of the overlapping cells, and taking the value of the cell containing the centroid instead introduces a step change whenever a field’s centroid crosses a cell boundary — including when a boundary is edited. Use exact area weighting, not centroid sampling, and the field’s value changes smoothly with its geometry.

Area-weighted aggregation versus centroid sampling On the left, a field polygon straddles four weather grid cells; the share of field area falling in each cell is labelled, and the weighted mean temperature is computed from those shares. On the right, centroid sampling takes the value of the single cell containing the field centroid, producing a different and less stable answer. Area-weighted mean over overlapping cells 18.2 °C 18.9 °C 17.8 °C 18.5 °C 14% 39% 11% 36% weighted mean = 18.55 °C — moves smoothly when the boundary is edited Centroid sampling 18.2 °C 18.9 °C 17.8 °C 18.5 °C centroid takes 18.9 °C entire — jumps to 18.5 °C if the boundary shifts 200 m south

2. Step-by-Step Implementation

From a gridded product to one row per field-day Five stages: open the gridded store lazily, subset it to the farm's bounding box with a pad, compute an exact area-weighted mean over each field boundary, accumulate thermal time from the planting date, and run rainfall and reference evapotranspiration through a single-bucket water balance. Open lazily chunked, no read yet Subset farm bounds plus a pad Zonal mean exact area weights Accumulate thermal time from planting Balance rain minus reference ET Only the subset step reads bytes; everything before it is metadata, which is what keeps a decade of history cheap to backfill.

Step 1 — Open the grid lazily and clip to the farm

PYTHON 
import xarray as xr
import geopandas as gpd

def open_weather(store: str, fields: gpd.GeoDataFrame, pad_deg: float = 0.2) -> xr.Dataset:
    """Open a gridded product and subset to the farm's bounding box plus a pad."""
    ds = xr.open_zarr(store, chunks={"time": 90})       # lazy: nothing read yet
    minx, miny, maxx, maxy = fields.to_crs(4326).total_bounds
    ds = ds.sel(
        longitude=slice(minx - pad_deg, maxx + pad_deg),
        latitude=slice(maxy + pad_deg, miny - pad_deg),  # note: many grids run north→south
    )
    assert ds.sizes["latitude"] > 0 and ds.sizes["longitude"] > 0, (
        "empty subset — latitude is probably ascending in this product; flip the slice")
    return ds

The assertion earns its place immediately: roughly half of gridded products store latitude descending and half ascending, and a slice in the wrong order returns an empty selection rather than an error. Every downstream aggregate then quietly produces NaN.

Step 2 — Area-weighted zonal statistics per field

PYTHON 
import numpy as np
import pandas as pd
from exactextract import exact_extract

def field_daily_series(ds: xr.Dataset, fields: gpd.GeoDataFrame, var: str) -> pd.DataFrame:
    """One row per field per day: the area-weighted mean of `var` over the boundary."""
    da = ds[var]
    frames = []
    for day in da["time"].values:
        layer = da.sel(time=day).rio.write_crs("EPSG:4326")
        stats = exact_extract(layer, fields, ["mean"], output="pandas")
        frames.append(pd.DataFrame({
            "field_id": fields["field_id"].to_numpy(),
            "date": pd.Timestamp(day).date(),
            var: stats["mean"].to_numpy(),
        }))
    out = pd.concat(frames, ignore_index=True)
    assert out[var].notna().all(), (
        f"{out[var].isna().sum()} field-days with no {var} — a field lies outside the grid")
    return out

exact_extract computes the fractional area of every cell inside the polygon, which is exactly the weighting described above. Where it is unavailable, rasterstats.zonal_stats(..., all_touched=False) with a finer resampled grid is an acceptable approximation; centroid sampling is not.

Step 3 — Accumulate growing degree days

PYTHON 
CROP_BASE_C = {"maize": 10.0, "soybean": 10.0, "wheat": 0.0, "canola": 0.0, "sunflower": 7.0}
CROP_CAP_C = {"maize": 30.0}          # daily maximum capped for maize; others uncapped


def growing_degree_days(tmin_c: np.ndarray, tmax_c: np.ndarray, crop: str) -> np.ndarray:
    """Daily thermal time by the modified average method, in degree-days Celsius."""
    base = CROP_BASE_C[crop]                      # KeyError is the correct behaviour here
    cap = CROP_CAP_C.get(crop)
    tmax = np.minimum(tmax_c, cap) if cap is not None else tmax_c
    tmin = np.maximum(tmin_c, base)               # below base contributes nothing
    tmax = np.maximum(tmax, base)
    gdd = (tmin + tmax) / 2.0 - base
    return np.clip(gdd, 0.0, None)


def accumulate_from_planting(daily: pd.DataFrame, planted_on) -> pd.DataFrame:
    """Cumulative GDD from the planting date — never from 1 January."""
    d = daily[daily["date"] >= planted_on].sort_values("date").copy()
    d["gdd_cum"] = d["gdd"].cumsum()
    return d

The KeyError on an unknown crop is deliberate. A default base temperature is the single most damaging convenience in this whole area: silently applying 10 °C to winter wheat understates thermal time so badly that a model trained on it learns the crop’s calendar rather than its physiology.

Step 4 — Rainfall, reference evapotranspiration and a running balance

PYTHON 
def water_balance(daily: pd.DataFrame, awc_mm: float = 150.0, kc: float = 1.0) -> pd.DataFrame:
    """A single-bucket soil water balance in millimetres. Crude, and useful."""
    d = daily.sort_values("date").copy()
    store, series = awc_mm * 0.6, []               # start at 60% of available water capacity
    for rain, et0 in zip(d["precip_mm"], d["et0_mm"]):
        store = min(awc_mm, store + rain) - kc * et0
        store = max(0.0, store)
        series.append(store)
    d["soil_water_mm"] = series
    d["deficit_mm"] = (awc_mm * 0.5 - d["soil_water_mm"]).clip(lower=0)
    return d

This is a one-bucket model and makes no claim to be more. Its value is comparative: two fields in the same week with the same index value and deficits of 5 mm and 60 mm are telling different stories, and the second is where a scout should go.

Step 5 — Validate against something measured

PYTHON 
def compare_with_gauge(gridded: pd.DataFrame, gauge: pd.DataFrame, tol_mm: float = 25.0) -> None:
    """Season-total rainfall from the grid should track an on-farm gauge."""
    g = gridded["precip_mm"].sum()
    m = gauge["precip_mm"].sum()
    assert abs(g - m) < max(tol_mm, 0.15 * m), (
        f"gridded season rainfall {g:.0f} mm vs gauge {m:.0f} mm — "
        "check the grid's units (kg m-2 s-1 is not mm) and its accumulation window")

Units are the usual culprit. Many products store precipitation as a flux in kilograms per square metre per second, which converts to millimetres per day by multiplying by 86,400; forgetting it produces a season total near zero, and a naive pipeline reports a drought.

3. Key Parameters and Tuning

Parameter Type Default Agronomic effect
Base temperature °C crop-specific Wrong base shifts accumulated thermal time by hundreds of units, misplacing every stage-dependent decision
Upper cap °C 30 (maize) Without the cap, a heatwave inflates thermal time and the model predicts a growth stage the crop has not reached
Accumulation start date planting Starting on 1 January conflates seasons with different planting dates and destroys cross-field comparability
Grid resolution km 1–4 Adequate for temperature everywhere; inadequate for convective summer rainfall at any resolution above about 1 km
Available water capacity mm 150 Sets how quickly the balance drains; a sandy field at 80 mm and a clay loam at 200 mm behave completely differently in the same week
Crop coefficient kc float 1.0 Scales reference evapotranspiration to the crop and stage; leaving it at 1.0 overstates early-season water use before canopy closure
Aggregation method choice area-weighted Centroid sampling introduces step changes when boundaries are edited and cannot represent a field straddling cells

4. Edge Cases and Failure Modes

Units that look like millimetres and are not. Flux versus depth, Kelvin versus Celsius, accumulations over an hour versus a day. Assert the plausible range of every variable on load — daily mean temperature between −40 and 50, daily rainfall between 0 and 300 — rather than trusting an attribute string.

Four ways a product can store precipitation A table of four precipitation units and their conversion to millimetres per day: a flux in kilograms per square metre per second needs multiplying by 86,400, metres by 1,000, and mass per unit area and millimetres need no conversion. Getting the first two wrong produces a season total near zero. Stored unit Conversion to mm/day If you skip it kg m⁻² s⁻¹ a flux, not a depth multiply by 86,400 Season total near zero — a fake drought m depth in metres multiply by 1,000 A wet season reads as 0.4 mm kg m⁻² mass per area already millimetres No error either way mm depth in millimetres no conversion No error either way

Fields outside the grid. A farm near a coastline or a national boundary can fall in cells the product masks. The zonal statistic returns NaN, the cumulative sum silently propagates it, and the whole season’s covariates for that field become null. Fail on the first NaN rather than at the end.

Replanting. A field replanted after a hail event has two planting dates, and thermal time must restart. Model the accumulation against the field-season’s current planting date and keep the abandoned accumulation rather than overwriting it — the first stand’s history explains the second stand’s compaction.

Southern-hemisphere seasons. A season that spans a calendar-year boundary breaks any code that keys on year alone. Use the field-season’s own start and end dates as the accumulation window; the season column is a label, not a date range.

Forecast contamination. Many products blend analysis and forecast within the same file. Aggregating both produces covariates that change retrospectively as forecasts are replaced by analysis, and a model trained on them is trained partly on predictions. Read the product’s data-status flag and keep forecast rows in a separate column.

Leap days and daylight saving in hourly data. Aggregating hourly to daily with a local-time offset produces 23- and 25-hour days twice a year. Aggregate in UTC and label the day by local date only at the end.

5. Verification and Output Validation

PYTHON 
def validate_covariates(df: pd.DataFrame, crop: str, season_days: int) -> None:
    """Assert a field-season covariate table is physically and agronomically plausible."""
    assert df["date"].is_monotonic_increasing and df["date"].is_unique, "dates repeat or run backwards"
    assert len(df) >= season_days * 0.95, f"only {len(df)} of ~{season_days} days present"

    assert df["gdd"].ge(0).all(), "negative daily thermal time — base temperature applied twice?"
    assert df["gdd_cum"].is_monotonic_increasing, "accumulated thermal time must never decrease"

    total_gdd = float(df["gdd_cum"].iloc[-1])
    expected = {"maize": (1200, 1900), "soybean": (1000, 1700), "wheat": (1400, 2600)}[crop]
    assert expected[0] < total_gdd < expected[1], (
        f"season thermal time {total_gdd:.0f} outside the plausible range {expected} for {crop} — "
        "check base temperature, cap and accumulation start")

    assert df["precip_mm"].between(0, 300).all(), "daily rainfall outside 0–300 mm — unit error"
    assert df["soil_water_mm"].between(0, 400).all(), "soil water outside physical bounds"

The thermal-time range check is the one that catches real mistakes. Every individual component can look reasonable while the season total lands at 3,400 degree-days for maize — a sure sign the base temperature was zero, and something no per-day assertion would notice.

As always, prove the gate works: feed it a table built with the wrong crop’s base temperature and assert it raises.

6. Integration with the Broader Pipeline

The daily covariate table is a spine that the rest of the platform hangs off. Joined to the accepted imagery dates from satellite imagery APIs and STAC catalogues, it converts an index time series into a growth-stage-aware one, which is what makes vegetation index selection for crop stages operational rather than theoretical — NDRE is the right index at V8, and thermal time is how you know a field is at V8. Joined to yield outcomes from yield monitor data cleaning and telemetry QA, it separates a genuinely poor zone from a zone that was merely dry in the six days that mattered, which changes what management zone classification algorithms should be clustering on.

Storage is ordinary: one row per field-season-day in PostGIS, indexed on field-season and date, as covered in PostGIS schema design for farm data. The scheduling is the daily case in orchestrating seasonal pipelines with Airflow.

Frequently Asked Questions

Should I use an on-farm weather station instead of a grid? Use both. A station is authoritative for the point it sits at and says nothing about a field eight kilometres away; a grid is smooth, complete and systematically wrong in convective rain. The productive arrangement is a grid for the daily series and a station for continuous validation — a station drifting from the grid by more than its usual bias is often a blocked gauge rather than a weather event.

How far back should I backfill weather? As far as your yield history goes. Weather is the only covariate that can be reconstructed retrospectively for seasons you did not observe, which makes it the cheapest way to give a model years of context. Backfilling a decade for a few hundred fields is hours of compute and a few hundred megabytes.

Can I use this to schedule irrigation? The single-bucket balance here is a screening tool, not an irrigation scheduler. Real scheduling needs soil-specific water-holding capacity by depth, a crop coefficient curve by growth stage, and ideally soil moisture sensors to correct the drift that any water balance accumulates over a season.

This topic is part of Farm Data Platform Engineering: APIs, Storage & Orchestration — see there for how these covariates are stored and scheduled.

Parent section Farm Data Platform Engineering: APIs, Storage & Orchestration Related Cloud-Optimized Storage for Field Imagery Related Machine Data APIs for Management-System Integration Related PostGIS Schema Design for Farm Data Related Satellite Imagery APIs & STAC Catalogues for Field Monitoring Related Orchestrating Seasonal Pipelines with Airflow