Yield Monitor Data Cleaning & Telemetry QA

A raw yield file is not a yield map. It is a record of what a mass-flow sensor reported, at a position recorded some seconds after the grain was actually cut, over an area the combine may or may not have fully covered, at a moisture that varied through the afternoon. The output of this topic is a cleaned point layer that can honestly be interpolated: flow delay corrected, start and end of pass removed, effective width resolved, moisture normalised, and outliers flagged by a spatial test rather than a global percentile. It sits under yield mapping and variable rate prescription generation and feeds directly into spatial interpolation for yield data.

Cleaning routinely removes 10–25% of raw samples. That sounds drastic until you look at what is removed: the seconds after the header enters standing crop, the seconds after it leaves, the passes that were half-width along a boundary, and the moment the operator paused to unload. None of it is measurement of anything.

Prerequisites

  • Python 3.11+, pandas 2.2., geopandas 1.0., numpy 1.26., scipy 1.13., shapely 2.0.*
  • Telemetry with, at minimum: timestamp, position, mass flow or wet yield, moisture, ground speed, header width and header status
  • Fix quality per sample where available — the budget from RTK GPS accuracy and positional error budgets determines how tightly anything here can be trusted
  • A projected CRS in metres for the field, per understanding CRS in precision agriculture
  • Scale tickets or a weigh wagon total for the field, for calibration
  • The field boundary, for headland handling and for the area denominator

1. Concept: Six Artefacts, One Order

The order matters. Correcting flow delay after removing ramp-up removes the wrong samples; normalising moisture before removing zeros divides by numbers that were never measurements.

The cleaning order and what each stage removes Six sequential stages with the typical share of samples removed at each: pass reconstruction removes nothing but splits the data, flow delay correction shifts values without removing, ramp trimming removes eight to fifteen percent, width correction adjusts rather than removes, moisture normalisation and calibration adjust values, and spatial outlier flagging removes two to five percent. A note records that raw telemetry is never edited. 1 · Passes split on gaps, turns, fix loss removes none 2 · Flow delay shift yield back 10–20 s along track moves, never drops 3 · Ramp trim start / end of pass, unload pauses removes 8–15% 4 · Width overlap and partial header passes adjusts the area 5 · Moisture to standard %, then scale-ticket factor rescales values 6 · Spatial outliers neighbourhood test, not a global percentile — a genuinely poor zone is not an outlier removes 2–5% Raw telemetry is never edited Cleaning writes a derived layer with a flag column, so a better rule next season can be re-applied to every past harvest. Typical net retention on a well-run harvest: 75–90% of raw samples.

2. Step-by-Step Implementation

How much of a yield file survives cleaning Bars showing the share of raw samples retained through cleaning: ramp and pause trimming removes about twelve percent, spatial outlier flagging a further three, leaving roughly eighty-five percent. A pipeline that retains only sixty percent is shaping the map rather than cleaning it. Raw samples from the monitor 100% — what the file contains After ramp and pause trimming ≈ 88% — 12% removed After spatial outlier flagging ≈ 85% — a further 3% A pipeline removing 40% 60% — the map is being sculpted Percent of raw samples retained. Print this at every stage — a jump between seasons is a pipeline bug until proven otherwise.

Step 1 — Reconstruct passes

PYTHON 
import numpy as np
import pandas as pd
import geopandas as gpd

def split_passes(gdf: gpd.GeoDataFrame, gap_s: float = 5.0,
                 turn_deg: float = 60.0) -> gpd.GeoDataFrame:
    """Assign a pass identifier, breaking on time gaps and sharp heading changes."""
    assert gdf.crs is not None and gdf.crs.is_projected, "project to metres before pass splitting"
    g = gdf.sort_values("ts").copy()

    dt = g["ts"].diff().dt.total_seconds().fillna(0.0)
    dx = g.geometry.x.diff().fillna(0.0)
    dy = g.geometry.y.diff().fillna(0.0)
    heading = np.degrees(np.arctan2(dy, dx))
    turn = np.abs((heading.diff().fillna(0.0) + 180) % 360 - 180)

    new_pass = (dt > gap_s) | (turn > turn_deg)
    g["pass_id"] = new_pass.cumsum().astype("int32")
    sizes = g.groupby("pass_id").size()
    assert sizes.max() > 20, "no pass longer than 20 samples — thresholds are too aggressive"
    return g

Splitting on heading change is what separates a working pass from a headland turn. Without it, the turn’s samples — recorded while the header is out of crop and the machine is pivoting — are treated as part of the pass and their positions are the least reliable in the file.

Step 2 — Correct grain flow delay

PYTHON 
def apply_flow_delay(g: gpd.GeoDataFrame, delay_s: float = 12.0) -> gpd.GeoDataFrame:
    """Shift yield readings backwards in time within each pass."""
    out = []
    for pass_id, chunk in g.groupby("pass_id", sort=False):
        c = chunk.sort_values("ts").copy()
        dt = c["ts"].diff().dt.total_seconds().median()
        assert dt and dt > 0, f"pass {pass_id}: cannot determine sample interval"
        shift = int(round(delay_s / dt))
        c["yield_wet"] = c["yield_wet"].shift(-shift)      # value measured later belongs here
        c["moisture"] = c["moisture"].shift(-shift)
        out.append(c)
    res = pd.concat(out).dropna(subset=["yield_wet"])
    return gpd.GeoDataFrame(res, geometry="geometry", crs=g.crs)

The delay is machine- and crop-specific and is worth measuring rather than assuming: harvest a strip, stop cleanly at a known point, and see how many seconds of flow continue after the header leaves the crop. Twelve seconds is a reasonable default for a modern combine in maize; small-grain machines are often faster.

Step 3 — Trim ramp-up, ramp-down and pauses

PYTHON 
def trim_pass_ends(g: gpd.GeoDataFrame, ramp_s: float = 8.0,
                   min_speed_m_s: float = 0.6) -> gpd.GeoDataFrame:
    """Drop the unstable seconds at each end of a pass, and any near-stationary samples."""
    keep = []
    for _pass_id, chunk in g.groupby("pass_id", sort=False):
        c = chunk.sort_values("ts")
        t = (c["ts"] - c["ts"].iloc[0]).dt.total_seconds()
        span = t.iloc[-1]
        mask = (t > ramp_s) & (t < span - ramp_s) & (c["speed_m_s"] > min_speed_m_s)
        keep.append(c[mask])
    out = pd.concat(keep)
    removed = 1 - len(out) / len(g)
    assert removed < 0.4, f"ramp trimming removed {removed:.0%} of samples — passes are too short"
    print(f"ramp trim removed {removed:.1%} of samples")
    return gpd.GeoDataFrame(out, geometry="geometry", crs=g.crs)

The near-stationary filter catches unloading on the move and the pause at the end of a row. A stationary combine with grain still moving through it reports yield over zero area, which after division becomes an arbitrarily large number — the single most common source of impossible values.

Step 4 — Effective width

PYTHON 
from shapely.geometry import LineString

def effective_width(g: gpd.GeoDataFrame, header_m: float,
                    swath_tolerance: float = 0.15) -> gpd.GeoDataFrame:
    """Estimate the width actually cut, using the distance to the neighbouring pass."""
    g = g.copy()
    centres = (g.groupby("pass_id")
                 .apply(lambda c: LineString(list(zip(c.geometry.x, c.geometry.y)))
                        if len(c) > 1 else None))
    centres = centres.dropna()
    widths = {}
    for pid, line in centres.items():
        others = [ln for other_pid, ln in centres.items() if other_pid != pid]
        if not others:
            widths[pid] = header_m
            continue
        nearest = min(line.distance(ln) for ln in others)
        widths[pid] = float(np.clip(nearest, header_m * swath_tolerance, header_m))
    g["width_eff_m"] = g["pass_id"].map(widths)
    assert g["width_eff_m"].between(header_m * swath_tolerance, header_m).all(), (
        "effective width outside plausible bounds")
    return g

Crediting a half-width finishing pass with the full header is what makes mapped totals exceed scale tickets. The distance-to-neighbour estimate is crude but captures the dominant case: the last pass down a field edge.

Step 5 — Moisture normalisation and calibration

PYTHON 
STANDARD_MOISTURE = {"maize": 15.5, "soybean": 13.0, "wheat": 13.5, "canola": 8.5}


def to_dry_basis(g: gpd.GeoDataFrame, crop: str) -> gpd.GeoDataFrame:
    std = STANDARD_MOISTURE[crop]                      # KeyError on an unknown crop, deliberately
    m = g["moisture"].clip(lower=5.0, upper=40.0)
    g = g.copy()
    g["yield_std"] = g["yield_wet"] * (100.0 - m) / (100.0 - std)
    return g


def calibrate_to_ticket(g: gpd.GeoDataFrame, ticket_kg: float, field_ha: float) -> float:
    """Single multiplicative factor bringing the mapped total onto the weighed total."""
    mapped_kg = float((g["yield_std"] * g["area_ha"]).sum())
    assert mapped_kg > 0, "mapped total is zero — check units before calibrating"
    factor = ticket_kg / mapped_kg
    assert 0.8 < factor < 1.25, (
        f"calibration factor {factor:.2f} is outside the plausible range; a factor this large "
        "usually means uncorrected overlap or a unit mismatch, not sensor drift")
    return factor

A calibration factor is a last step, not a first one. Applying it before overlap correction hides the overlap error inside the factor, which then fails to transfer to the next field.

Step 6 — Spatial outliers, not global ones

Removing the top and bottom 2% of yields globally deletes the genuinely poor corner of the field — exactly the ground a zone map exists to find. The defensible test is local: compare each point with its neighbours.

PYTHON 
from scipy.spatial import cKDTree

def flag_spatial_outliers(g: gpd.GeoDataFrame, radius_m: float = 15.0,
                          k: int = 12, z_limit: float = 3.5) -> gpd.GeoDataFrame:
    """Flag points that disagree sharply with their spatial neighbourhood."""
    coords = np.c_[g.geometry.x, g.geometry.y]
    tree = cKDTree(coords)
    idx = tree.query_ball_point(coords, r=radius_m)
    vals = g["yield_std"].to_numpy()

    flags = np.zeros(len(g), dtype=bool)
    for i, neigh in enumerate(idx):
        neigh = [j for j in neigh if j != i]
        if len(neigh) < k:
            continue                                   # too sparse to judge — keep it
        local = vals[neigh]
        med = np.median(local)
        mad = np.median(np.abs(local - med)) or 1e-6
        flags[i] = abs(vals[i] - med) / (1.4826 * mad) > z_limit

    g = g.copy()
    g["outlier"] = flags
    print(f"flagged {flags.mean():.1%} of samples as spatial outliers")
    return g

Median absolute deviation rather than standard deviation matters here: a handful of impossible values inflates a standard deviation enough to hide themselves.

3. Key Parameters and Tuning

Parameter Type Default Agronomic effect
Flow delay s 12 Under-correcting smears yield down-track and shifts headland values inward; over-correcting drags standing-crop yield onto already-harvested ground
Ramp trim s 8 each end Too short leaves the unstable filling and emptying of the threshing system in the map; too long deletes real headland yield
Minimum speed m/s 0.6 Removes unloading pauses; set too high it deletes legitimately slow passes through heavy crop, biasing the map against high-yielding areas
Turn threshold ° 60 Separates passes from headland turns; too low fragments curved passes on contoured fields
Swath tolerance fraction 0.15 Lower bound on plausible effective width; too high discards genuine narrow finishing passes
Outlier radius / k m / int 15 / 12 Sets the neighbourhood; too large and a sharp real boundary looks like an outlier, too small and there is nothing to compare against
MAD z limit float 3.5 Below 3 the filter starts removing real variation; above 5 it stops catching sensor spikes
Calibration factor bounds float 0.8–1.25 A guard rail, not a target: a factor outside it is a modelling error rather than sensor drift

4. Edge Cases and Failure Modes

Two combines on one field. Different machines have different flow delays, header widths and calibration states. Clean per machine, calibrate per machine against its own tickets, and only then merge — a single field-wide factor applied to two machines splits the difference and is wrong for both.

Three cases where the right answer is not to filter Three panels on situations that look like data quality problems and are not: two combines needing separate calibration rather than one shared factor, headlands that are genuinely different rather than dirty, and a stuck moisture sensor that makes normalisation a no-op over half the field. Two combines, one factor Different flow delays and header widths One calibration factor splits the difference. Both machines' areas end up wrong. Clean and calibrate per machine, then merge. Headlands folded into the field Harvested first, compacted, often earlier Genuinely different, not dirty data. Deleting them hides a real management area. Label them; include or exclude deliberately. Moisture sensor stuck Constant value for half the field Normalisation silently does nothing there. Half the map is on a different basis. Assert moisture varies across the file.

Grain cart and unload-on-the-move. Yield continues to be recorded while unloading, and the mass-flow signal is disturbed. If the file records an unload flag, use it; otherwise the near-stationary and speed-variance filters catch most of it.

A moisture sensor that fails mid-field. Moisture pinned at a constant value for the second half of a field means normalisation is doing nothing there. Assert that moisture varies: a standard deviation near zero across an afternoon is a failed sensor, not a uniform crop.

Headlands harvested first. Grain from headlands harvested before the field body is legitimately different — compacted, often earlier — and cleaning should not try to fix that. Keep headlands as a labelled subset rather than deleting them; whether to include them in zone work is an agronomic decision, not a data-quality one.

Negative or absurd yields. Physically impossible values are the sign of division by a near-zero area or a sensor fault. Remove them before any statistic is computed, and count them: a rate above about 1% of samples means the machine needs attention.

Over-cleaning. The most damaging failure in this whole topic, because it is invisible. A pipeline that removes 40% of samples produces a beautifully smooth map that has been sculpted by its filters rather than measured. Print the removal share at every stage, keep the total below roughly 25%, and treat a large jump between seasons as a bug in the pipeline until proven otherwise.

5. Verification and Output Validation

PYTHON 
def validate_clean_yield(raw: gpd.GeoDataFrame, clean: gpd.GeoDataFrame,
                         ticket_kg: float, field_ha: float, crop: str) -> None:
    retained = len(clean) / len(raw)
    assert 0.6 < retained < 0.98, f"retained {retained:.0%} of samples — cleaning is out of range"

    mapped_kg = float((clean["yield_std"] * clean["area_ha"]).sum())
    err = abs(mapped_kg - ticket_kg) / ticket_kg
    assert err < 0.05, f"mapped total is {err:.1%} from the scale ticket after calibration"

    mean_yield = mapped_kg / field_ha
    plausible = {"maize": (4000, 16000), "soybean": (1500, 6000), "wheat": (2000, 11000)}[crop]
    assert plausible[0] < mean_yield < plausible[1], (
        f"mean yield {mean_yield:.0f} kg/ha outside the plausible range for {crop} — "
        "check units; bushels per acre and kilograms per hectare differ by a factor near 60")

    covered_ha = float(clean["area_ha"].sum())
    assert 0.8 * field_ha < covered_ha < 1.15 * field_ha, (
        f"covered area {covered_ha:.1f} ha against a {field_ha:.1f} ha field — overlap correction")

The covered-area check is the one that catches the error nobody looks for. Mapped totals can match a scale ticket perfectly while the area is 30% too large, because a calibration factor will happily absorb the discrepancy — and the resulting map has the right total in the wrong places, which is worse than being uniformly wrong.

Prove the gates fire: run the validator against a deliberately uncorrected overlap case and against a bushels-per-acre file, and assert both raise.

6. Integration with the Broader Pipeline

Cleaned points are the input to interpolating sparse yield monitor data with kriging and to the comparison in comparing IDW and kriging for yield maps — both of which assume the artefacts here have already gone, because a variogram fitted to uncleaned data models the flow delay rather than the field. The interpolated surface then drives k-means clustering for yield zone delineation and ultimately variable-rate export to ISOXML.

Upstream, the raw telemetry arrives from machine data APIs or a monitor export and is stored per the partitioning in PostGIS schema design for farm data. Positional quality — whether the fix was good enough to trust a 15 m neighbourhood at all — comes from RTK GPS accuracy and positional error budgets.

Frequently Asked Questions

How much data should cleaning remove? Ten to twenty-five percent on a well-run harvest, most of it ramp-up and end-of-pass. Below 5% suggests the filters are not running; above 35% suggests they are misconfigured, or the harvest itself was interrupted enough that the map deserves a caveat.

Can I skip cleaning if I am only making zones? No — zones are exactly where it matters. Flow delay shifts every boundary down-track by 15–25 m, which is comparable to the zone dimensions themselves, so an uncleaned map produces zones displaced from the ground they describe. The prescription is then applied to the wrong part of the field with complete confidence.

Should cleaned data be stored or recomputed? Store it as a derived layer with a version, and keep the raw. Recomputing on demand is fine at one field and painful at four hundred; keeping only the cleaned form means a better flow-delay estimate next season cannot be applied to this one.

This topic is part of Yield Mapping & Variable Rate Prescription Generation — see there for the full path from telemetry to a controller-ready prescription.

Parent section Yield Mapping & Variable Rate Prescription Generation Related Converting Prescriptions Between Controller Formats Related Debugging Prescription Export Errors Related Management Zone Classification for Precision Agriculture Related Shapefile Validation for Farm Equipment Related Spatial Interpolation for Yield Data in Python Related Variable Rate Export to ISOXML for Precision Agriculture