Removing Yield Outliers with Neighborhood Statistics

One-sentence answer: compare each sample with the median of its neighbours within about fifteen metres, scale the difference by the neighbourhood’s median absolute deviation, and flag anything beyond roughly 3.5 — a poor zone agrees with its neighbours and survives, while a sensor spike does not.

Context

Every yield file contains values that cannot be true: a reading over a stationary combine still emptying its threshing system, a spike from a sensor knock, a negative flow after a recalibration. They must go before any statistic is computed, because a single impossible value distorts a variogram, a mean and every zone boundary derived from them.

The tempting fix is a global percentile trim, and it is the wrong one. Yield is spatially structured: the poor corner of a field is a contiguous region of genuinely low values, and it is the region a variable-rate prescription exists to treat differently. A filter that cannot tell it from a spike removes the reason for the whole exercise. This guide implements the final stage of yield monitor data cleaning and telemetry QA.

What each filter removes from the same field The same field shown twice. On the left, a global percentile filter deletes every sample in a genuinely low-yielding corner as well as two isolated sensor spikes. On the right, a neighbourhood median-absolute-deviation filter keeps the poor corner intact because its samples agree with their neighbours, and removes only the two spikes. Global percentile trim (bottom 2%) poor corner — deleted the zone map now has no low zone in it Neighbourhood MAD filter poor corner — kept only the two isolated spikes are flagged Amber marks what each filter removes. Both remove the spikes; only one removes the field's most important feature.

Prerequisites

Beyond the parent topic’s stack: scipy 1.13.* for the spatial index, telemetry already corrected for flow delay as in filtering combine flow delay and ramp-up errors, and a projected CRS so the radius is in metres.

Step-by-step

1. Remove impossible values — negative, zero-area, or beyond any credible rate.

Why two of these filters delete the answer A table of four outlier filters: a global percentile trim, a global mean and standard deviation rule, a neighbourhood median-absolute-deviation test, and a physical range check. The percentile trim deletes the genuinely poor corner, and the standard deviation rule is defeated by the very spikes it targets. Filter What it removes What it also removes Global percentile, bottom 2% distribution-based Sensor spikes The genuinely poor corner Global mean ± 3σ distribution-based Extreme values Nothing — the spikes inflate σ Neighbourhood median ± 3.5 MAD spatial Isolated spikes Nothing that agrees with neighbours Physically impossible values range Negatives and absurd rates Nothing real

2. Build a KD-tree over the projected coordinates.

3. Compare each sample with its neighbourhood median, scaled by MAD.

4. Skip sparse neighbourhoods rather than judging on three points.

5. Flag, count, and check the removal share.

PYTHON 
import geopandas as gpd
import numpy as np
from scipy.spatial import cKDTree

PLAUSIBLE_KG_HA = {"maize": (0, 25_000), "soybean": (0, 9_000), "wheat": (0, 16_000)}


def drop_impossible(g: gpd.GeoDataFrame, crop: str, col: str = "yield_std") -> gpd.GeoDataFrame:
    """Remove values that cannot be measurements, before any statistic is computed."""
    lo, hi = PLAUSIBLE_KG_HA[crop]
    bad = ~g[col].between(lo, hi) | g[col].isna()
    share = float(bad.mean())
    print(f"{share:.2%} of samples outside the plausible {lo}–{hi} kg/ha range")
    assert share < 0.05, (
        f"{share:.1%} of samples are physically impossible — that is a machine or unit problem, "
        "not something a filter should quietly absorb")
    return g.loc[~bad].copy()


def flag_neighborhood_outliers(g: gpd.GeoDataFrame, col: str = "yield_std",
                               radius_m: float = 15.0, min_neighbors: int = 12,
                               z_limit: float = 3.5) -> gpd.GeoDataFrame:
    """Flag samples that disagree with their spatial neighbourhood, using a robust z score."""
    assert g.crs is not None and g.crs.is_projected, "project to metres before using a radius"

    coords = np.c_[g.geometry.x, g.geometry.y]
    tree = cKDTree(coords)
    neighbours = tree.query_ball_point(coords, r=radius_m)
    values = g[col].to_numpy(dtype="float64")

    flags = np.zeros(len(g), dtype=bool)
    scores = np.full(len(g), np.nan)
    sparse = 0
    for i, idx in enumerate(neighbours):
        idx = [j for j in idx if j != i]
        if len(idx) < min_neighbors:
            sparse += 1
            continue                              # too few neighbours to judge — keep it
        local = values[idx]
        med = np.median(local)
        mad = np.median(np.abs(local - med))
        scale = 1.4826 * mad if mad > 0 else np.finfo(float).eps
        scores[i] = abs(values[i] - med) / scale
        flags[i] = scores[i] > z_limit

    out = g.copy()
    out["outlier"] = flags
    out["robust_z"] = scores
    print(f"flagged {flags.mean():.2%} as outliers; {sparse / len(g):.1%} of samples had "
          f"fewer than {min_neighbors} neighbours within {radius_m:.0f} m")
    return out

Inline verification — prove the filter keeps a real low zone and removes a real spike:

PYTHON 
clean = flag_neighborhood_outliers(drop_impossible(points, crop="maize"))

# 1. Overall removal must stay modest.
share = float(clean["outlier"].mean())
assert share < 0.06, f"flagged {share:.1%} — the threshold is too aggressive for this field"

# 2. A contiguous low-yielding region must survive.
low_zone = clean[clean.geometry.within(poor_corner_polygon)]
kept = 1 - float(low_zone["outlier"].mean())
assert kept > 0.90, (
    f"only {kept:.0%} of the known poor corner survived — the filter is deleting real "
    "agronomy; increase the radius or raise z_limit")

# 3. An injected spike must be caught.
probe = clean.copy()
probe.loc[probe.index[500], "yield_std"] = 1e5
probe = flag_neighborhood_outliers(probe)
assert bool(probe.loc[probe.index[500], "outlier"]), "the filter did not catch an obvious spike"

The third check is the one that keeps the filter honest. A gate that has never rejected anything proves nothing, and it is easy to tune a threshold until nothing is flagged at all.

Settings worth being deliberate about

Setting Default here Why it matters
Radius 15 m Should span the current pass and its neighbours. Too small and there is nothing to compare against; too large and a real soil boundary looks like an outlier from both sides
Minimum neighbours 12 Below this the local median is unstable. Samples with fewer are left alone rather than judged on three points
Robust z limit 3.5 Below 3 the filter starts eating real variation; above 5 it stops catching sensor spikes
Scale estimator 1.4826 × MAD Makes the MAD comparable to a standard deviation for normal data while staying immune to the spikes being hunted
Impossible-value range crop-specific Applied first, because one impossible value distorts any statistic computed alongside it
Total removal budget under ~6% A filter removing more than this is reshaping the field rather than cleaning it

Report the removal share on every run and watch it across seasons. A field whose flagged share jumps from 3% to 12% has either a failing sensor or a changed pipeline, and both are worth knowing before the map is used.

Gotchas and edge cases

  • The radius interacts with the sampling density. A combine at 6 km/h logging at 1 Hz places samples 1.7 m apart along the track and a header width apart across it, so a 15 m radius typically captures both the current pass and its neighbours. Too small and there is nothing to compare with; too large and a sharp real boundary — a soil-type change, a variety strip — looks like an outlier on both sides.
The filter declines to judge where it cannot A decision diagram for the neighbourhood outlier test. A sample with twelve or more neighbours within fifteen metres is judged against the local median; one with fewer sits at the edge of the point cloud, where there is not enough local evidence, and is left alone. A sample disagrees with its neighbours are there 12 or more neighbours within 15 m? yes Judge it against the neighbourhood compare with the local median, scaled by the median absolute deviation, and flag beyond 3.5 no Leave it alone at the edge of the point cloud there is not enough local evidence to judge, and filtering on three neighbours removes real headland data

  • A zero MAD means a uniform neighbourhood. It happens on a monitor that quantises heavily. Guarding with machine epsilon, as above, prevents a division by zero that would flag every sample; a neighbourhood with genuinely no variation should flag nothing.

  • Edge samples have fewer neighbours by construction. Headland and boundary samples sit at the edge of the point cloud, so a strict minimum-neighbour rule silently excludes them from filtering. That is the right default — better unfiltered than judged against half a neighbourhood.

  • Flag, do not delete. Store the flag with the raw telemetry, exclude it in the derived view. Deleting rows makes every future improvement to the filter unusable on past seasons, which is the same reasoning applied to cleaning as a whole in the parent topic.

  • The loop is O(n) with a small constant but Python-level. At a few hundred thousand points it takes seconds; at ten million, vectorise with tree.query over a fixed k instead of a radius, accepting a slightly different neighbourhood definition.

This guide is part of Yield Monitor Data Cleaning & Telemetry QA — see there for the full cleaning order and the validation that follows.

Parent section Yield Monitor Data Cleaning & Telemetry QA Related Filtering Combine Flow Delay and Ramp-Up Errors