Backfilling a Season of Field Imagery Without Duplicates

One-sentence answer: enumerate the backfill as small field-by-month units, subtract what the manifest already holds, run them through a parameterised DAG that shares the provider’s pool at a lower priority weight, and let idempotent tasks make any overlap with live ingestion a no-op.

Context

Backfills happen for good reasons: a new grower brings three seasons of history, a cloud-mask fix means last year’s scenes should be reprocessed, a field was added mid-season and its imagery was never fetched. They also have a reputation for taking down the daily pipeline — a six-month backfill firing thousands of requests will exhaust a provider’s quota, and the live ingestion that shares that quota fails for the rest of the day.

Everything needed to avoid that already exists if the ingestion task is idempotent, as in writing idempotent Airflow tasks for imagery ingestion. What this guide adds is the shape of the run: unit size, ordering, and the two scheduler settings that keep live work in front.

A shared pool keeps the backfill behind live work Six pool slots over four time windows. Live ingestion claims a varying number of slots each window and backfill units fill whatever remains, so the total occupied slots stay constant at six and the provider sees a steady request rate rather than a burst. pool "stac_api" — 6 slots 08:00 2 live · 6 total 12:00 5 live · 6 total 18:00 6 live · 6 total 23:00 1 live · 6 total live ingestion (higher priority weight) backfill units (lower priority weight) Without a shared pool both DAGs open their own concurrency and the provider sees twelve simultaneous requests — then throttles both.

Prerequisites

Beyond the parent topic’s stack: an idempotent ingestion function, a populated ingest_manifest, and an Airflow pool named for the provider with a slot count below the provider’s concurrency limit.

Step-by-step

1. Enumerate in small units. Field-by-month is the right granularity: small enough that losing one costs minutes, large enough that scheduling overhead is negligible.

Unit size decides whether a backfill is restartable Bars comparing backfill unit granularity for four hundred fields over six months. Field by month produces about 2,400 task instances and is comfortable; weekly produces 10,400; daily produces 73,000 and the scheduler's metadata database becomes the bottleneck; and a single task for the whole backfill means losing it costs everything. Field × month, 400 fields × 6 months 2,400 task instances — comfortable Field × week 10,400 — scheduler starts to strain Field × day 73,000 — the metadata database is the bottleneck One task for the whole backfill 1 — losing it costs the entire run Bars are on a shared scale, so the daily bar is what 73,000 task instances looks like next to 2,400.

2. Subtract what exists by querying the manifest before scheduling, so the run size reflects real work.

3. Parameterise the DAG with the date range and an optional field list.

4. Share the pool, lower the weight.

5. Report coverage — a backfill that silently skipped a fortnight is worse than one that failed.

PYTHON 
from datetime import date, timedelta

from airflow.decorators import dag, task
from sqlalchemy import create_engine, text


def month_starts(start: date, end: date) -> list[date]:
    months, cur = [], start.replace(day=1)
    while cur <= end:
        months.append(cur)
        cur = (cur.replace(day=28) + timedelta(days=4)).replace(day=1)
    return months


def outstanding_units(dsn: str, field_ids: list[int], start: date, end: date) -> list[dict]:
    """Field-months with fewer published scenes than the season's expectation."""
    engine = create_engine(dsn, future=True)
    with engine.connect() as conn:
        have = {(r.field_id, r.month): r.n for r in conn.execute(text("""
            SELECT field_id, date_trunc('month', acquired_at)::date AS month, count(*) AS n
            FROM raster_artifact
            WHERE kind = 's2_window' AND acquired_at BETWEEN :s AND :e
              AND field_id = ANY(:ids)
            GROUP BY 1, 2
        """), {"s": start, "e": end, "ids": field_ids})}

    units = []
    for fid in field_ids:
        for m in month_starts(start, end):
            if have.get((fid, m), 0) < 2:            # fewer than two usable scenes: worth trying
                units.append({"field_id": fid, "month": m.isoformat()})
    print(f"{len(units)} outstanding field-month unit(s) of "
          f"{len(field_ids) * len(month_starts(start, end))} possible")
    return units


@dag(dag_id="backfill_field_imagery", schedule=None, catchup=False,
     start_date=date(2026, 1, 1), max_active_runs=1,
     params={"start": "2026-04-01", "end": "2026-09-30", "field_ids": []},
     tags=["backfill", "satellite"])
def backfill_field_imagery():

    @task
    def plan(**context) -> list[dict]:
        p = context["params"]
        ids = p["field_ids"] or all_active_field_ids()
        units = outstanding_units(DSN, ids, date.fromisoformat(p["start"]),
                                  date.fromisoformat(p["end"]))
        assert units, "nothing outstanding — the backfill would do no work"
        return units

    @task(pool="stac_api", priority_weight=1, max_active_tis_per_dag=4,
          retries=3, retry_exponential_backoff=True)
    def ingest_unit(unit: dict) -> dict:
        from platform_stac import sync_field_month
        return sync_field_month(unit["field_id"], unit["month"])   # manifest-guarded

    @task
    def report(results: list[dict], **context) -> None:
        published = sum(r["published"] for r in results if r)
        skipped = sum(1 for r in results if r and r["published"] == 0)
        failed = sum(1 for r in results if not r)
        print(f"backfill: {published} scene(s) published, {skipped} unit(s) already complete, "
              f"{failed} unit(s) failed")
        assert failed == 0, f"{failed} unit(s) failed — re-trigger the DAG for those fields"

    report(ingest_unit.expand(unit=plan()))


backfill_field_imagery()

Inline verification — prove the backfill overlapping live ingestion changes nothing:

PYTHON 
from sqlalchemy import create_engine, text

engine = create_engine(DSN, future=True)
with engine.connect() as conn:
    before = conn.execute(text("SELECT count(*) FROM ingest_manifest")).scalar()

sync_field_month(field_id=4471, month="2026-06-01")     # the live DAG's unit
sync_field_month(field_id=4471, month="2026-06-01")     # the backfill's unit, same ground

with engine.connect() as conn:
    after = conn.execute(text("SELECT count(*) FROM ingest_manifest")).scalar()
    dupes = conn.execute(text("""
        SELECT count(*) FROM (
          SELECT source, external_id, checksum FROM ingest_manifest
          GROUP BY 1,2,3 HAVING count(*) > 1
        ) d""")).scalar()

print(f"manifest grew by {after - before} row(s); {dupes} duplicate group(s)")
assert dupes == 0, "duplicate manifest rows — the unique constraint is missing"

Settings worth being deliberate about

Setting Default here Why it matters
Unit granularity field × month Small enough that losing one costs minutes, large enough that 2,400 units is a comfortable run. Field × day would be 73,000 task instances and the scheduler becomes the bottleneck
priority_weight 1, below the live DAG The single setting that keeps today’s ingestion in front of last season’s. Equal weights let the backfill claim slots as they free
Pool slots shared with live ingestion Two separate pools means the provider sees the sum of both, which is the problem the pool was meant to solve
max_active_runs 1 Two concurrent backfill runs over overlapping ranges do the same work twice; idempotency makes it harmless and it is still waste
Completeness threshold fewer than 2 scenes/month What counts as an outstanding unit. Requiring a fixed count makes genuinely cloudy months retry forever
Retries 3, exponential A unit that fails three times has a real problem — a missing asset, a revoked credential — and should surface rather than retry all night

Record attempted-but-empty units somewhere. Without that record a month with no clear scene looks outstanding to every future backfill, and each run spends its quota rediscovering that the weather was bad.

Gotchas and edge cases

  • A backfill runs today’s code against last season’s data. For ingestion that is usually correct. For derived products it is not: the output belongs to the current algorithm version and must be written under that version’s prefix, not over the original. Version derived outputs as described in the section overview.
How a backfill converges instead of retrying the weather forever A decision diagram about empty field-months. If a month was attempted and genuinely had no clear scene, recording that outcome lets later backfills skip it. Leaving it merely outstanding means every future run retries it and the backfill never converges. A field-month has no published scenes was it attempted and found empty? yes Record it as attempted-and-empty a manifest row with a null object key; later backfills skip it and the run converges no Leave it outstanding every future backfill retries the same cloudy month, spending quota to rediscover that the weather was bad

  • max_active_tis_per_dag bounds this DAG; the pool bounds everything. Both are needed. Without the pool, the backfill’s four concurrent tasks add to the live DAG’s eight and the provider sees twelve.

  • Empty field-months are legitimate. A month with no clear scene produces nothing, and the unit will look outstanding forever, so every subsequent backfill retries it. Record attempted-and-empty units in the manifest with a null object key, or keep a separate no_data table — otherwise the backfill never converges.

  • Enumerating fields at plan time, not task time. Reading the field list inside every task means four hundred identical queries and a plan that can change mid-run. Compute it once and pass it through.

  • Mapping over tens of thousands of units strains the scheduler. Six months × 400 fields is 2,400 units, which is comfortable. A daily granularity would be 73,000, which is not — the metadata database becomes the bottleneck long before the provider does.

  • Report coverage explicitly. The failure that costs the most is a backfill that ran, succeeded, and covered 60% of what was asked, because a filter silently excluded the rest. Print what was planned, what was published and what was already complete, and assert the failure count is zero.

This guide is part of Orchestrating Seasonal Pipelines with Airflow — see there for cadence, catch-up semantics and freshness alerting.

Parent section Orchestrating Seasonal Pipelines with Airflow Related Writing Idempotent Airflow Tasks for Imagery Ingestion