Paginating and Rate-Limiting Management-System API Pulls

One-sentence answer: follow the provider’s cursor with a hard loop guard, treat 429 as ordinary control flow by honouring Retry-After before falling back to jittered exponential backoff, and bound total in-flight requests with a semaphore rather than trusting per-task limits.

Context

A full-season pull from a grower management system is thousands of requests: field collections, operations, and one measurement-series download per operation. Two failure modes dominate. The first is silent data loss from offset pagination over a collection that is being written to while you read it — during harvest, that is every collection. The second is a retry storm: a burst of 429 responses meets naive retry logic, every worker retries at the same instant, the provider throttles harder, and a backfill that would have taken twenty minutes serially takes six hours and finishes incomplete.

Both are fixed by mechanics rather than by cleverness. This guide is the request layer under machine data APIs for management-system integration, and it assumes a token from authenticating with the Operations Center API.

Why jitter matters after a rate limit Two timelines. The upper one shows four workers retrying at fixed one-second intervals, their attempts stacking at the same instants and producing repeated rate-limit responses. The lower one shows the same four workers with exponential backoff plus random jitter, their attempts spread across the interval so the provider sees a smooth request rate. Fixed interval — every worker retries at the same instant t+1s → 429 t+2s → 429 t+4s → 429 quota never recovers Exponential backoff with jitter — the same four workers, spread out attempts land at distinct times; the provider sees a smooth rate and the pull completes Both timelines show four workers over the same eight seconds.

Prerequisites

Beyond the parent topic’s stack: httpx 0.27.* and tenacity 8.5.*, and knowledge of the provider’s documented per-application and per-grower request limits. If those numbers are not documented, measure them against a sandbox account before a backfill rather than during one.

Step-by-step

1. Follow the cursor. Take the next link from the response body; never construct the next page yourself.

One turn of the pagination loop Four steps repeated per page: request the page using the provider's cursor, yield its items immediately, check the loop guard for a repeated cursor or an implausible page count, and follow the provider's own nextPage link rather than constructing the next URL. Request page cursor or first URL Yield items consume immediately Guard cursor seen before? page count < 2000 Follow nextPage never build the URL yourself A cursor that returns itself is a real provider bug; without the guard it consumes a day's quota in about ninety seconds.

2. Guard the loop. A cursor that returns itself is a real provider bug and will consume a day’s quota in ninety seconds.

3. Handle 429 as control flow, honouring Retry-After when present.

4. Bound concurrency globally, not per task.

5. Checkpoint after each page so an interrupted pull resumes.

PYTHON 
import asyncio
import random
from typing import AsyncIterator

import httpx

RETRYABLE = {429, 500, 502, 503, 504}
MAX_PAGES = 2000
MAX_ATTEMPTS = 6


async def _get_with_backoff(client: httpx.AsyncClient, url: str, *, token: str,
                            params: dict | None = None) -> dict:
    """GET with Retry-After support and full-jitter exponential backoff."""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        resp = await client.get(url, params=params,
                                headers={"Authorization": f"Bearer {token}"})
        if resp.status_code not in RETRYABLE:
            resp.raise_for_status()
            return resp.json()

        if attempt == MAX_ATTEMPTS:
            resp.raise_for_status()

        retry_after = resp.headers.get("Retry-After")
        if retry_after and retry_after.isdigit():
            delay = float(retry_after)
        else:
            # Full jitter: uniform in [0, 2^attempt), capped — never a fixed interval.
            delay = random.uniform(0.0, min(60.0, 2 ** attempt))
        print(f"{resp.status_code} on {url} — attempt {attempt}, sleeping {delay:.1f}s")
        await asyncio.sleep(delay)
    raise RuntimeError("unreachable")


async def iter_pages(client: httpx.AsyncClient, url: str, *, token: str,
                     page_size: int = 100, checkpoint=None) -> AsyncIterator[dict]:
    """Yield each item of a cursor-paginated collection, checkpointing per page."""
    next_url: str | None = url
    params: dict | None = {"pageSize": page_size}
    seen_cursors: set[str] = set()

    for page in range(MAX_PAGES):
        if next_url is None:
            return
        if next_url in seen_cursors:
            raise RuntimeError(f"cursor repeated on {next_url} — provider pagination loop")
        seen_cursors.add(next_url)

        payload = await _get_with_backoff(client, next_url, token=token, params=params)
        for item in payload.get("values") or payload.get("items") or []:
            yield item

        if checkpoint is not None:
            checkpoint(page, next_url)
        next_url = next(
            (l["uri"] for l in payload.get("links", []) if l.get("rel") == "nextPage"), None)
        params = None                          # the cursor already carries the query
    raise RuntimeError(f"pagination exceeded {MAX_PAGES} pages — refusing to continue")


async def pull_all_growers(grower_urls: dict[str, str], tokens: dict[str, str],
                           max_inflight: int = 6) -> dict[str, int]:
    """Parallelise across growers, serial within each collection, globally bounded."""
    sem = asyncio.Semaphore(max_inflight)
    limits = httpx.Limits(max_connections=max_inflight, max_keepalive_connections=max_inflight)
    counts: dict[str, int] = {}

    async with httpx.AsyncClient(timeout=httpx.Timeout(60.0, connect=10.0), limits=limits) as client:
        async def one(grower_id: str, url: str) -> None:
            async with sem:
                n = 0
                async for _item in iter_pages(client, url, token=tokens[grower_id]):
                    n += 1
                counts[grower_id] = n

        await asyncio.gather(*(one(g, u) for g, u in grower_urls.items()))
    return counts

Inline verification — prove the loop guard and the jitter behave before pointing this at a live account:

PYTHON 
import statistics

delays = [random.uniform(0.0, min(60.0, 2 ** a)) for a in range(1, 7) for _ in range(200)]
assert statistics.pstdev(delays) > 1.0, "backoff is not jittered — workers will collide"

# A stub whose nextPage link points at itself must raise, not spin.
try:
    asyncio.run(_drain(iter_pages(stub_client, "/loop", token="t")))
except RuntimeError as exc:
    assert "pagination loop" in str(exc)
else:
    raise AssertionError("the cursor loop guard did not fire")

Settings worth being deliberate about

Setting Default here Why it matters
page_size 100 Fewer round trips at higher values, but field collections with complex boundary geometry commonly time out above 250 — and a timeout costs the whole page, not one record
MAX_PAGES 2000 A loop guard, not a limit you should reach. At 100 records per page it covers two million records; hitting it means the cursor is misbehaving
MAX_ATTEMPTS 6 With full jitter capped at 60 s, six attempts spans roughly two minutes of provider recovery. More attempts hide a systematic outage behind an hour of retries
max_inflight 6 Total concurrent requests across all growers. Size it below the documented per-application limit, and remember that a backfill sharing the quota counts against the same number
Connect timeout 10 s Distinguishes “the provider is unreachable” from “this collection is slow”, which matters because only the second is worth waiting for
Read timeout 60 s Measurement-series downloads are large; a 30 s read timeout drops precisely the biggest, most valuable operations of the season

The costs are asymmetric, which is what should drive these numbers. A request that is retried unnecessarily costs a second; a record silently skipped by unstable pagination costs an entire operation’s yield data, and nothing downstream will ever report it as missing.

Gotchas and edge cases

  • pageSize is a page size, not a result limit. Setting it to 1000 does not fetch a thousand records in one call on most providers — it either caps silently at the provider’s maximum or times out on collections with complex boundary geometry. Between 100 and 250 is the practical band.
Why offset pagination loses records during harvest Two panels comparing offset-based and cursor-based pagination while the underlying collection is being written to. Offsets shift when a record is inserted or deleted, so pages repeat or skip records with no error; a cursor encodes a position and returns each item once. Offset pagination under concurrent writes A record is inserted while you walk Everything after it shifts forward by one. Page 4 repeats a record already seen on page 3. A deletion does the reverse and skips one. Silent: no error, wrong result set. Cursor pagination under the same writes The cursor encodes a position, not an index Inserts and deletes do not shift it. Each item is returned once. The cursor expires in minutes — consume it now. Stable: prefer it whenever both are offered.

  • A cursor expires, often within minutes. Storing one between scheduled runs and resuming from it later produces a 400 that reads like an authorisation failure. Checkpoint the last item identifier for progress reporting, and restart the walk from the beginning if the run is interrupted — the manifest makes re-reading cheap.

  • Per-grower and per-application limits are different budgets. Six concurrent requests spread across six growers may be fine while six against one grower is not. Where the provider distinguishes them, hold a semaphore per grower as well as the global one.

  • Retry-After can be a date, not an integer. The HTTP specification permits both forms. The code above only trusts the integer form and falls back to jitter otherwise, which is the safe direction — parsing a date wrong and sleeping until next Tuesday is worse than backing off for a few seconds.

  • asyncio.gather with hundreds of growers builds every coroutine up front. For large fleets, feed the work through a bounded queue instead so memory stays flat and a failure surfaces without waiting for the whole batch.

This guide is part of Machine Data APIs for Management-System Integration — see there for the resource tree, incremental sync and field matching this request layer serves.

Parent section Machine Data APIs for Management-System Integration Related Authenticating with the John Deere Operations Center API