Geospatial File Formats for Farm Data Pipelines

A farm data pipeline moves the same field through half a dozen container formats before a prescription reaches a controller: RTK-surveyed boundaries arrive as Shapefiles, yield monitor logs land as CSV or proprietary binaries, orthomosaics stream as GeoTIFFs, analysts stage millions of points in a columnar store, and the machine expects a Shapefile or ISOXML back. Pick the wrong container at any hop and you pay for it — truncated column names that corrupt prescription rates, encoding mojibake in operator notes, 2 GB write failures on a season of point data, or full-file scans where a column read would have sufficed. This guide is the decision layer for that pipeline: what Shapefile, GeoPackage, GeoParquet, GeoTIFF, and Cloud Optimized GeoTIFF actually store, where each one fits, and how to convert between them with geopandas, pyogrio, and pyarrow without losing CRS, geometry, or attributes.

This page is part of the Ag-GIS Data Fundamentals & Spatial Reference Systems section. See that overview for the surrounding context on coordinate systems, ingestion, and boundary handling that these formats carry between stages.

Prerequisites

Python packages (exact versions tested):

• geopandas==0.14.4
• pyogrio==0.9.0 (vectorised OGR read/write, the modern geopandas I/O engine)
• pyarrow==16.1.0 (Parquet and GeoParquet backend)
• shapely==2.0.4
• rasterio==1.3.10
• GDAL>=3.6 with GeoPackage, GeoParquet (Arrow), and COG driver support

pip install geopandas==0.14.4 pyogrio==0.9.0 pyarrow==16.1.0 shapely==2.0.4 rasterio==1.3.10

Input data assumptions:

• Vector inputs carry an explicit CRS. Never assume WGS84 — a boundary Shapefile may ship in a state-plane or UTM CRS, and a silent mismatch corrupts every downstream area and rate calculation. Verify with the workflow in understanding CRS in precision agriculture.
• Field boundaries are single- or multi-polygon geometries; yield monitor logs are dense point layers (10k–500k points per field-season at 1–2 Hz logging); prescription maps are attributed polygon layers.
• Rasters (orthomosaics, index surfaces) are tiled GeoTIFFs in a projected CRS such as EPSG:32615.
• GDAL>=3.5 is required for the GeoParquet driver; pyarrow is the more portable path used throughout this guide.

1. Formats & Their Structural Trade-offs

There is no universal best container. Each format encodes geometry, attributes, and CRS with a different physical layout, and that layout dictates read speed, write speed, file size, editability, and interoperability. A precision-ag pipeline touches all five below.

Shapefile is the lingua franca of farm equipment and remains the default import target for John Deere Operations Center, Trimble, Raven, and CNH controllers. It is not one file but a set: .shp (geometry), .shx (index), .dbf (attributes), and .prj (CRS as WKT), plus optional .cpg (attribute encoding). That multi-file nature is its first liability — drop the .prj and the CRS is gone; drop the .dbf and every attribute is gone. Three hard limits bite in agriculture specifically: attribute field names are capped at 10 characters in the DBF, so prescription_rate and prescription_zone both truncate to prescripti and collide; each of the .shp and .dbf components is capped at 2 GB, which a full season of 1 Hz yield points can approach; and DBF encoding defaults to a legacy code page, so operator notes with accented crop names or non-ASCII characters corrupt unless a .cpg declares UTF-8. Geometry is stored uncompressed, so files are large and read speed is mediocre.

GeoPackage (GPKG) is a single SQLite database with a standardised spatial schema (OGC GeoPackage). One .gpkg file can hold many named layers — boundaries, soil sampling points, and management zones side by side — plus attribute indexes, full-length UTF-8 column names, and a proper CRS table. It supports random-access reads and edits through SQL, which makes it the natural desktop-GIS working format; QGIS treats it as a first-class citizen. The cost is that it is row-oriented, so an analytical query that touches one column of a million-row point layer still walks every row’s full record.

GeoParquet stores vector features in Apache Parquet: a columnar, compressed, chunked format with per-column statistics. Geometry lives in a WKB column with CRS and encoding declared in file-level geo metadata. Two properties make it decisive for large farm datasets. First, columnar compression — dictionary and run-length encoding on repeated attribute values (crop code, zone id, season) plus Snappy or Zstd on the WKB — routinely shrinks a yield-point layer several fold versus Shapefile. Second, predicate and projection pushdown — because each row group carries min/max statistics per column, a reader can skip entire row groups that fall outside a filter and can read only the columns a query names, so read_parquet(..., columns=["geometry", "yield_bu_ac"]) never touches the other forty columns. It is an analytical store, not an editing or hand-off format.

GeoTIFF is the standard raster container: pixels plus an embedded CRS and affine transform (the geotransform). It carries orthomosaics, vegetation-index surfaces, and rasterised prescriptions. It supports internal tiling, overviews, and lossless (LZW, Deflate) or lossy compression.

Cloud Optimized GeoTIFF (COG) is a GeoTIFF with a disciplined internal layout — tiled, with overviews, and an IFD arranged so an HTTP range request can fetch just the tiles a window needs. For orthomosaics served to a web map or read window-by-window across a compute cluster, a COG turns a full-file download into a byte-range fetch. It is a strict superset of GeoTIFF, so any GeoTIFF reader opens a COG.

The rule of thumb across a farm pipeline: GeoParquet or GeoPackage for internal analytics and storage, Shapefile or ISOXML at the equipment boundary, COG for any raster you read partially or serve. The two child guides drill into the two decisions engineers hit most — the head-to-head format comparison for farm data and the mechanics of converting Shapefiles to GeoParquet with GeoPandas.

2. Step-by-Step Implementation

Step 1 — Inspect source format, CRS, and schema

Never convert blind. Read the metadata first and assert the CRS is present and projected before anything else.

import geopandas as gpd
import pyogrio

SRC = "field_boundaries.shp"

# List layers (a Shapefile has one; a GeoPackage may have many)
for layer in pyogrio.list_layers(SRC):
    print("layer:", layer)

gdf = gpd.read_file(SRC, engine="pyogrio")
assert gdf.crs is not None, "Source has no CRS (.prj missing?) — refuse to convert"
print(f"CRS: EPSG:{gdf.crs.to_epsg()}  projected={gdf.crs.is_projected}")
print(f"Features: {len(gdf)}  geom types: {sorted(gdf.geom_type.unique())}")
print("Columns:", list(gdf.columns))

# Flag any attribute name that will truncate in a Shapefile round-trip
long_names = [c for c in gdf.columns if c != "geometry" and len(c) > 10]
if long_names:
    print(f"WARNING: {len(long_names)} field names exceed 10 chars: {long_names}")

Step 2 — Map each pipeline stage to a target format

Decide the target by role, not by habit. Boundaries and zones that a person edits in QGIS → GeoPackage. Dense yield points scanned analytically → GeoParquet. Anything handed to a controller → Shapefile. Any raster read in windows → COG. The parameter table in section 3 encodes the driver options each choice implies.

Step 3 — Convert vector layers with geopandas and pyogrio drivers

geopandas exposes every OGR vector driver through to_file (with the pyogrio engine) and a dedicated to_parquet for GeoParquet. The layer argument names a GeoPackage layer; multiple to_file calls with mode="a" append additional layers into one .gpkg.

import geopandas as gpd

boundaries = gpd.read_file("field_boundaries.shp", engine="pyogrio")
zones = gpd.read_file("management_zones.shp", engine="pyogrio")
yield_pts = gpd.read_file("yield_2025.gpkg", layer="points", engine="pyogrio")

# Preserve the source CRS explicitly through every write
assert boundaries.crs == zones.crs, "Layers disagree on CRS — reproject before packing"

# --- GeoPackage: one container, multiple named layers ---
boundaries.to_file("farm_2025.gpkg", layer="boundaries", driver="GPKG", engine="pyogrio")
zones.to_file("farm_2025.gpkg", layer="zones", driver="GPKG", mode="a", engine="pyogrio")

# --- GeoParquet: columnar analytical store for the dense point layer ---
yield_pts.to_parquet(
    "yield_2025.parquet",
    compression="zstd",       # smaller than snappy for repetitive ag attributes
    geometry_encoding="WKB",  # portable across readers
    write_covering_bbox=True, # per-row bbox enables spatial predicate pushdown
)

# --- Shapefile: only at the equipment boundary; rename long fields first ---
rename_map = {"prescription_rate": "rx_rate", "prescription_zone": "rx_zone"}
zones.rename(columns=rename_map).to_file(
    "rx_for_controller.shp", driver="ESRI Shapefile", engine="pyogrio"
)

Step 4 — Write rasters as Cloud Optimized GeoTIFF

For orthomosaics and index surfaces, write a COG so downstream windowed reads and web serving are cheap. Use the GDAL COG driver or the rio-cogeo conventions: tiled, overviews, and predictor-aware compression.

import rasterio
from rasterio.shutil import copy as rio_copy

def to_cog(src_path: str, dst_path: str, compress: str = "deflate") -> str:
    """Rewrite a GeoTIFF as a validated COG, preserving CRS and transform."""
    with rasterio.open(src_path) as src:
        assert src.crs is not None and src.crs.is_projected, (
            f"Raster CRS must be projected; got {src.crs}"
        )
        profile = src.profile.copy()
        profile.update(
            driver="COG",
            compress=compress,
            predictor=2,          # horizontal differencing for continuous rasters
            blocksize=512,
            overview_resampling="average",
        )
    rio_copy(src_path, dst_path, **profile)
    return dst_path

to_cog("ortho_field7.tif", "ortho_field7_cog.tif")

Step 5 — Verify round-trip fidelity

Every conversion is a candidate for silent loss. Read the output back and assert CRS, feature count, and a checksum of the attribute values match the source. This step is expanded in section 5.

3. Key Parameters & Tuning

These are the driver and encoding parameters that materially change file size, read behaviour, and agronomic correctness across the formats above.

4. Edge Cases & Failure Modes

10-character field-name truncation silently reassigns rates. The most dangerous Shapefile failure in agriculture is not an error — it is silence. Writing a zone layer with seed_rate_target and seed_rate_actual produces two columns both truncated to seed_rate_; OGR appends a numeric suffix (seed_rate1), and a controller mapping expecting seed_rate_target reads the wrong column. Always inspect names before a Shapefile write (Step 1) and rename to explicit short tokens, keeping the mapping dictionary for restoration on the way back in.

Encoding corruption in the DBF. A .dbf without a .cpg sidecar is read using a legacy code page. Crop or variety names with accented characters, and operator free-text notes, become mojibake on the next read. Always pass encoding="UTF-8" on write and confirm a .cpg file appears alongside the .shp.

The 2 GB component ceiling on dense point data. A full season of 1 Hz yield monitoring across a large operation can push the .shp or .dbf past 2 GB, at which point the write fails or the file becomes unreadable. This is the clearest signal to move the analytical store to GeoParquet — the same data compresses several fold and has no such ceiling. Convert with the Shapefile-to-GeoParquet workflow.

CRS dropped in a multi-file Shapefile move. Copying a Shapefile by moving only the .shp loses the .prj, and the layer reads back with crs=None. Every consumer then guesses — usually wrong. GeoPackage and GeoParquet embed CRS in the single file, eliminating this class of bug. When you must ship a Shapefile, zip all sidecars together.

GeoParquet consumed by a tool that does not understand geo metadata. A plain pandas.read_parquet reads the WKB column as opaque bytes and ignores the CRS. Only geopandas.read_parquet (or another GeoParquet-aware reader) reconstructs geometries and CRS. If a downstream step lost the geometry, check that it used a spatial reader, not a plain Parquet reader.

Mixed geometry types in one layer. Shapefile permits only one geometry type per file; a layer mixing Polygon and MultiPolygon is coerced, and mixing Point and Polygon fails outright. GeoPackage and GeoParquet tolerate mixed types, but many controllers do not — normalise to a single type (e.g. explode() or promote all to MultiPolygon) before the equipment hand-off. Boundary geometry cleanup upstream is covered in field boundary extraction with GeoPandas.

5. Verification & Output Validation

Treat every format conversion as untrusted until a round-trip check passes. The function below reads the output back and asserts that CRS, feature count, and attribute content survived.

import geopandas as gpd
import numpy as np

def verify_vector_roundtrip(src_gdf: gpd.GeoDataFrame, dst_path: str,
                            layer: str | None = None,
                            key_cols: list[str] | None = None) -> dict:
    """Assert a converted vector file preserved CRS, count, and key attributes."""
    out = (gpd.read_parquet(dst_path) if dst_path.endswith(".parquet")
           else gpd.read_file(dst_path, layer=layer, engine="pyogrio"))

    # 1. CRS preserved (compare EPSG, not object identity)
    assert out.crs is not None, "Output lost its CRS"
    assert out.crs.to_epsg() == src_gdf.crs.to_epsg(), (
        f"CRS drift: {src_gdf.crs.to_epsg()} -> {out.crs.to_epsg()}"
    )

    # 2. Feature count preserved
    assert len(out) == len(src_gdf), f"Row count changed: {len(src_gdf)} -> {len(out)}"

    # 3. Key numeric attributes preserved within float tolerance
    report = {"crs_ok": True, "count_ok": True}
    for col in (key_cols or []):
        # Shapefile may have truncated the name; match on the first 10 chars
        match = col if col in out.columns else col[:10]
        assert match in out.columns, f"Attribute '{col}' missing after write"
        a = np.sort(src_gdf[col].to_numpy())
        b = np.sort(out[match].to_numpy())
        assert np.allclose(a, b, equal_nan=True), f"Values changed in '{col}'"
        report[f"{col}_ok"] = True
    return report

src = gpd.read_file("management_zones.shp", engine="pyogrio")
src.to_parquet("management_zones.parquet")
print(verify_vector_roundtrip(src, "management_zones.parquet",
                              key_cols=["zone_id"]))

For rasters, confirm the COG is valid and that the CRS and transform match the source pixel grid:

import rasterio

def verify_cog(src_path: str, cog_path: str) -> None:
    with rasterio.open(src_path) as a, rasterio.open(cog_path) as b:
        assert a.crs == b.crs, "COG CRS changed"
        assert a.transform == b.transform, "COG geotransform changed — pixels shifted"
        assert (a.width, a.height) == (b.width, b.height), "COG dimensions changed"
        assert b.overviews(1), "COG has no overviews — not cloud-optimized"
    print("COG verified: CRS, transform, dimensions, and overviews intact")

verify_cog("ortho_field7.tif", "ortho_field7_cog.tif")

A conversion that passes both checks has preserved everything a downstream stage depends on: spatial reference, geometry count, attribute values, and (for rasters) the pixel grid and overview pyramid.

6. Integration with the Pipeline

These formats are the connective tissue between every other stage of a precision-ag workflow.

Upstream — ingestion and boundaries. Boundaries arrive as Shapefiles from surveyors or FMIS exports and are cleaned and dissolved before storage. That cleanup, including geometry repair and CRS normalisation, is handled in field boundary extraction with GeoPandas; the output is best stored as a GeoPackage layer or GeoParquet file rather than round-tripped back to Shapefile.

CRS as a cross-cutting concern. Format choice never absolves you of CRS discipline. Every conversion here asserts the CRS is present and projected; the reasoning and the reprojection mechanics live in understanding CRS in precision agriculture. A format that embeds CRS in one file (GeoPackage, GeoParquet) removes an entire failure mode relative to the multi-file Shapefile.

Analytical middle — the columnar store. Once yield points, soil samples, and index summaries are staged in GeoParquet, downstream analytics read only the columns and the spatial windows they need. The conversion mechanics, including encoding and field-name handling, are the subject of converting Shapefiles to GeoParquet with GeoPandas.

Downstream — the equipment hand-off. The final step exports a Shapefile or ISOXML for the controller. Choosing between formats for a specific machine or FMIS is exactly the decision walked through in GeoPackage vs GeoParquet vs Shapefile for farm data.

Frequently Asked Questions

Why do my attribute column names get truncated when I write a Shapefile?

The Shapefile DBF attribute table limits field names to 10 characters, so a column such as prescription_rate becomes prescripti and silently collides with any other column sharing that prefix. Rename columns to short unique tokens before writing, or write to GeoPackage or GeoParquet, both of which allow full-length names. Keep a mapping dictionary so downstream code can restore the original names.

Is GeoParquet compatible with farm equipment controllers and FMIS platforms?

Not directly. Equipment controllers such as John Deere GreenStar, Trimble, and Raven, and most farm management information systems, still expect Shapefile or ISOXML at the import boundary. Use GeoParquet as the internal analytical store for its columnar compression and predicate pushdown, then export a Shapefile or ISOXML at the final hand-off step to the machine or the FMIS.

When should I use a GeoPackage instead of GeoParquet for farm data?

Use a GeoPackage when you need a single portable file that QGIS and desktop GIS tools open natively, when you want several related layers such as boundaries, sampling points, and zones in one container, and when random-access editing matters. Use GeoParquet when the workload is analytical scans over millions of yield points where columnar compression and column pruning cut read time and storage several fold.

Related

• GeoPackage vs GeoParquet vs Shapefile for Farm Data — the head-to-head decision matrix with per-use-case guidance and a benchmark script
• Converting Shapefiles to GeoParquet with GeoPandas — the step-by-step conversion preserving CRS, encoding, and field names
• Field Boundary Extraction with GeoPandas — cleaning and dissolving boundaries before they enter the format pipeline
• Understanding CRS in Precision Agriculture — the coordinate-system discipline every conversion in this guide depends on
• Ag-GIS Data Fundamentals & Spatial Reference Systems — the parent section covering ingestion, CRS, and boundary handling end to end