Converting Shapefiles to GeoParquet with GeoPandas

TL;DR: Read the Shapefile with an explicit encoding, confirm its CRS is set and projected, rename any 10-character-truncated field names back to full tokens, then call gdf.to_parquet("out.parquet", compression="zstd", write_covering_bbox=True) and verify by reading it back and asserting CRS, row count, and attribute values match.

Why Convert Shapefiles to GeoParquet

A legacy Shapefile store is the single biggest drag on a scaled farm-data workflow. Every read walks the full DBF even when one column is needed, dense yield layers creep toward the 2 GB .shp/.dbf ceiling, attribute names over 10 characters are already truncated, and a missing .prj silently discards the CRS. GeoParquet fixes all four: columnar reads that touch only the columns and spatial windows a query names, several-fold smaller files, unlimited field names, and CRS embedded in one file. On a multi-season archive the read-time and storage difference is not marginal — filtered scans that took seconds against a Shapefile complete in a fraction of the time once the same data is columnar. This guide converts a field-boundary or yield-point Shapefile to GeoParquet without losing CRS, mangling encoding, or leaving truncated names unrepaired.

For where GeoParquet fits among the other pipeline formats, this guide sits under Geospatial File Formats for Farm Data Pipelines, and the format trade-offs are compared directly in GeoPackage vs GeoParquet vs Shapefile for farm data.

Four files with limits, or one file with statistics On the left, a shapefile is four separate sidecar files — geometry, index, attributes and projection — carrying a ten-character field-name limit, an ambiguous attribute encoding and a two-gigabyte size ceiling. On the right, a single GeoParquet file stores each attribute as a column chunk with per-row-group minimum and maximum statistics, so a filtered read skips groups without decoding them. Shapefile — four files that must travel together .shp geometry .shx index .dbf attributes .prj CRS, often absent 10-character field names · ambiguous encoding · 2 GB ceiling a missing sidecar loses the CRS or the attributes entirely to_parquet GeoParquet — one file, columnar, self-describing geometry (WKB) field_name yield_dry season Per-row-group statistics: min, max, null count for every column a season filter skips whole row groups without decoding them CRS travels inside the file — no sidecar to lose Long field names survive, encoding is UTF-8 by definition, and there is no practical size ceiling.

Prerequisites

Only the Parquet backend differs from the parent environment:

TEXT 
geopandas==0.14.4
pyogrio==0.9.0
pyarrow==16.1.0
BASH 
pip install geopandas==0.14.4 pyogrio==0.9.0 pyarrow==16.1.0

Input requirements:

  • A Shapefile set with all sidecars present — .shp, .shx, .dbf, and crucially .prj (the CRS). A .cpg declaring the attribute encoding is ideal but often absent.
  • Know your source encoding. Older FMIS and controller exports frequently use a legacy code page (latin-1 / cp1252) rather than UTF-8.

Step-by-Step

The two things a shapefile lost before you opened it Four conversion steps: read the shapefile with an explicitly declared attribute encoding, recover field names that the ten-character limit truncated, set the coordinate reference system from the projection sidecar or explicitly, and write GeoParquet with sensible row groups and compression. Read declare the encoding Recover names 10-char truncation Set the CRS from .prj, or explicitly Write Parquet row groups, compression Neither the encoding nor the original field names can be recovered from the file itself — they have to come from whoever produced it.

Step 1 — Read with an explicit encoding and confirm the CRS

Do not rely on the default encoding. If the DBF was written with a legacy code page and no .cpg, reading it as UTF-8 corrupts accented crop names and operator notes. Read with the known encoding, then assert the CRS is present and projected before doing anything else.

PYTHON 
import geopandas as gpd

SRC = "yield_points_2025.shp"

# Read with the source's real encoding (latin-1 is common in legacy exports)
gdf = gpd.read_file(SRC, engine="pyogrio", encoding="latin-1")

assert gdf.crs is not None, "Source .prj missing — set the CRS explicitly before writing"
print(f"CRS: EPSG:{gdf.crs.to_epsg()}  projected={gdf.crs.is_projected}")
print(f"Features: {len(gdf):,}  columns: {list(gdf.columns)}")

If the source .prj was missing, gdf.crs is None. Set the known CRS explicitly — never guess — using the reasoning in understanding CRS in precision agriculture:

PYTHON 
if gdf.crs is None:
    gdf = gdf.set_crs("EPSG:32615")  # replace with the field's actual UTM zone

Step 2 — Restore truncated field names

The Shapefile has already truncated any name longer than 10 characters. GeoParquet will faithfully carry whatever names it receives, so repair them now with a mapping you maintain. This is also the moment to catch collisions where two long names truncated to the same stub.

PYTHON 
# Map the truncated stubs the Shapefile actually stores back to full names
RESTORE = {
    "prescripti": "prescription_rate",
    "yield_bu_a": "yield_bu_ac",
    "moisture_p": "moisture_pct",
}
present = {old: new for old, new in RESTORE.items() if old in gdf.columns}
gdf = gdf.rename(columns=present)
print("Restored names:", present)

# Guard against a silent truncation collision
assert len(set(gdf.columns)) == len(gdf.columns), "Duplicate column names after rename"

Step 3 — Write GeoParquet

Write with zstd compression for the smallest file on repetitive farm attributes, WKB geometry encoding for portability, and write_covering_bbox=True so later spatial filters can skip row groups.

PYTHON 
gdf.to_parquet(
    "yield_points_2025.parquet",
    compression="zstd",
    geometry_encoding="WKB",
    write_covering_bbox=True,
)

Step 4 — Verify the round-trip

Read the output back with geopandas.read_parquet (not plain pandas) and assert that CRS, feature count, and a sorted checksum of a key column all match the source.

The complete, directly runnable script:

PYTHON 
import numpy as np
import geopandas as gpd

SRC = "yield_points_2025.shp"
DST = "yield_points_2025.parquet"
SRC_ENCODING = "latin-1"
FALLBACK_EPSG = 32615  # the field's actual UTM zone; only used if .prj is missing
RESTORE = {
    "prescripti": "prescription_rate",
    "yield_bu_a": "yield_bu_ac",
    "moisture_p": "moisture_pct",
}
KEY_COL = "yield_bu_ac"

# ── 1. Read with explicit encoding and confirm CRS ────────────────────────
gdf = gpd.read_file(SRC, engine="pyogrio", encoding=SRC_ENCODING)
if gdf.crs is None:
    gdf = gdf.set_crs(f"EPSG:{FALLBACK_EPSG}")
assert gdf.crs.is_projected, "Expected a projected CRS for metric farm analytics"
src_epsg = gdf.crs.to_epsg()

# ── 2. Restore truncated field names ──────────────────────────────────────
present = {old: new for old, new in RESTORE.items() if old in gdf.columns}
gdf = gdf.rename(columns=present)
assert len(set(gdf.columns)) == len(gdf.columns), "Column-name collision after rename"
assert KEY_COL in gdf.columns, f"Key column '{KEY_COL}' missing after rename"

n_src = len(gdf)
checksum_src = np.sort(gdf[KEY_COL].dropna().to_numpy())

# ── 3. Write GeoParquet ───────────────────────────────────────────────────
gdf.to_parquet(DST, compression="zstd", geometry_encoding="WKB",
               write_covering_bbox=True)

# ── 4. Read back and assert fidelity ──────────────────────────────────────
back = gpd.read_parquet(DST)
assert back.crs is not None and back.crs.to_epsg() == src_epsg, (
    f"CRS drift: {src_epsg} -> {back.crs.to_epsg() if back.crs else None}"
)
assert len(back) == n_src, f"Row count changed: {n_src} -> {len(back)}"
checksum_dst = np.sort(back[KEY_COL].dropna().to_numpy())
assert np.array_equal(checksum_src, checksum_dst), f"'{KEY_COL}' values changed on write"

print(f"OK: {n_src:,} features, EPSG:{src_epsg}, '{KEY_COL}' preserved")

# Prove the columnar read advantage: pull just geometry + one column
subset = gpd.read_parquet(DST, columns=["geometry", KEY_COL])
print(f"Column-subset read returned {len(subset):,} rows, "
      f"{len(subset.columns)} columns")

Running against a real yield-point layer prints the feature count, the preserved EPSG code, and confirmation that the key measured column is byte-for-byte identical after the round-trip — the proof that the conversion lost nothing an analytical stage depends on.

Gotchas & Edge Cases

  • Reading GeoParquet with pandas.read_parquet loses geometry and CRS. Always use geopandas.read_parquet. Plain pandas returns the WKB geometry as opaque bytes with no CRS attached, and a downstream spatial join silently fails.
  • A missing .prj yields crs=None, and the output inherits it. GeoParquet cannot invent a CRS. If Step 1 finds no CRS, you must set_crs the correct EPSG before writing — guessing WGS84 on data that is actually UTM corrupts every area and distance calculation downstream.
  • Truncated names are not recoverable from the file. The 10-character truncation is destructive and happened at Shapefile write time. Only a mapping you maintain restores the full names; there is no metadata in the .dbf that records the original.
  • write_covering_bbox needs a recent pyarrow/geopandas. On older stacks the argument is ignored and no bbox column is written, so later spatial predicate pushdown quietly does nothing. Confirm the column exists in the output schema if you rely on spatial filtering.
  • Very large layers should be written in row groups, not one block. For multi-million-point archives, let pyarrow chunk into row groups (the default) so readers can skip them; a single giant row group defeats predicate pushdown.
Four things the format lost that Parquet will preserve A table of four shapefile limitations and how each one surfaces in Python: field names truncated to ten characters, dates stored as text, an undeclared attribute encoding, and a missing projection sidecar. The last is the only one that should stop the conversion. What the shapefile did What arrives in Python What to do about it Truncated field names to ten characters yield_dry_ becomes ambiguous Map names explicitly on read Stored dates as text no date type in DBF Strings that sort wrongly Parse to datetime before writing Left encoding undeclared cp1252 or UTF-8? Mojibake in grower names Pass encoding= explicitly Wrote no .prj or an unparsable one crs is None Refuse the file, do not guess

Parent Guide

This guide is part of Geospatial File Formats for Farm Data Pipelines — see there for the full pipeline context, including where GeoParquet sits relative to GeoPackage, Shapefile, and Cloud Optimized GeoTIFF.

Parent section Geospatial File Formats for Farm Data Pipelines Related GeoPackage vs GeoParquet vs Shapefile for Farm Data