1

I am converting an excel sheet to .shp via pandas i.e. I have a dataframe with 150+ columns and ~900 records (increasing by about 50 every day), that I convert to geodataframe and export to shp:

output_data.to_file('myfilename.shp', driver='ESRI Shapefile')

Saving this data to shp in Global Mapper creates a 3.5MB dbf

In QGIS, a 14MB dbf

Using Geopandas 6.5MB dbf

I've seen that the geodataframe has 15 decimal places for every numeric field, including whole numbers. Same thing for QGIS. Global Mapper's field lengths are equal to the lengthiest field in each column.

How can I keep geopandas from adding 15 decimals? I assume this is what bloats the filesize since I refactored some fields in QGIS and got a much smaller output dbf. The fields are also shorter in pandas df, before converting to geodf.

Edit: Couldn't find a way to use GDAL RESIZE before saving as shp.. What I ended up doing:

export to shp using geopandas

output_data.to_file('myfilename.shp', driver='ESRI Shapefile')

resize using gdal/ogr

dbf = ogr.Open('myfilename.shp', 1)

dbf.ExecuteSQL('RESIZE myfilename', dialect='sqlite')

This took a 47MB dbf down to 27MB, however only shortened string fields.

8
  • 4 Methods to Round Values in Pandas DataFrame
    – gene
    Commented Feb 13, 2021 at 17:31
  • You may need to use some of the fiona options to specify a precision in the output shapefile. See fiona.readthedocs.io/en/stable/manual.html#writing-vector-data Does it have to be a shapefile? Can you save to a GeoPackage?
    – Spacedman
    Commented Feb 13, 2021 at 17:37
  • thanks @gene I have tried that, but as soon as the df is written to dbf the trailing digits reappear
    – Andre Geo
    Commented Feb 14, 2021 at 13:00
  • 1
    Section 1.6.2 shows how to set data type and length, eg "float:15.2" is a floating point number with two decimals points. What does ogrinfo show for the formats for your various shapefiles?
    – Spacedman
    Commented Feb 14, 2021 at 13:24
  • 1
    Probably quite large lengths are used by default if they are not explicitly set. As a comparison see the GDAL documentation gdal.org/drivers/vector/shapefile.html. GDAL also offers a special SQL function RESIZE.
    – user30184
    Commented Feb 14, 2021 at 13:33

1 Answer 1

0

geopandas uses fiona to save shp. You can use schema to define field length and precision. Or to preserve same schema as input shp. More info:

https://fiona.readthedocs.io/en/stable/manual.html#format-drivers-crs-bounds-and-schema

input_gdf = gpd.read_file(shp_path)
# original schema
with fiona.open(shp_path) as src:
    # Extract the schema
    schema = src.schema

# perform your stuff

input_gdf.to_file('res_path', schema=schema)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.