0

I am attempting to load FCC Mobile Deployment Form 477 Data into Google BigQuery for analysis and combining with other data I am using. I converted the shapefiles into new-line delimited geojsons using ogr2ogr and concatenated the files for each data release. Unfortunately, BigQuery has a 100MB limit for each row (feature). Initially, I tried to fix this by breaking multipolygons into single polygons, however there are still a large number of features that are much bigger than 100MB (at least one is 500MB). I've attempted simplification at multiple tolerances and the final size is still too large.

What's the best way to divide up the giant polygon features into smaller features that can be loaded into BigQuery?

Improve this question
2
  • Could you post specific error you get from BigQuery, and maybe detailed command you use with ogr2ogr? I think there could be various limits here, one is limit on input data, another is limit on BigQuery stored data - which might need different solutions.
    – Michael Entin
    Commented Feb 11, 2021 at 5:17
  • The BigQuery error is "Error while reading data, error message: JSON parsing error in row starting at position 518895576: Row size is larger than: 104857600" when I'm attempt to load a geojsonl into a table. The ogr2ogr command is just function shp2geojson() { echo "$1"; ogr2ogr -t_srs crs:84 "$1.geojsonl" "$1.shp"; } for var in *.shp; do shp2geojson ${var%\.*}; done For each of the folders with the extract shapefiles
    – Nathaniel Lovin
    Commented Feb 11, 2021 at 5:54

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.