0

I am attempting to create a magnetic map of Northern Ireland using this dataset (https://www.opendatani.gov.uk/@geological-survey-of-northern-ireland/gsni-tellus-regional-airborne-geophysical-survey-magnetics), which is presented as .csv format. I've been attempting without luck for some time now, either with interpolation/rasterisation outputting blank files to problems with the sheer size of the raw .csv data (16 million rows).

I add the file to QGIS as normal, setting the x, y and what I want to use as the z value. Whenever I attempt to convert the .csv to a vector shapefile for interpolation, the associated .dbf file ends up being enormous (>100gbs) and causes trouble when trying to rasterize. At this point, I'm not sure if its something I'm doing wrong or something is amiss with the raw data itself.

Improve this question
5
  • dBase is limited to 4GiB. Shapefile is limited to 2GiB. You can't generate a valid 5-vertex polygon shapefile with 16M rows -- The limit is 15,790,320 rows (see How does shapefile 2GB limit equate to 70 million points)
    – Vince
    Commented Sep 7, 2023 at 16:51
  • Thank you. Perhaps I am not understanding the relationship between .dbf and .shp files, but the actual shapefile which I get from converting the .csv to a vector shapefile is only ~529mb, compared to the 105gb of the .dbf.
    – user229793
    Commented Sep 7, 2023 at 17:04
  • The dbf part does not have an upper size limit (at least in how GDAL deals with .dbf, see gdal.org/drivers/vector/shapefile.html#size-issues, "Attributes: The dbf format does not have any offsets in it, so it can be arbitrarily large"), but if the .shp part is truncated the shapefile is invalid.
    – user30184
    Commented Sep 7, 2023 at 17:15
  • The relationship between the .shp of a shapefile and the .dbf of a shapefile is by record number (variable and semi-fixed offsets, respectively). A standalone dBase file has no relation at all to shapefile. dBase also has a maximum width (4000 bytes/record), which doesn't correspond well to 16M rows at 105GiB.
    – Vince
    Commented Sep 7, 2023 at 17:35
  • 2
    By experience I know that if user tries to write too much data into shapefile with GDAL, once the .shp part has reached the size limit it naturally does not grow any more, but GDAL continues to add data into the .dbf file which can grow however big. The behavior is not intended and I have never checked if this, in a way orphaned, .dbf file is valid or not. The shapefile as a whole is anyway invalid. I suggest to convert the original csv file into GeoPackage.
    – user30184
    Commented Sep 7, 2023 at 18:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.