I'm trying to find an optimum vector file format to be used to effectively store data (compressed binary?) and select subregions.
I have a quite large ascii point file containing 3 columns (x and y projected coordinates and the point ID). A test file is about 20MB, containing approximately 600000 points, but I plan to create even larger files with significantly more points. After writing those files to ascii, I'm searching for a way (i.e. file format) to store the data more effectively while still allowing for the following two tasks:
- the files shall be loaded into QGIS (v2.14.17)
- subregions shall be selected (e.g. via GDAL/OGR v2.1 "ogr2ogr -spat") and rewritten in the original ascii format
Do you have any suggestions, which file format shall be used here?
EDIT: Just to add some more information the data looks like
CoordX,CoordY,ID 1200400,-399600,1 1201200,-399600,2 1202000,-399600,3 1202800,-399600,4 ...
I've created a VRT file as follows:
<OGRVRTDataSource> <OGRVRTLayer name="points"> <SrcDataSource>points.csv</SrcDataSource> <GeometryType>wkbPoint</GeometryType> <LayerSRS>EPSG:3031</LayerSRS> <GeometryField encoding="PointFromColumns" x="CoordX" y="CoordY"/> <Field name="X" src="CoordX" type="integer" /> <Field name="Y" src="CoordY" type="integer" /> <Field name="ID" src="ID" type="integer" /> </OGRVRTLayer> </OGRVRTDataSource>
As Luke suggested (What format to use for highly compressed vector data?) I tried SpatiaLite with compressed geometry:
ogr2ogr -f SQLite -dsco SPATIALITE=YES -lco COMPRESS_GEOM=YES -lco COMPRESS_COLUMNS=x,y,id points.sqlite points.vrt but the result was a 90MB file (compared to 14MB of the now slightly modified CSV-Input).
Playing around a bit showed, that leaving away the
-lco COMPRESS_* did not have any effect on the 90MB file size. Instead, leaving away
-dsco SPATIALITE=YES resulted in a 26MB file (I know, this is not SpatiaLite any more). I'm working on Ubuntu 14.04. GDAL 2.1.0 was installed using the UbuntuGIS-PPA. Is there any additional package (besides libspatialite5 v4.1.1) necessary to enable compression in SpatiaLite?