I'm trying to find an optimum vector file format to be used to effectively store data (compressed binary?) and select subregions.
I have a quite large ascii point file containing 3 columns (x and y projected coordinates and the point ID). A test file is about 20MB, containing approximately 600000 points, but I plan to create even larger files with significantly more points. After writing those files to ascii, I'm searching for a way (i.e. file format) to store the data more effectively while still allowing for the following two tasks:
- the files shall be loaded into QGIS (v2.14.17)
- subregions shall be selected (e.g. via GDAL/OGR v2.1 "ogr2ogr -spat") and rewritten in the original ascii format
Do you have any suggestions, which file format shall be used here?
EDIT: Just to add some more information the data looks like
CoordX,CoordY,ID
1200400,-399600,1
1201200,-399600,2
1202000,-399600,3
1202800,-399600,4
...
I've created a VRT file as follows:
<OGRVRTDataSource>
<OGRVRTLayer name="points">
<SrcDataSource>points.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<LayerSRS>EPSG:3031</LayerSRS>
<GeometryField encoding="PointFromColumns" x="CoordX" y="CoordY"/>
<Field name="X" src="CoordX" type="integer" />
<Field name="Y" src="CoordY" type="integer" />
<Field name="ID" src="ID" type="integer" />
</OGRVRTLayer>
</OGRVRTDataSource>
As Luke suggested (What format to use for highly compressed vector data?) I tried SpatiaLite with compressed geometry:
ogr2ogr -f SQLite -dsco SPATIALITE=YES -lco COMPRESS_GEOM=YES -lco COMPRESS_COLUMNS=x,y,id points.sqlite points.vrt
but the result was a 90MB file (compared to 14MB of the now slightly modified CSV-Input).
Playing around a bit showed, that leaving away the -lco COMPRESS_*
did not have any effect on the 90MB file size. Instead, leaving away -dsco SPATIALITE=YES
resulted in a 26MB file (I know, this is not SpatiaLite any more). I'm working on Ubuntu 14.04. GDAL 2.1.0 was installed using the UbuntuGIS-PPA. Is there any additional package (besides libspatialite5 v4.1.1) necessary to enable compression in SpatiaLite?
x
andy
fields. You only need them in the geometry field, leaving only one integer column forid
. If you extract your data back to CSV, you may useGEOMETRY=AS_XY
to recover x and y.