I am importing about 60 million records of csv data (about 25gb) to Postgresql server using ogr2ogr
After googling million times, I found out that I can make VRT file and uploading my csv file using ogr2ogr. Here's the code I used to importing data and vrt file
<OGRVRTDataSource> <OGRVRTLayer name="combine"> <SrcDataSource>D:\combine.csv</SrcDataSource> <GeometryType>wkbPoint</GeometryType> <LayerSRS>WGS84</LayerSRS> <GeometryField encoding="PointFromColumns" x="longitude" y="latitude"/> </OGRVRTLayer> </OGRVRTDataSource> ogr2ogr -overwrite -progress -gt 999999999 -skipfailures -f "PostgreSQL" PG:"host=000.000.000.000 port=0000 dbname=myDB user=me password=youtellme" -lco GEOMETRY_NAME=shape -t_srs EPSG:3857 D:\input.vrt -nln output
I thought I could adjust -gt option to speed up the importing time, yet it doesn't help much.
It has been 3 days since I started to importing data to Postgresql server and only 20 million data are uploaded so far. (means that It will take almost a week to upload)
Seems like my ogr2ogr code (or VRT) can only import 100 - 200 records per second depending on the server status (see the screen capture below)