I'm trying to import a large dataset to Postgres, of about 600MB, that sits on a remote AWS server. But I get the following error

Not enough memory to allocate requested memory (nRecords=26923892).
Probably broken SHP file

I have a single core, 2GB ram machine. My question is, should I scale up the machine or if only I increase the memory allocation I get shp2pgsql to run smoothly even for big datasets (600MB and bigger)

  • What tool are you using to do this? have you tried shp2pgsql? What server type are you using? Jan 22 '16 at 16:35
  • I'm using shp2pgsql in a simple Azure Machine
    – tvieira
    Jan 22 '16 at 16:39
  • 3
    2Gb RAM isn't enough to run an operating system, much less a database. Datasets aren't measured in Mb so much as rows. 27m rows for a point shapefile doesn't overrun the storage format, but for line/polygon it would. Is it possible the error message is correct, and the shapefile is corrupt?
    – Vince
    Jan 22 '16 at 16:44
  • 1
    Yeah, it seems that it's going to be the case. It has been working for small datasets, like 100mb, be I'll be importing bigger and bigger datasets
    – tvieira
    Jan 22 '16 at 17:25
  • 1
    you can split the shapefiles into 6 100MB chunks and import them then append them in Postgres later. Not a lasting solution though.
    – yanes
    Jan 22 '16 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.