osm2pgsql: Failed to fork helper process

Question

I am trying to import planet.osm on am Amazon EC2 machine. It is a a machine with 16 processors and 122GB RAM.

The parameters I use are: osm2pgsql -s -H aws.rds.host -d osm -U user -C 92000 --number-processes 16 --cache-strategy dense --hstore --hstore-add-index -K -G -W -p osm --extra-attributes -v planet.osm.pbf

However, after the processing phase I get the following error:

WARNING: Failed to fork helper process 1: Trying to recover.
WARNING: Failed to fork helper process 2: Trying to recover.
...
WARNING: Failed to fork helper process 15: Trying to recover.

There is another question here with a memory issue (https://gis.stackexchange.com/questions/78879/osm-problem-with-memory), but this is not the error I get. There is no mention of "Cannot allocate memory".

Does anyone know of these issues or have any experience importing on EC2 machines?

Answer 1

I think you are hitting https://github.com/openstreetmap/osm2pgsql/issues/90 (i.e. this is a known issue on EC2).

A possible reason is that your cache size is way too high. Based on the osm2pgsql instructions, you should be using 24000.

Not specific to your question, but if you intend to render off this, remember that EC2 isn't a great database solution: see http://blog.gravitystorm.co.uk/2010/07/05/map-rendering-on-ec2/ for one user's experience.