I am trying to import whole planet data to my i7 3770k + postgres on SSD + 16gb RAM in order to render my custom tiles.

But the process is painfully slow and multi core is not helping (though you have specified)

osm2pgsql -d planet -U osm --slim -C 8000 --number-process 4 planet-120704.osm.bz2

From htop what I can see is osm2pgsql take 100% of 1 core and postgres takes only 10% of cpu.

and current prompt is:

Processing: Node(208320k 147.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)

Q: Is my computer too slow? Any method to speed up (without complexing too much hardware)?


I switched to imposm http://imposm.org/docs/imposm/latest/ for the same reason and everything worked out great. I was trying to install osm-bright for mapbox https://github.com/mapbox/osm-bright/ as all of the desired styling was present. Hope this helps out.

  • Did the same. But don't underestimate postgres config ;)
    – Styp
    Aug 13 '12 at 14:47
  • Martin, I'm a newb and always afraid to touch any config file. What can be changed in the config file?
    – geomajor56
    Aug 13 '12 at 15:03
  • try to set these 2 values as I have them. Makes pg use up to 8gb ram, which is the peak of efficency and performance in version 8.xx, shared_buffers = 4096MB, effective_cache_size = 8128MB - further don't forget kernel.shmmax, otherwise pg will not start. ;) That's it
    – Styp
    Aug 13 '12 at 15:07
  • Warning: imposm currently does not support differential updates. This means the database can't be updated with newer osm data after the import.
    – kontextify
    Sep 14 '15 at 7:55

I have a similar machine but with HDD not SSD and my speed for processing nodes is up to 12x faster than yours. Therefore, from your specs, I suspect that your configurations of both Postgres and osm2pgsql are not optimal. I have done large areas such as the whole of Europe a few times and well know how different settings on the same machine can massively alter performance!

Firstly, read about optimization for import. Note that the settings suggested here are for importing osm and not for general running of PostGIS (i.e. once you have finished your import, you may want to turn autovacuum back on etc). Some of the settings suggested in the 'NOObs' section actually make very little difference if you read the preceding tables, so configuring Postgresql for spatial (e.g. see here).

Once you have postgis optimised, for a big import shut everything else down and clear your buffers/cache etc. to free memory (or just reboot if you can). Now for the osm2pgsql settings. The first link will give you some of the settings and you can see a very handy reference and discussion here for all the osm2pgsql flags. If you have 'normal' HDD drives use flat-nodes and parallelization. This will give you the biggest speed boost along with getting postgis optimised correctly. Using SSD, it seems to help but less dramatically (though you ought to have much better speed than me because of that anyway). The osm2pgsql cache setting is (I believe) per process so if you have four processes and set -C to 8000 that is a cache of 32GB which is twice your RAM. If you have 8 cores, I'd suggest a setting of 6 processes with -C 2500 (= total of 15GB), which leaves 1GB for the operating system and other sundry tasks.

Lastly, I note that you are using the bzipped osm xml file. The bench-marking alluded to earlier states that:

the 'pbf' input reader reads OSM PBF files about twice as fast as the 'libxml2' parser parses an equivalent OSM XML file

It also states:

the experimental 'primitive' XML parser reads OSM XML input about 30% faster than the default 'libxml2' reader (but seams it currently as a bug in supporting some xml entities such as ' )

If the bug has been fixed then, use the reader flag in osm2pgsql to set it to primitive, but I always use PBF and not BZ.

See here for osm2pgsql for some bench-marking, now a little dated but valuable nonetheless as the various settings for postgres and osm2pgsql are listed for a variety of machines, with the times. Lastly don't panic if you have got a good speed out of the nodes and see the ways crawling along. In my experience, the k/s reading for ways slowly increases, but expect it to be at least an order of magnitude or more less than the speed for processing Nodes

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.