Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to import whole planet data to my i7 3770k + postgres on SSD + 16gb RAM in order to render my custom tiles.

But the process is painfully slow and multi core is not helping (though you have specified)

osm2pgsql -d planet -U osm --slim -C 8000 --number-process 4 planet-120704.osm.bz2

From htop what I can see is osm2pgsql take 100% of 1 core and postgres takes only 10% of cpu.

and current prompt is:

Processing: Node(208320k 147.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)

Q: Is my computer too slow? Any method to speed up (without complexing too much hardware)?

share|improve this question

I have a similar machine but with HDD not SSD and my speed for processing nodes is up to 12x faster than yours. Therefore, from your specs, I suspect that your configurations of both Postgres and osm2pgsql are not optimal. I have done large areas such as the whole of Europe a few times and well know how different settings on the same machine can massively alter performance!

Firstly, read about optimization for import. Note that the settings suggested here are for importing osm and not for general running of PostGIS (i.e. once you have finished your import, you may want to turn autovacuum back on etc). Some of the settings suggested in the 'NOObs' section actually make very little difference if you read the preceding tables, so configuring Postgresql for spatial (e.g. see here).

Once you have postgis optimised, for a big import shut everything else down and clear your buffers/cache etc. to free memory (or just reboot if you can). Now for the osm2pgsql settings. The first link will give you some of the settings and you can see a very handy reference and discussion here for all the osm2pgsql flags. If you have 'normal' HDD drives use flat-nodes and parallelization. This will give you the biggest speed boost along with getting postgis optimised correctly. Using SSD, it seems to help but less dramatically (though you ought to have much better speed than me because of that anyway). The osm2pgsql cache setting is (I believe) per process so if you have four processes and set -C to 8000 that is a cache of 32GB which is twice your RAM. If you have 8 cores, I'd suggest a setting of 6 processes with -C 2500 (= total of 15GB), which leaves 1GB for the operating system and other sundry tasks.

Lastly, I note that you are using the bzipped osm xml file. The bench-marking alluded to earlier states that:

the 'pbf' input reader reads OSM PBF files about twice as fast as the 'libxml2' parser parses an equivalent OSM XML file

It also states:

the experimental 'primitive' XML parser reads OSM XML input about 30% faster than the default 'libxml2' reader (but seams it currently as a bug in supporting some xml entities such as ' )

If the bug has been fixed then, use the reader flag in osm2pgsql to set it to primitive, but I always use PBF and not BZ.

See here for osm2pgsql for some bench-marking, now a little dated but valuable nonetheless as the various settings for postgres and osm2pgsql are listed for a variety of machines, with the times. Lastly don't panic if you have got a good speed out of the nodes and see the ways crawling along. In my experience, the k/s reading for ways slowly increases, but expect it to be at least an order of magnitude or more less than the speed for processing Nodes

share|improve this answer

I switched to imposm http://imposm.org/docs/imposm/latest/ for the same reason and everything worked out great. I was trying to install osm-bright for mapbox https://github.com/mapbox/osm-bright/ as all of the desired styling was present. Hope this helps out.

share|improve this answer
    
Did the same. But don't underestimate postgres config ;) – Styp Aug 13 '12 at 14:47
    
Martin, I'm a newb and always afraid to touch any config file. What can be changed in the config file? – geomajor56 Aug 13 '12 at 15:03
    
try to set these 2 values as I have them. Makes pg use up to 8gb ram, which is the peak of efficency and performance in version 8.xx, shared_buffers = 4096MB, effective_cache_size = 8128MB - further don't forget kernel.shmmax, otherwise pg will not start. ;) That's it – Styp Aug 13 '12 at 15:07
    
nice, thanks for the info – geomajor56 Aug 13 '12 at 17:50
    
Warning: imposm currently does not support differential updates. This means the database can't be updated with newer osm data after the import. – kontextify Sep 14 '15 at 7:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.