1

I have read that osm2pgrouting loads all the data into memory, which makes it faster.

On the other side the size of the .osm file which it can parse is limited.

How can I exactly know how much memory would I need to parse for example hamburg.osm into Postgres with osm2pgrouting? (hamburg.osm is 17 mb zipped and more than 100 mb unzipped).

Is that possible to calculate the required memory size at least approximately?

P.S. I have tried to parse a 300 Mb .osm file on a Linux machine with 512 Mb of RAM. It got to "Classes table created" and after that has not returned any "Finished" message. It also created 3 tables in the database - and populated the Nodes table with about 1.5 million nodes. (I have followed the the recent pgRouting workshop).

Improve this question
2
  • What kind of system are you running on? Do you know how to profile memory usage of an application on it? (512 RAM is not very much for this kind of application.)
    – Caleb
    Commented May 18, 2011 at 10:17
  • Ubuntu 9.04 LTS, 4-core CPU, 512 RAM. It was just a first try, I had no idea of how much memory I would need.
    – user10528
    Commented May 18, 2011 at 10:42

2 Answers 2

Reset to default
0

you can make few tests with small datasets and monitor memory use with the top command. You would then be able to determine how much memory you need.

See also these threads :

Improve this answer
4
  • I used htop, and here are the results: I passed a 1.1 Mb .osm file (city with roads) to osm2pgrouting and it took about 15 minutes to parse it. During the process the memory usage by that process was up to 60%, i.e. 307 Mb.
    – user10528
    Commented May 18, 2011 at 10:44
  • My apologies! I have made a mistake. The results are the following: I passed a 297 Mb file to osm2pgrouting and the memory usage was up to 60%, i.e. 307 Mb. So we can infer that the memory needed for osm2pgrouting to parse a file is roughly equal to the osm file's size.
    – user10528
    Commented May 18, 2011 at 11:06
  • Does it make any difference what is in the OSM file? What about a city export that has building and landmark data instead of just roads?
    – Caleb
    Commented May 18, 2011 at 11:23
  • I will report that if I have a chance to parse such kind of file (working with pgRouting as a beginner a bit tedious process)
    – user10528
    Commented May 18, 2011 at 13:25
0

The formula for this should be pretty simple with a bit of fuzzy for different data sets. I think the major factor would be how applicable the data in the osm file is. If it includes a bunch of data irrelevant to routing, hopefully the data that gets loaded up into ram would just be the relevant bits.

I would suggest trying this with some smaller data sets and profiling your system process to see how much ram it uses. Try a 20MB and a 50MB data set and check the memory trend yourself.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.