3

I'm using osm2pgrouting. It works for small datasets very well but when I run it on a large dataset, it start good but after some processing like 40% it starts giving following error

[*******************|                               ] (38%) Total processed: 1760000     Vertices inserted: 395 Split ways inserted 510
[*******************|                               ] (39%) Total processed: 1780000     Vertices inserted: 228 Split ways inserted 250
[*******************|                               ] (39%) Total processed: 1800000     Vertices inserted: 200 Split ways inserted 262
[********************|                              ] (40%) Total processed: 1820000
ERROR:  relation "__ways8380" does not exist

While processing FROM 1800000th          to: 1820000th way
count1820000 While processing FROM 1800000th     to: 1820000th way
[********************|                              ] (40%) Total processed: 1840000
ERROR:  relation "__ways8380" does not exist

While processing FROM 1820000th          to: 1840000th way
count1840000 While processing FROM 1820000th     to: 1840000th way
[********************|                              ] (40%) Total processed: 1860000
ERROR:  relation "__ways8380" does not exist

While processing FROM 1840000th          to: 1860000th way
count1860000 While processing FROM 1840000th     to: 1860000th way
[********************|                              ] (41%) Total processed: 1880000
ERROR:  relation "__ways8380" does not exist

While processing FROM 1860000th          to: 1880000th way
count1880000 While processing FROM 1860000th     to: 1880000th way
[********************|                              ] (41%) Total processed: 1900000

I'm running it on PostgreSQL13. Any ideas why it is giving this error.

1
  • I have the same problem! Doesn't work on IL or TX but works on smaller osm files like WY. Please let me know if you know some answer. Thanks. Feb 23, 2021 at 12:24

1 Answer 1

0

This problem ate many days of my time but here is how I am just able to fix it. Actually, there are two good ways:

  1. Download .pbf files https://extract.bbbike.org/ by creating polygons of areas of interest; overlapping and getting into outside of your area of interest should be fine. This way you can have multiple smaller .osm files to load via osm2pgrouting. Use Osmconvert utility to convert the .pbf files to .osm files with --drop-author and --drop-version flags to reduce the output .osm file sizes. Load them in a sequence via a .bat script using default osm2pgrouting options.

  2. Download your .pbf file from Geofabrik.de and use the Osmconvert utility to reduce the output .osm file size. Load using the usual osm2pgrouting process but put a flag like --chunk 40000 . This has worked for me for the State of Illinois where the .osm file was a little over 3 GB and had 3 million+ Ways. If the chunk of 40000 is still doesn't help then increase it: In case of TX load, I had to increase to 90000.

#2 is the much better than 1 but will require a lot of RAM. I have ran mine on a computer with 32 GB of RAM on a Windows machine with Core i7 10th generation processor and it barely made it--RAM usage was over 90%. It would fail in less RAM. Maybe look into using Virtual Memory to gain RAM via hard drive.

Hope this helps someone and also someone from osm2pgrouting would notice this issue which only manifests when the .osm file has many nodes.

Best luck!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.