2

I using a local instance of the overpass api, because I need to make a lot of queries (over 100k). The solution I'm using right now is to call bin/osm3s_query for every single query.

This is very slow, because osm3s_query has to start, answer the query and shut down for every single query.

Is there a way of having some kind of daemon waiting for my queries?

I guess if osm3s_query doesn't have to start and shutdown for every single query, this should accelerate things a lot. The "daemon" for running a web server seems to do exactly what I'm doing right now (firing up one osm3s_query process for every query). Therefore I couldn't find anything helpful in the documentation

4
  • Can you elaborate a bit what kind of queries you're running (a share link from overpass-turbo.eu might be good), how much time they take, if you run several queries in parallel and most importantly, what exact Overpass API version do you have in use on your local instance.
    – mmd
    Jan 22, 2016 at 17:01
  • Maybe, you want to take a look at and subscribe to the overpass dev list as well, as this might be a bit of a longer discussion after all: listes.openstreetmap.fr/wws/info/overpass - that's where the project owner also can respond to your question (he is not around on GIS SE).
    – mmd
    Jan 22, 2016 at 17:02
  • I'm using v 0.7.52. My query is basically checking for traffic lights in a given bbox. overpass-turbo.eu/s/dVI I want to find out if a cyclist crossed a traffic light. I'm using gps tracks and I'm checking for every two gps coordinates if there is one or more traffic lights on the way. I'm doing this by adding a margin (20 meters at the moment) to the two coordinates I have, create a boundingBox with this and then I query my local overpass api. For now I'm just multithreading my queries (one process for every gps track I have) but I would not mind to improve further :-) Jan 23, 2016 at 14:07
  • I will have a look at the mailing list after the weekend, thanks for the tip :-) Jan 23, 2016 at 14:08

1 Answer 1

1

The best option in your case is probably to fetch all traffic light at once according to your gps track's bounding box. Cache that result locally in whatever language you're processing the Overpass API result and you're probably all set. Don't worry about the size, even a 300km by 300km area just takes some seconds to run.

Right now, there's no option in stock Overpass API to handle several requests in a row. Apache calls the interpreter binary via CGI for each single query. Usually, that's not that much of a concern, as most queries take more time, so start up costs are negligible.

BTW: There's also a FastCGI prototype for Overpass API on Github to reduce some of the startup costs. However, even with the support of FastCGI, each query usually has some specific fix cost in the range of maybe 50+ milliseconds, and as data needs to be fetched from disk (resp. buffer cache) and decompressed for each single query, you're better off with fewer but larger queries.

2
  • I need to do something similar, but I need "around" queries for about 6 million points. A large bounding box won't help here.
    – retorquere
    May 11, 2018 at 16:32
  • @retorquere: please ask a new question with your complete use case description. Running 6 mio queries isn't an option for the existing public instances.
    – mmd
    May 12, 2018 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.