2

I using a local instance of the overpass api, because I need to make a lot of queries (over 100k). The solution I'm using right now is to call bin/osm3s_query for every single query.

This is very slow, because osm3s_query has to start, answer the query and shut down for every single query.

Is there a way of having some kind of daemon waiting for my queries?

I guess if osm3s_query doesn't have to start and shutdown for every single query, this should accelerate things a lot. The "daemon" for running a web server seems to do exactly what I'm doing right now (firing up one osm3s_query process for every query). Therefore I couldn't find anything helpful in the documentation

  • Can you elaborate a bit what kind of queries you're running (a share link from overpass-turbo.eu might be good), how much time they take, if you run several queries in parallel and most importantly, what exact Overpass API version do you have in use on your local instance. – mmd Jan 22 '16 at 17:01
  • Maybe, you want to take a look at and subscribe to the overpass dev list as well, as this might be a bit of a longer discussion after all: listes.openstreetmap.fr/wws/info/overpass - that's where the project owner also can respond to your question (he is not around on GIS SE). – mmd Jan 22 '16 at 17:02
  • I'm using v 0.7.52. My query is basically checking for traffic lights in a given bbox. overpass-turbo.eu/s/dVI I want to find out if a cyclist crossed a traffic light. I'm using gps tracks and I'm checking for every two gps coordinates if there is one or more traffic lights on the way. I'm doing this by adding a margin (20 meters at the moment) to the two coordinates I have, create a boundingBox with this and then I query my local overpass api. For now I'm just multithreading my queries (one process for every gps track I have) but I would not mind to improve further :-) – Felix Fiskare Jan 23 '16 at 14:07
  • I will have a look at the mailing list after the weekend, thanks for the tip :-) – Felix Fiskare Jan 23 '16 at 14:08
1

The best option in your case is probably to fetch all traffic light at once according to your gps track's bounding box. Cache that result locally in whatever language you're processing the Overpass API result and you're probably all set. Don't worry about the size, even a 300km by 300km area just takes some seconds to run.

Right now, there's no option in stock Overpass API to handle several requests in a row. Apache calls the interpreter binary via CGI for each single query. Usually, that's not that much of a concern, as most queries take more time, so start up costs are negligible.

BTW: There's also a FastCGI prototype for Overpass API on Github to reduce some of the startup costs. However, even with the support of FastCGI, each query usually has some specific fix cost in the range of maybe 50+ milliseconds, and as data needs to be fetched from disk (resp. buffer cache) and decompressed for each single query, you're better off with fewer but larger queries.

  • I need to do something similar, but I need "around" queries for about 6 million points. A large bounding box won't help here. – retorquere May 11 '18 at 16:32
  • @retorquere: please ask a new question with your complete use case description. Running 6 mio queries isn't an option for the existing public instances. – mmd May 12 '18 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.