6

I would like to scrape geodata off three websites, which prove quite difficult for me.

  1. https://www.aldi-suisse.ch/filialen/ - Here I would like to get all stores locations from the map. I checked all the network traffic in Chrome, but there seems to be no location data sent.
  2. https://map.geo.admin.ch/?lang=en&selectedNode=&topic=inspire&layers=ch.astra.unfaelle-personenschaeden_alle&bgLayer=ch.swisstopo.pixelkarte-farbe&X=160125.00&Y=554500.00&zoom=1&layers_timestamp= - This one seems by far tougher. From what I can see, they are rendering the location markers server side into pictures, then overlay the maps with those pictures (no location data is sent at all). API calls are based on individual click locations on the map, making it impossible to click all locations.

Any ideas for either of those websites?

  • You may want to look into the scrapy, selenium or beautiful soup packages in python. Those are the common ones for scrapying, scraping an interactive map though would be quite the challenge. – GISKid May 13 '16 at 13:18
  • @GISKid Thanks for the help, but the solutions you mention are not really relevant here. It's not about scraping websites and parsing xpath, but getting data from a maps app. I could imagine a possible solution with selenium, but that would have to be very sophisticated (automatically searching icons on the map and clicking on them) – Ruehri May 13 '16 at 16:14
  • You might want to exercise caution when scraping as you typically need express permission to do so, the second website has a way to order the datasets from the specific gov agency help.geo.admin.ch/?id=39&lang=en – GISKid May 13 '16 at 17:54
  • You didn't look enough e.g yellowmap.de/Presentation/AldiSued/de-CH/… It's the clue to get the shop informations although as stated by other people, I'm not sure it's really allowed... – ThomasG77 May 13 '16 at 23:54
  • @ThomasG77 Thanks, I must have overlooked that, that's the answer for 1. Happy to upvote if you take it out of the comments as a real answer (can't do it in the comments) – Ruehri May 14 '16 at 9:40
3
  • For 1st case, you didn't look enough. You should look at this type of URL. It's the clue to get the shop informations although as stated by other people, I'm not sure it's really allowed...

  • For the 2nd case, you just need to reuse the URL calls your already discovered. Using a grid, you can get a series of regular coordinates to set geometry parameter in the URL. As it has a 10(px) tolerance, your grid should be every 5km (1px is approximatively 500m when looking all Switzerland in the map).

You can build grids using QGIS or https://github.com/Turfjs/turf-point-grid or other tools of your choice to create your data sets (I would recommand GeoJSON output, very simple to manipulate in any programming language).

You may need in both case to add a delay to your URL calls to avoid being banned by remote servers and also not putting an overload on a server you do not own...

  • The second solution sounds very cool, but not sure if I have the skills and time to do that at the moment. Can see that I would run into problems with pixel density at different zoom levels etc. Might come back if I do give it a try – Ruehri May 15 '16 at 21:00
1

It looks like this site is using google maps data to power it. If you go to google maps and search for Aldi (at the same extent) you can compare to check if this is true or not:

https://www.google.co.uk/maps/search/Aldi/@51.5256398,-0.2283121,11z/data=!3m1!4b1

If it is using google maps data then you can use the google maps geocoder api to scrape for supermarkets or whatever:

https://developers.google.com/maps/documentation/geocoding/intro

There are also pre packaged G-Maps scrapers around.

  • Unfortunately I do not think it is the same data source. It seems Aldi is only using map tiles from google, but not Google Places (one quick seach, Route de Gland 36, 1267 Vich does not show in Google Maps, but on Aldi's site). – Ruehri May 13 '16 at 16:11
  • Alternatively you could scrape Aldi's own store finder tool and geocode the results. aldi.co.uk/store-finder However, you might need to ask Aldi first. – Oliver Burdekin May 15 '16 at 17:05
  • Thanks, a couple of people have mentioned the legal concerns already, will look into that (and am happy about any input from others). For Aldi, ThomasG77 has already pointed out the API link where one can get JSONs directly (mixed with a lot of trash) – Ruehri May 15 '16 at 21:09
1

If you're just trying to pull the geographic data, there seems to be an API for the REST services that are feeding the map.

  • Thanks, this is the best answer so far, but doesn't fully hit the nail on its head. The API does not have all layers that are also on the map, e.g., the data in link in my original is not included in the API REST service, at least I cannot find it in the documentation... – Ruehri May 14 '16 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.