How to scrape/extract data from Esri ArcGIS from website

Question

I was trying to extract data from an ESRI map embedded in a website. The objective would be by introducing geographic coordinates to be able to access the values ​​present on the map.

I leave here a print of the map and the respective address. I just cannot understand which method I should use since the map is embedded in the site. What processes should I use?

*for academic purposes

https://zonamentopf.portaldasfinancas.gov.pt/simulador/default.jsp

Answer 1

What I usually do is look at the requests made by the browser. Press F12 to open the 'developer tools', and then select the Network tab. (I'm not sure if the names are the same in different browsers, but they all behave quite similar in any case)

When zooming around the map, you will see many requests that go to server.arcgisonline.com and services.arcgisonline.com, these are just the (imagery) background tiles and can be ignored. When you zoom in sufficiently, you will also see requests like this:

https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/export?dpi=96&transparent=true&format=png8&layers=show%3A1&bbox=-946807.1692328664%2C4890281.264462542%2C-931806.4024319208%2C4901364.633563878&bboxSR=102100&imageSR=102100&size=785%2C580&f=image

There are some things to note here:

1. A proxy is used (proxy.jsp). The actual ArcGIS service is in this case not accessible from outside, but only through this proxy. Everything you want to get from the service has to go through this
2. This is a request used to generate a PNG image of the current map extent for this particular service (note the part that says MapServer/export). If you need the actual data, you'll need to modify the request, see below
3. If you try to open this URL in a separate browser tab, it will give an error about an invalid 'referer'. The referer will be set by the browser to the URL of the current page. The easiest way to get around this is to use a tool such as Postman, and manually add a Referer header to the request

To get to the actual data (not an image) of this service, you need some knowledge about how to access an ArcGIS mapservice. In particular, you need to know how to query a layer: https://developers.arcgis.com/rest/services-reference/query-map-service-layer-.htm

Here's an example:

This service has 5 layers (Comércio, Habitação, Indústria, Serviços and Terrenos). Let's say you want data from the first one (index 0). A typical request would look like this: http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/0/query?f=json&where=1%3D1 (Of course this needs to be prefixed with the proxy url, and properly escaped)

This example will return the data of the first layer of this service in JSON format (KMZ and geojson are also available) with a where clause of 1=1, which basically means everything, but since this parameter is required you need to set it to something. Usually the number of returned records in a single request is limited by ArcGIS Server, so you'll only get 500 or 1000 records. If you need more, you should play around with the where, e.g. one request with objectid<1000, one with objectid>=1000 and objectid<2000 etcetera.