Download Dataset From ArcGIS REST Service

Question

Looking at this information page for a dataset hosted on esri.com:
http://fema-services2.esri.com/arcgis/rest/services/2012_Sandy/ImageCat_NLT/MapServer/layers

Trying to figure out how I can get access to the raw data (either raw lat/lng coordinates or SHP).

It looks like the data is available, I just can't figure out how to pull it from this interface.

Answer 1

That map service is a ArcGIS Server Dynamic Map service which typically only returns images and specific query results, much like a WMS. Some ArcGIS Server image services allow for data download, but this isn't one of those.

You can get the information you're looking for through the query operation, but it will take multiple queries as the max record return limit on this service is 1000, and there are over 58000 records in the v14 layer alone.

To submit queries, go to the layer endpoint and scroll to the bottom for supported operations, then click Query. To find the total number of records, enter 1=1 in the Where field, and select True for Return Count Only, looks like 58919 records. It will only return 1000 records at a time, so you will have to issue 59 queries to get all the records. Seems easiest to filter by OBJECTID, so the first query would be WHERE OBJECTID < 1000, Out Fields = * , Return Geometry = TRUE, Return Count = False, and Format is HTML. The service returns all the features 1- 999 and their attributes. You could change the output format to KMZ and convert to desired format later, iterate through and you would have all the data.

Another method that would work is to programatically grab the individual feature information using the feature endpoint. In the HTML formatted response from above, the OBJECTID attribute is actually a hyperlink to the attributes for just that feature. The REST URL ends with the OBJECTID, so you could just increment that and grab each response in JSON to make it easier to parse.

NOTE - some of these features/links are only applicable to ArcGIS Server 10.1 REST endpoints.

Answer 2

I always find myself in situations where I need to export all the data from a Map Service into a shapefile. Here is a very easy utility to use that will export every feature from a service and save it as a shapefile and geojson if you need it. You will need to have or install node.js.

https://github.com/tannerjt/AGStoShapefile

Once you've put the files on your system, just navigate to the folder, add your map service to the services.txt file and run this from command line:

node AGStoSHP.txt services.txt ./output/

Make sure to put a pipe(|) in your services.txt file to add a name for your service.

Answer 3

Download data stored on an ArcGIS REST MapServer one layer at a time using the command line and the Python package pyesridump.

Example command:

esri2geojson http://gis.naperville.il.us/arcgis/rest/services/OpenData/OpenDataMapService/MapServer/4 naperville_parking_lots_122417.geojson

Answer 4

To download this data, you can visit https://geodatadownloader.com, specify your URL, pick an extent or a where clause if you like, then click "download". It'll save the output file to your computer, containing all features regardless of max query size.

I've had this issue many times, so I decided to a build an open source website to do this work automatically, and without any custom tool installation necessary (just load the website and point it to your layer). Example layer: https://gismaps.kingcounty.gov/arcgis/rest/services/Environment/KingCo_SensitiveAreas/MapServer/11

It does the work completely client side, so the layer's data is only ever seen by your browser. You can also run it yourself by cloning the github project.

It uses a similar method to what @wwnick described in his answer. The flow is:

1. Fetch <url>/query?where=1=1&returnIdsOnly=true
   1. This returns ALL object ids matching the query, with no limit. ESRI documentation mentions this "exploit", so it appears to be unofficially supported at the very least.
2. It then builds a query like: <url>/query?where=OBJECTID IN (1,2,3,...) for each "page" of results. It then immediately streams that data into a user specified file to avoid RAM limitations with large datasets.

Geodatadownloader supports GeoJSON, CSV, SHP and GPKG as export formats

Answer 5

The functionality you are looking for is that of a Web Feature Service (WFS), not a Web Map Service (WMS). Here's a little more info... http://blog.geoserver.org/2006/11/27/the-wfs-and-wms-services/

Answer 6

I had to do this recently and this was my best attempt so far. I was originally trying to do an "objectid non in {}".format(ids) where ids would be a tuple of collected objectid's but the url would not return any data, there must be a limit on how long the where clause string can be. some of this code is hard coded and if the ids are non-sequential than this script most likely won't work. but anyways I hope this helps guide

import os, arcpy, json, requests
arcpy.env.workspace=r'C:\path'
arcpy.env.overwriteOutput=True

def non_esri_extract(url,where,idlist):
    dic={"where": where,"outFields": "*","returnGeometry": "true","f":"json"}
    resp=requests.get(url, params=dic)
    data=resp.json()
    for i in data['features']:
        idlist.append(int(i['attributes']['OBJECTID']))
    maximum=max(idlist)
    minimum=min(idlist)
    return maximum,minimum

def esri_extract(url,e_w):
    fields="*"
    esri_param="?where={}&outFields={}&returnGeometry=true&f=json".format(e_w, fields)
    fsURL=url+esri_param
    fs = arcpy.FeatureSet()
    fs.load(fsURL)
    outname="interm"+str(x)
    arcpy.CopyFeatures_management(fs, outname)


x=0
maximum=0
minimum=1
baseURL="http://gismaps.vita.virginia.gov/arcgis/rest/services/VA_Base_layers/VA_Parcels/FeatureServer/0/query"

while maximum!=minimum:
    print "number of loops = {}".format(str(x))
    if x==0:
        ids=[]
        maximum,minimum=non_esri_extract(baseURL,"LOCALITY = 'Franklin County'",ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1
    else:
        ids=[]
        interm_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(maximum,maximum+999)
        maximum,minimum=non_esri_extract(baseURL,interm_where,ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1

fcs = arcpy.ListFeatureClasses()
arcpy.Merge_management(fcs, "Merged")

Answer 7

I know this is old, but I found an easier way using the ArcGIS Arcpy Python site package. If you don't need the GeoJSON/JSON and instead want the entire feature class as a result (or a subset using a where clause), you can just feature class to feature class it.

import arcpy

OutputLocation = "PATH_TO_OUTPUT_LOCATION"

Features = {
"USFS_ROADS_CLOSED" : "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/1",
"USFS_ROADS" : "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/0"
}

def Download_Feature(outfileName, url, outLocation):
  arcpy.FeatureClassToFeatureClass_conversion(url, OutputLocation, outfileName)

if __name__ == '__main__':
    for key, value in Features.items():
      print("Downloading: " + key)
      outfileName, url = key, value
      Download_Feature(outfileName, url, OutputLocation)

Any number of items can be put into this dictionary, which can then be iterated against downloading each one. Alternatively, you could download just the single feature class of interest like so:

import arcpy

OutputLocation = "PATH_TO_OUTPUT_LOCATION"

OutputName = "USFS_ROADS"

URL = "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/0"

arcpy.FeatureClassToFeatureClass_conversion(URL, OutputLocation, OutputName)

Answer 8

Easier yet: Just use the geoprocessing tool Feature Class to Geodatabase.

Answer 9

How about https://geodatadownloader.com/maps/create ? I found this site very useful.