29

Looking at this information page for a dataset hosted on esri.com:
http://fema-services2.esri.com/arcgis/rest/services/2012_Sandy/ImageCat_NLT/MapServer/layers

Trying to figure out how I can get access to the raw data (either raw lat/lng coordinates or SHP).

It looks like the data is available, I just can't figure out how to pull it from this interface.

0

8 Answers 8

39

That map service is a ArcGIS Server Dynamic Map service which typically only returns images and specific query results, much like a WMS. Some ArcGIS Server image services allow for data download, but this isn't one of those.

You can get the information you're looking for through the query operation, but it will take multiple queries as the max record return limit on this service is 1000, and there are over 58000 records in the v14 layer alone.

To submit queries, go to the layer endpoint and scroll to the bottom for supported operations, then click Query. To find the total number of records, enter 1=1 in the Where field, and select True for Return Count Only, looks like 58919 records. It will only return 1000 records at a time, so you will have to issue 59 queries to get all the records. Seems easiest to filter by OBJECTID, so the first query would be WHERE OBJECTID < 1000, Out Fields = * , Return Geometry = TRUE, Return Count = False, and Format is HTML. The service returns all the features 1- 999 and their attributes. You could change the output format to KMZ and convert to desired format later, iterate through and you would have all the data.

Another method that would work is to programatically grab the individual feature information using the feature endpoint. In the HTML formatted response from above, the OBJECTID attribute is actually a hyperlink to the attributes for just that feature. The REST URL ends with the OBJECTID, so you could just increment that and grab each response in JSON to make it easier to parse.

NOTE - some of these features/links are only applicable to ArcGIS Server 10.1 REST endpoints.

4
  • 3
    +1 Sometimes ObjectIDs are not sequential though (as with ArcSDE), so perhaps a more general solution would be to return IDs only which returns all 58919 ids. Nov 7, 2012 at 2:56
  • Good point, also good to note that this isn't exposed in the same way for tiled map services.
    – wwnick
    Nov 7, 2012 at 4:01
  • @wwnick you said that "grab each response in JSON" when try to get data by object ID. But in this case data format is "pjson" or "json" but how to convert these two format into shapefil or others common geo-data format and what is easiest one?
    – SIslam
    May 24, 2015 at 13:05
  • I would use OGR to convert json to whatever format you need.
    – wwnick
    May 26, 2015 at 2:02
17

I always find myself in situations where I need to export all the data from a Map Service into a shapefile. Here is a very easy utility to use that will export every feature from a service and save it as a shapefile and geojson if you need it. You will need to have or install node.js.

https://github.com/tannerjt/AGStoShapefile

Once you've put the files on your system, just navigate to the folder, add your map service to the services.txt file and run this from command line:

node AGStoSHP.txt services.txt ./output/

Make sure to put a pipe(|) in your services.txt file to add a name for your service.

2
  • This answer is identical to one posted on gis.stackexchange.com/questions/98485/… Identical answers usually means that the questions they are being applied to should be duplicates. If that is the case please flag them as such.
    – PolyGeo
    Sep 12, 2015 at 5:57
  • 1
    Thanks, I flagged the other question as duplicate. They are asked in different ways, but basically the same.
    – jOshT
    Sep 15, 2015 at 21:08
8

Download data stored on an ArcGIS REST MapServer one layer at a time using the command line and the Python package pyesridump.

Example command:

esri2geojson http://gis.naperville.il.us/arcgis/rest/services/OpenData/OpenDataMapService/MapServer/4 naperville_parking_lots_122417.geojson
1
  • Really handy, worked well
    – Matt
    Nov 16, 2021 at 16:18
3

To download this data, you can visit https://geodatadownloader.com, specify your URL, pick an extent or a where clause if you like, then click "download". It'll save a geojson file to your computer, containing all features regardless of max query size.

I've had this issue many times, so I decided to a build an open source website to do this work automatically, and without any custom tool installation necessary (just load the website and point it to your layer). It does the work completely client side, so the layers data is only ever seen by your browser. You can also run it yourself by cloning the github project.

It uses a similar method to what @wwnick described in his answer. The flow is:

  1. Fetch <url>/query?where=1=1&returnIdsOnly=true
    1. This returns ALL object ids matching the query, with no limit. ESRI documentation mentions this "exploit", so it appears to be unofficially supported at the very least.
  2. It then builds a query like: <url>/query?where=OBJECTID IN (1,2,3,...) for each "page" of results. It then immediately streams that data into a user specified file to avoid RAM limitations with large datasets.

Caveats:

  • You can only use a browser that supports the Window.showSaveFilePicker() API. Right now that means only Chrome, Edge and Opera
  • Right now it only supports geojson as the output format, so if you need other formats you'll need to use something like ogr2ogr to convert it. Hopefully support will be added later.
1
  • This downloader tool worked great for me. Thank you Myles for building and maintaining this excellent tool. I was not able to get the URL working in QGIS as an arcgis feature server layer, but the same URL worked in your downloader with the Opera browser. For those doing this, I found the layer's URL using the browser's "inspect element" on right-click menu > network > reloaded page > filtered results to "feature" > then copied and pasted the URLs into notepad and looked for the one with the name of the layer I was looking to download > shortened the URL to end in "/FeatureServer/0/query"
    – JJD
    Jun 20 at 16:46
2

The functionality you are looking for is that of a Web Feature Service (WFS), not a Web Map Service (WMS). Here's a little more info... http://blog.geoserver.org/2006/11/27/the-wfs-and-wms-services/

0
1

I had to do this recently and this was my best attempt so far. I was originally trying to do an "objectid non in {}".format(ids) where ids would be a tuple of collected objectid's but the url would not return any data, there must be a limit on how long the where clause string can be. some of this code is hard coded and if the ids are non-sequential than this script most likely won't work. but anyways I hope this helps guide

import os, arcpy, json, requests
arcpy.env.workspace=r'C:\path'
arcpy.env.overwriteOutput=True

def non_esri_extract(url,where,idlist):
    dic={"where": where,"outFields": "*","returnGeometry": "true","f":"json"}
    resp=requests.get(url, params=dic)
    data=resp.json()
    for i in data['features']:
        idlist.append(int(i['attributes']['OBJECTID']))
    maximum=max(idlist)
    minimum=min(idlist)
    return maximum,minimum

def esri_extract(url,e_w):
    fields="*"
    esri_param="?where={}&outFields={}&returnGeometry=true&f=json".format(e_w, fields)
    fsURL=url+esri_param
    fs = arcpy.FeatureSet()
    fs.load(fsURL)
    outname="interm"+str(x)
    arcpy.CopyFeatures_management(fs, outname)


x=0
maximum=0
minimum=1
baseURL="http://gismaps.vita.virginia.gov/arcgis/rest/services/VA_Base_layers/VA_Parcels/FeatureServer/0/query"

while maximum!=minimum:
    print "number of loops = {}".format(str(x))
    if x==0:
        ids=[]
        maximum,minimum=non_esri_extract(baseURL,"LOCALITY = 'Franklin County'",ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1
    else:
        ids=[]
        interm_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(maximum,maximum+999)
        maximum,minimum=non_esri_extract(baseURL,interm_where,ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1

fcs = arcpy.ListFeatureClasses()
arcpy.Merge_management(fcs, "Merged")
0

I know this is old, but I found an easier way using the ArcGIS Arcpy Python site package. If you don't need the GeoJSON/JSON and instead want the entire feature class as a result (or a subset using a where clause), you can just feature class to feature class it.

import arcpy

OutputLocation = "PATH_TO_OUTPUT_LOCATION"

Features = {
"USFS_ROADS_CLOSED" : "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/1",
"USFS_ROADS" : "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/0"
}

def Download_Feature(outfileName, url, outLocation):
  arcpy.FeatureClassToFeatureClass_conversion(url, OutputLocation, outfileName)

if __name__ == '__main__':
    for key, value in Features.items():
      print("Downloading: " + key)
      outfileName, url = key, value
      Download_Feature(outfileName, url, OutputLocation)

Any number of items can be put into this dictionary, which can then be iterated against downloading each one. Alternatively, you could download just the single feature class of interest like so:

import arcpy

OutputLocation = "PATH_TO_OUTPUT_LOCATION"

OutputName = "USFS_ROADS"

URL = "https://services1.arcgis.com/gGHDlz6USftL5Pau/ArcGIS/rest/services/R03_Roads_Data/FeatureServer/0"

arcpy.FeatureClassToFeatureClass_conversion(URL, OutputLocation, OutputName)
0

Easier yet: Just use the geoprocessing tool Feature Class to Geodatabase.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.