18

Looking at this information page for a dataset hosted on esri.com:
http://fema-services2.esri.com/arcgis/rest/services/2012_Sandy/ImageCat_NLT/MapServer/layers

Trying to figure out how I can get access to the raw data (either raw lat/lng coordinates or SHP).

It looks like the data is available, I just can't figure out how to pull it from this interface.

27

That map service is a ArcGIS Server Dynamic Map service which typically only returns images and specific query results, much like a WMS. Some ArcGIS Server image services allow for data download, but this isn't one of those.

You can get the information you're looking for through the query operation, but it will take multiple queries as the max record return limit on this service is 1000, and there are over 58000 records in the v14 layer alone.

To submit queries, go to the layer endpoint and scroll to the bottom for supported operations, then click Query. To find the total number of records, enter 1=1 in the Where field, and select True for Return Count Only, looks like 58919 records. It will only return 1000 records at a time, so you will have to issue 59 queries to get all the records. Seems easiest to filter by OBJECTID, so the first query would be WHERE OBJECTID < 1000, Out Fields = * , Return Geometry = TRUE, Return Count = False, and Format is HTML. The service returns all the features 1- 999 and their attributes. You could change the output format to KMZ and convert to desired format later, iterate through and you would have all the data.

Another method that would work is to programatically grab the individual feature information using the feature endpoint. In the HTML formatted response from above, the OBJECTID attribute is actually a hyperlink to the attributes for just that feature. The REST URL ends with the OBJECTID, so you could just increment that and grab each response in JSON to make it easier to parse.

NOTE - some of these features/links are only applicable to ArcGIS Server 10.1 REST endpoints.

  • 3
    +1 Sometimes ObjectIDs are not sequential though (as with ArcSDE), so perhaps a more general solution would be to return IDs only which returns all 58919 ids. – Kirk Kuykendall Nov 7 '12 at 2:56
  • Good point, also good to note that this isn't exposed in the same way for tiled map services. – wwnick Nov 7 '12 at 4:01
  • @wwnick you said that "grab each response in JSON" when try to get data by object ID. But in this case data format is "pjson" or "json" but how to convert these two format into shapefil or others common geo-data format and what is easiest one? – SIslam May 24 '15 at 13:05
  • I would use OGR to convert json to whatever format you need. – wwnick May 26 '15 at 2:02
9

I always find myself in situations where I need to export all the data from a Map Service into a shapefile. Here is a very easy utility to use that will export every feature from a service and save it as a shapefile and geojson if you need it. You will need to have or install node.js.

https://github.com/tannerjt/AGStoShapefile

Once you've put the files on your system, just navigate to the folder, add your map service to the services.txt file and run this from command line:

node AGStoSHP.txt services.txt ./output/

Make sure to put a pipe(|) in your services.txt file to add a name for your service.

Hope this helps others who need this functionality.

  • This answer is identical to one posted on gis.stackexchange.com/questions/98485/… Identical answers usually means that the questions they are being applied to should be duplicates. If that is the case please flag them as such. – PolyGeo Sep 12 '15 at 5:57
  • Thanks, I flagged the other question as duplicate. They are asked in different ways, but basically the same. – jOshT Sep 15 '15 at 21:08
4

Download data stored on an ArcGIS REST MapServer one layer at a time using the command line and the Python package pyesridump.

Example command:

esri2geojson http://gis.naperville.il.us/arcgis/rest/services/OpenData/OpenDataMapService/MapServer/4 naperville_parking_lots_122417.geojson
  • can you elaborate more on this method? – NULL.Dude Apr 17 '18 at 11:38
  • 1
    @Geo.Dude To use the command that starts with "esri2geojson", you will need to install the Python package called "pyesridump" via the command line. After installing that you can copy and paste the command that starts with "esri2geojson" into your command line (on a Mac you can use the built-in Terminal app) and press Enter key to run the command. – stevevance Apr 18 '18 at 15:05
0

I had to do this recently and this was my best attempt so far. I was originally trying to do an "objectid non in {}".format(ids) where ids would be a tuple of collected objectid's but the url would not return any data, there must be a limit on how long the where clause string can be. some of this code is hard coded and if the ids are non-sequential than this script most likely won't work. but anyways I hope this helps guide

import os, arcpy, json, requests
arcpy.env.workspace=r'C:\path'
arcpy.env.overwriteOutput=True

def non_esri_extract(url,where,idlist):
    dic={"where": where,"outFields": "*","returnGeometry": "true","f":"json"}
    resp=requests.get(url, params=dic)
    data=resp.json()
    for i in data['features']:
        idlist.append(int(i['attributes']['OBJECTID']))
    maximum=max(idlist)
    minimum=min(idlist)
    return maximum,minimum

def esri_extract(url,e_w):
    fields="*"
    esri_param="?where={}&outFields={}&returnGeometry=true&f=json".format(e_w, fields)
    fsURL=url+esri_param
    fs = arcpy.FeatureSet()
    fs.load(fsURL)
    outname="interm"+str(x)
    arcpy.CopyFeatures_management(fs, outname)


x=0
maximum=0
minimum=1
baseURL="http://gismaps.vita.virginia.gov/arcgis/rest/services/VA_Base_layers/VA_Parcels/FeatureServer/0/query"

while maximum!=minimum:
    print "number of loops = {}".format(str(x))
    if x==0:
        ids=[]
        maximum,minimum=non_esri_extract(baseURL,"LOCALITY = 'Franklin County'",ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1
    else:
        ids=[]
        interm_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(maximum,maximum+999)
        maximum,minimum=non_esri_extract(baseURL,interm_where,ids)
        esri_where="objectid >={} and objectid <={} and LOCALITY = 'Franklin County'".format(minimum,maximum)
        esri_extract(baseURL,esri_where)
        x+=1

fcs = arcpy.ListFeatureClasses()
arcpy.Merge_management(fcs, "Merged")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.