I've made a script to get a list of projects from a ArcGIS REST end point, it works maybe 75% of the time but seems to suffer from time outs, possibly from making too many requests to the server?

I'm a bit ignorant on the server-side of things, but just wanted to see if I could make my code more efficient / make less requests as a workaround? I make go to the json for each individual project as a workaround so I'm wondering if there is a better approach for this step.

When it does fail, the error looks like below - because I assume the script can't reach the response json:

File "/home/script.py", line 82, in main
    if 'Name' in response_json['attributes']:
KeyError: 'attributes'

Code snippet below:

#construct URL with query 
service_type = 'Foo'
service_name = 'Bar'

prefix = f'https://my-domain.com/'
service = f'arcgis/rest/services/{service_type}/{service_name}/ImageServer/'
query = 'query?where=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=&returnGeometry=false&outSR=&returnIdsOnly=true&returnCountOnly=false&pixelSize=&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnDistinctValues=false&multidimensionalDefinition=&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&f=html'
token = '11111'

url_query = prefix + service + query + token

#Lists to populate
object_ids = []
projects = []

#Read rest end point using constructed URL
parser = 'html.parser'
page = urlopen(url_query)
soup = BeautifulSoup(page,parser,from_encoding=page.info().get_param('charset'))

#Get object id links and append to object list
for link in soup.find_all('a', href=True, text=re.compile('^[0-9]+$')):

project_count = 0

#Iterate through object id list, construct a url and append names to project list
for x in object_ids:
    url = prefix + str(x) + '?f=pjson' + token

    response_json = json.load(urlopen(url))

  • 2
    The HTML format for ArcGIS REST API is really meant for human consumption, I suggest working solely with JSON format for scripting and/or processing the information returned from ArcGIS REST API calls.
    – bixb0012
    Apr 12, 2022 at 1:32

1 Answer 1


Rethinking my entire approach landed me here - it works much better and reduces the amount of calls to the endpoint, resulting in hardly any timeouts. Credit to Ablajan Sulaiman for this method to bypassing the query limit which helped a lot.

My code:

        import requests
        import csv
        import os
        import datetime
        import click
        #construct URL 
        service_type = 'Foo'
        service_name = 'Bar'
        prefix = f'https://my-domain.com/'
        service = f'arcgis/rest/services/{service_type}/{service_name}/ImageServer/'
        token = '11111'
        #case handling for name field
        url = prefix + service + "?f=pjson" + token
        fields_json = requests.get(url).json()
        fields = fields_json['fields']
        name_field = f['name']
        # Get object ids of features
        where = "1=1"
        obj_query = f"query?where={where}&returnIdsOnly=true&f=json"

        urlstring = prefix + service + obj_query + token
        #identify number of records
        response = requests.get(urlstring)
        js = response.json()
        idfield = js["objectIdFieldName"]
        idlist = js["objectIds"]
        numrec = len(idlist)
        print(f"\nNumber of target records: {numrec}")
        # Gather features
        print("\nGathering records...")
        for i in range(0, numrec, maxrc):
            torec = i + (maxrc - 1)
            if torec > numrec:
                torec = numrec - 1
            fromid = idlist[i]
            toid = idlist[torec]
            where = f"{idfield} >= {fromid} and {idfield} <= {toid}"
            query = f"query?where={where}&returnGeometry=false&outFields={name_field}&f=json"
            print(f"  {where}")
            url_query = prefix + service + query + token
            response_json = requests.get(url_query).json()
            features = response_json['features']
            for f in features:
                name = f['attributes'][name_field]
            # Timestamp
        def timeStamped(fname, fmt='{fname}_%Y-%m-%d'):
            return datetime.datetime.now().strftime(fmt).format(fname=fname)
            #Add records to CSV
        with open((timeStamped(os.path.join(outpath, service_name)) + ".csv"), 'w', newline="") as file:
            wr = csv.writer(file)
            for project in projects:
                project_count += 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.