Concurrent requests while querying a map service returns duplicate sets

Question

I have a Python script that hits a query endpoint for a map service on an unfederated server. I am avoiding the arcgis library because it is not performant enough and its compatibility with geopandas is awful. So I'm resorting to using the requests and concurrent.futures libraries to query the API in parallel.

import concurrent.futures
import json
import requests

import geopandas as gpd

def from_api(url: str, endpoint: str = None, params: dict = {}) -> dict:
    api_url = f"{url}/{endpoint}" if endpoint else url
    resp = requests.get(api_url, params=params)
    try:
        txt = json.loads(resp.text)
    except json.JSONDecodeError:
        txt = {}
    return txt

# open data url
bfr_url = "https://maps.bouldercolorado.gov/arcgis/rest/services/fire/FireResponseTimesOpenData/MapServer/0"

# Get a count of the total number of records in the where clause, along with the max step allowed for pagination
where = "RESPONSEYEAR = 2022"
params = {
    "where": where,
    "returnCountOnly":True,
    "f": "geojson"
}
tot_records = from_api(url=bfr_url, endpoint="query", params=params)["count"] # 15,463
step = from_api(url=bfr_url, params={'f':'json'})["maxRecordCount"] # 1,000

I make sure to paginate the results so that I can get all records for a given where clause, and I also make sure to order the results so that each page theoretically consists of no duplicates.

# Query endpoint parameters (note that I order fields, and iteratively specify offsets)
params = {
    "where": where,
    "outFields": '*',
    "outSr": 4326,
    "f": "geojson",
    "orderByFields": "OBJECTID",
    "resultRecordCount": step
}
with concurrent.futures.ThreadPoolExecutor() as ex:
    futures = []
    for offset in range(0, tot_records, step):
        params["resultOffset"] = offset
        futures.append(ex.submit(from_api, bfr_url, "query", params))
    results = [f.result() for f in concurrent.futures.as_completed(futures)]
    features = [feat for r in results for feat in r["features"]]

gdf = gpd.GeoDataFrame.from_features(features, crs=4326)

I say "theoretically" because it doesn't do this at all: many of the concurrent threads return duplicated results despite carefully designating offsets. A quick gdf.OBJECTID.is_unique proves false.

Is there a problem with my code, or is this a server-side bug?

EDIT 1:

Attempting this API call serially does give the results I expect, as below:

params = {
    "where": where,
    "outFields": '*',
    "outSr": 4326,
    "f": "geojson",
    "orderByFields": "OBJECTID"
}
features = []
for offset in range(0, 3000, 1000): # first three pages
    params["resultOffset"] = offset
    gjson = from_api(bfr_url, "query", params)
    features += gjson["features"]

gdf2 = gpd.GeoDataFrame.from_features(features, crs=4326)
gdf2.OBJECTID.is_unique # returns True
len(gdf2) == tot_records # returns True

Answer 1

There are two different issues occurring, one is a client-side coding issue and the other is a server-side configuration issue. The fix for the client-side coding issue is simple once you understand what is happening and why. The fix for the server-side configuration issue is outside of your control, but you may be able to work around it, maybe.

Client-side Coding Issue

Looking to the client-side coding issue, the issue can be resolved by changing

futures.append(ex.submit(from_api, bfr_url, "query", params))

to

futures.append(ex.submit(from_api, bfr_url, "query", {**params}))

The change in the parameters argument, {**params}, is making a copy of the params dicttionary to prevent queued up jobs from having their parameters changed on them by future iterations through the loop.

From 4. More Control Flow Tools - Python 3.x documentation:

The actual parameters (arguments) to a function call are introduced in the local symbol table of the called function when it is called; thus, arguments are passed using call by value (where the value is always an object reference, not the value of the object).[1]

Footnotes

[1] Actually, call by object reference would be a better description, since if a mutable object is passed, the caller will see any changes the callee makes to it (items inserted into a list).

I think the footnote states it best, arguments in Python are called by object reference, which is a form of passing by reference. For immutable data types this behavior isn't an issue because if the original object is changed, then a new object is created even if the variable name stays the same because the data type is immutable. In this case with queued up threads, the old/original object remains with the function in the thread waiting to get processed while a new object is created in the main thread. For mutable data types, like a Python dictionary in this specific case, any changes to the object in the main thread will be seen in functions that are queued up and haven't run yet in the threads. The original code was getting duplicate data because it was making duplicate calls.

The mistake of passing a mutable data structure to the ThreadPoolExecutor would not have been noticed if the data set in question was smaller enough or the machine had enough CPUs that jobs never got queued up. So why did jobs get queued up and this issue surfaced? From the data set and code, it is known that 16 jobs are needed to retrieve all the data. From ThreadPoolExecutor — Launching parallel tasks - Python 3.x documentation, we know the "Default value of max_workers is changed to min(32, os.cpu_count() + 4)." Although it isn't rare to see personal machines with 12 or more CPU counts today, it is still quite common to see less. For a machine with 8 CPU count, the default max_workers would be 12. With 12 workers available, that means 4 jobs would be queued, and 4 duplicative calls made to the ArcGIS Server.

Server-side Configuration Issue

After making the client-side coding fix, I ran into the server-side configuration issue. Specifically, I noticed a job or two would sometimes fail to return results. Deeper inspection showed Cloudflare 524 errors (Troubleshooting Cloudflare 5XX errors - Cloudflare Help Center):

Error 524: a timeout occurred

Error 524 indicates that Cloudflare successfully connected to the origin web server, but the origin did not provide an HTTP response before the default 100 second connection timed out. This can happen if the origin server is simply taking too long because it has too much work to do - e.g. a large data query, or because the server is struggling for resources and cannot return any data in time.

Although there is obviously a Cloudfare configuration issue, I suspect that is being seen because of how the ArcGIS Server site is configured on the back-end. If an ArcGIS Server is configured to have a fairly low number of maximum ArcSOC processes for a service, and that service receives lots of requests in a very short period of time (let's say calls from a multi-threaded script to download data), the jobs will queue on the ArcGIS Server while it waits for previous calls to finish. The combination of queueing and waiting on the client and queueing and waiting on the server is probably creating a long enough delay to cause a timeout.

One way to possibly mitigate this from happening is to cut back on the number of threads making simultanous calls to ArcGIS Server. If less calls are coming in at the same time, there is a better chance an ArcSOC process will be available when the next round of calls is made. Therefore, I suggest cutting back on the max_workers and not going with more than 8. I would even start with 4.

Change

with concurrent.futures.ThreadPoolExecutor() as ex:

to

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as ex: