More efficient approach for processing large polygon dataset

Question

Using ArcGIS Desktop 10.6, I have the following workflow in a python script to try and calculate coverage area affected on a wireless network if certain towers went down:

1. User enters a list of polygon IDs (these are the tower outages)
2. Using an in-memory layer, script selects all polygons from a layer in a file geodatabase, except the ones listed by the user

Important note: Polygon layer is 500GB in size and cannot be generalized.
Contains complex geometries and often intentional overlap between polygons

3. Polygons from #2 are merged (arcpy.Merge_management)

4. Loop through all the polygons that were removed and clip any overlapping areas from merged result in #3

5. Merge output from #4 and calculate its area.

I am fairly confident that I have optimized the script as much as possible, and due to the size of the data, it is taking a significantly large time to process a result.

The question I have is what is a good alternative approach that will significantly reduce the processing time for the user but still accomplish the same workflow?

Things I am currently contemplating:

• Pre-cooking every combination of result using a union on the original layer. This may be a beast to process, but would only need to happen every ~6mths.
• Storing in SQL Geometry and using native SQL spatial functions to return results. Not tested, but I assume some of the more complex geoms might cause some issues.
• Raster based approach? Not put a lot of thought into this one.

Answer 1

Not sure I understand you completely, you want to find offline areas, not covered by any of the online antennas? Try the Union alternative you are thinking about and then Select By Attributes. The Union will be pretty slow but not as slow as merge-clip-merge every time you want to run the analysis and you only need to Union every six months. And SQL queries are fast.

Steps:

Split by Attributes (Object ID or some other ID) and then Union all outputs: Construct a Query using Python, select and sum:

import arcpy

fc = r'C:\Arenden\Default.gdb\Network_areas'
splitfield = 'SplitID'

offline_towers = [10,11,12]

offline_fields = ['FID_T'+str(f) for f in offline_towers]
online_fields = ['FID_T'+str(f[0]) for f in arcpy.da.SearchCursor(fc,splitfield) if f[0] not in offline_towers]

sql_list = []
for field in offline_fields:
    sql_list.append("{0}>-1 AND {1}".format(field,
                                ' AND '.join(['{0}<0'.format(f) for f in online_fields])))

sql = ' OR '.join(sql_list)

arcpy.SelectLayerByAttribute_management(in_layer_or_view="Network_areas_Union", 
                                       where_clause=sql)
print sum([i[0] for i in arcpy.da.SearchCursor("Network_areas_Union",'SHAPE@AREA')])