Thanks to Vince for his advice. I tried to create a fishnet grid of 10x10 of the study area but got the error “ERROR 001018: Number of output features exceeds OID limit (2,147,483,647)“. I then tried a 100x100 grid and that ran for 20 mins before I stopped it. I then ran a 500x500 fishnet (took about 10 mins) and intersected it with the boundary polygon, ran ...
The issue is the cross join on non-intersection. Your query reads:
for every row in g, return every row in s that does not intersect with the current g
which seems in fact what you want, but about the worst performing operation on unfiltered and non-indexed relations in SQL.
Example: for 100 rows in g and 100 rows in s, and with no intersection between any ...
As you only have 20 more maps to go I would just shutdown ArcMap and restart. An ArcMap session holds onto all sorts of things you are probably unaware of and doing repetitive things just stacks up and slow things down.
I think you need to design your analysis to fit into memory and optimise so that you're only loading data from the Sentinel-5 data once.
Here's a suggestion:
Load all your polygons
Use the find_datasets command in the ODC to identify the extents of each dataset
Do an intersection with polygons and datasets, to group the polygons by dataset
Load one dataset, ...