I am using fiona and shapely together to buffer points and then merge them and write them to a shapefile. For almost all of my datasets, this is working fine. However, I am currently getting a MemoryError on the mapping(merged) call.
Here's the error that I'm getting:
Traceback (most recent call last):
File "C:\Working\Projects\USACE_Data\Toolbox\V4.1\Tools\ImportEHydro.py", line 149, in execute helper.ExcludeNewerDataWithinArea(lyrUSACE, fileTemp, fcSurveyJob, long(sordat.strftime("%Y%m%d")))
File "C:\Working\Projects\USACE_Data\Toolbox\V4.1\Tools\HelperClass.py", line 92, in ExcludeNewerDataWithinArea self.USACEMinimumBoundingPolygon(usaceLayer, newerDataPoly , "SORDAT")
File "C:\Working\Projects\USACE_Data\Toolbox\V4.1\Tools\HelperClass.py", line 609, in USACEMinimumBoundingPolygon firstBufferFC = self.BufferFeatureClass(tempCopy, "new_buffer", mean + (4*std), mergeType, mergeField)
File "C:\Working\Projects\USACE_Data\Toolbox\V4.1\Tools\HelperClass.py", line 540, in BufferFeatureClass 'geometry': mapping(merged)
File "C:\python\ArcGIS10.5\lib\site-packages\fiona\collection.py", line 353, in write self.writerecords([record])
File "C:\python\ArcGIS10.5\lib\site-packages\fiona\collection.py", line 347, in writerecords self.session.writerecs(records, self)
File "fiona\ogrext.pyx", line 1215, in fiona.ogrext.WritingSession.writerecs MemoryError
The relevant bits of code are below:
with collection(os.path.join(newShpDir, outputShp + ".shp"), "w", "ESRI Shapefile", schema) as output:
# Merge all of the features into one feature
for i in range(len(features)):
try:
merged = cascaded_union(features[i])
except:
arcpy.AddWarning(" Could not merge features for " + (str(featureList[i]) if len(featureList > 0) else "(unknown)") + "...skipping...")
continue
# write the output to a shapefile
dictProperties = {}
if mergeAll:
dictProperties = {mergeField : str(bufferDist)}
else:
dictProperties = {mergeField : featureList[i]}
output.write({
'properties': dictProperties,
'geometry': mapping(merged)
})
The error is happening at the 'geometry':mapping(merged)
It's trying to merge about 350k shapes into one. Anyone know if there is a limit to how many it can handle or if this is normal? I can handle it with an exception and then union some, merge, union some, merge, etc...but I'd rather not if I don't have to...