1

I have a geodatabase with 300 featureclasses with around 4.5 million features in each featureclass. I am updating each featureclass with a value calculated through an algorithm.

The IDLE ide for arcpy script freezes after running for 20 featureclasses. Then I need to forcibly close the script.

How can memory management be done for such a huge dataset?

Improve this question
7
  • If your first 20 feature classes run successfully and the freeze happens on the 21st, what happens when you re-order your feature classes so that the 21st one is processed first? If it freezes, then let's forget about the other 299 and focus this question on that one.
    – PolyGeo ♦
    Nov 26, 2015 at 11:55
  • Arcpy scripts run-time gets slower when running from one feature class to the next. i.e; time taken for first feature class is not equal to the time taken of the 20th featureclass. How can we clean up memory after running one featureclass sothat for each feature class it runs efficiently. I have already used arcpy.delete_management for deleting the temporary layers.
    – anamika singh
    Nov 26, 2015 at 12:01
  • 1
    Are you sure that the arcpy script freeze? I had several time the issue that just the outputs freezes but the script is still running in the background. Try to run the script just over 25 featureclasses and check if it comes to an end. You may could split into 15 featureclasse packages.
    – wittich
    Nov 26, 2015 at 12:14
  • You seemed to avoid my question. Without seeing a code snippet that can be used to reproduce what you are describing, it makes suggesting what to change hard.
    – PolyGeo ♦
    Nov 26, 2015 at 12:34
  • Yes the script freezed. waited for about 3 hours. the system wasn't responding either. it needed a restart. I think i should try to run on 25 feature classes only and check for the output. @ polygeo the codesnippet we work on is not internet connection so would be not able to post.
    – anamika singh
    Nov 26, 2015 at 12:54

1 Answer 1

Reset to default
1

One thing I would certainly recommend, is to index all fields referenced in Definition Queries of every feature class participating in your batch process.

I have a mixed ModelBuilder / ArcPy tool that regularly churns out 400 feature classes in a single File Geodatabase, the biggest of which was an OpenStreetMap dataset for entire France that resulted in 400 thematic Feature Classes and total file size for the File Geodatabase of 271 GB, and largest Feature Class with almost 42M records. Actually, one of the base Feature Classes that represents the source data for the creation of this large set of Feature Classes, is a point dataset with over 334M records.

This is all running on a modest W7, Core i5 quadcore desktop with 16GB RAM (most of which is actually of no use to ArcGIS for Desktop, since it can't use it like 64 bit ArcGIS Pro).

Making sure all relevant attribute fields had proper indexes, proved to make a real difference on performance. That said, ArcGIS can still appear to "hang" now and then during certain processes, but it rarely fails on this tool despite the huge number of Feature Classes created.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.