Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

In a current project, I am attempting to extract several fields for each record of a table in a GDB that has ~8.5 million records. My current strategy is to create a search cursor, fetch the data I want from each row, and pass it to a csv.writerow to write a line with the desired information. Unfortunately, after about 340k records, I get a message from Windows telling me Pythonw.exe has stopped working, and the process stops.

Why would this happen? Is there a better method for extracting large amounts of data to text? DBF isn't an option because these files could become larger than the 2GB limit.

Interestingly enough, after a few more tests, it appears that it always crashes on the 341051st record, and always at the same point on the line.

Edit Now attempting to split up the cursor to read only a subset of the data each time. It is apparently a memory issue. I created a list for each subset query eg:

squery = ['OBJECTID >=0 AND OBJECTID <=300000',
'OBJECTID >=300001 AND OBJECTID <=600000']    

And run the search cursor as an iteration:

for query in squery:
    InRows = gp.SearchCursor(InFC,query)
    for row in InRows:
        val = row.val
        val2 = row.val2
        items = [val,val2]
        OutCsv.writerow(items)
    del row
    del InRows
share|improve this question
1  
read the first 325k records then store it as a file (1a.csv) then flush the memory then start at row 325001 do the next 325k records (1b.csv)... then append all the 1x.csv files for the final file. –  Mapperz Mar 31 '11 at 18:21
    
@Mapperz I'm trying that tiling now. Is there a command to flush memory that I need to worry about besides deleting the reference to the cursor and the row object? –  Nathanus Mar 31 '11 at 18:22
    
I would delete the cursor. You will likely want to create your cursor with filter/where criteria. e.g. where oid >0 and oid < 250000. When done, delete the cursor. For the next iteration, create a new cursor where oid >=250000 and oid < 500000. –  DavidF Mar 31 '11 at 18:30
    
@David/Mapperz See edits for new information. –  Nathanus Mar 31 '11 at 18:53
    
I would say that there are at least two possibilities. 1. deleting the search cursor really isn't freeing up the memory, in which case, you may need to delete the gp too. 2. The text file is too big to read into memory, in which case you would want to follow Mapperz's suggestion to write each chunk of the cursor to a separate text file. 3. Maybe there is a weird character or something in row 341051 that is causing a crash. Maybe print that row along with the neighboring rows to see if the data is weird. –  DavidF Mar 31 '11 at 19:04

3 Answers 3

Just to let you know, I have been Python geoprocessing large quantities of data ~8.5 million records (odd coincidence) doing multiple spatial joins and ran into serious crashing problems.

What I found turns out to be a bug with ArcGIS 10's geoprocessor engine, even at SP1. There is a memory leak issue. When a geoprocessing operation starts up and during execution, memory is tied up. At the completion of the operation, memory should be freed back up but is not. If you run successive geoprocessing operations on large datasets, your system memory runs out eventually and you get a nasty crash with a 999999 or 999998 unspecified error.

This problem happens with expensive geoprocessing operations, such as spatial joins. Memory does not get released back to the system until ArcGIS is completely closed, regardless of whether or not the process succeeds or fails.

This is a known, confirmed bug. ESRI is working on a fix for it, but geoprocessing large data (anything over 1 million records is most definitely big data) with memory-expensive geoprocessing operations will continue to be a problem until the fix is done.

share|improve this answer
    
Good to know. Thank you. I wasn't doing any geoprocessing in this example, but it is good information to keep in mind, regardless. I wonder if it applies to cursor operations. –  Nathanus Apr 4 '11 at 21:31
    
I am working on grids of up to 305,000 points. I have an extensive memory leak, in that when you start, the process takes around 8 seconds, but ends up taking up to 2 minutes. ArcGIS isn't realeasing the memory or objects, I think. ESRI stated this was fixed in Sp2, but it isn't. One other note, is that AggregatePoints_cartography no longer works for me in SP2, so we've had to roll back. The new 10.21 beta also doesnt fix the memory issue. –  Hairy Aug 8 '11 at 9:23

Use WeakRef http://docs.python.org/library/weakref.html

or Garbage Collection

http://docs.python.org/glossary.html#term-garbage-collection

share|improve this answer
    
I really need to wrap my head around classes, although the python docs tend to make my head spin anyway. Doug Hellman to the rescue! doughellmann.com/PyMOTW/weakref/index.html –  Nathanus Mar 31 '11 at 20:54
up vote 3 down vote accepted

So the problem is solved with some help from the kindly Mapperz and DavidF. I deleted the SearchCursor Object after each iteration of a subset of the input features, passing the subset to the csv filed using csv.writerow. Interestingly enough, I found that references to a string object do not require extra quotes to read as a single argument ( 'OBJECTID >=0 AND OBJECTID <=300000' vs. '"OBJECTID >=0 AND OBJECTID <=300000"'. At any rate, thank you Mapperz and DavidF for your help working through this!

Note: The above post features the code that eventually succeded.

share|improve this answer
    
glad that works –  Mapperz Mar 31 '11 at 20:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.