In a current project, I am attempting to extract several fields for each record of a table in a GDB that has ~8.5 million records. My current strategy is to create a search cursor, fetch the data I want from each row, and pass it to a
csv.writerow to write a line with the desired information. Unfortunately, after about 340k records, I get a message from Windows telling me Pythonw.exe has stopped working, and the process stops.
Why would this happen? Is there a better method for extracting large amounts of data to text? DBF isn't an option because these files could become larger than the 2GB limit.
Interestingly enough, after a few more tests, it appears that it always crashes on the 341051st record, and always at the same point on the line.
Edit Now attempting to split up the cursor to read only a subset of the data each time. It is apparently a memory issue. I created a list for each subset query eg:
squery = ['OBJECTID >=0 AND OBJECTID <=300000', 'OBJECTID >=300001 AND OBJECTID <=600000']
And run the search cursor as an iteration:
for query in squery: InRows = gp.SearchCursor(InFC,query) for row in InRows: val = row.val val2 = row.val2 items = [val,val2] OutCsv.writerow(items) del row del InRows