1

I'm attempting to export an attribute table that contains 30 million rows (with 10 columns) to a .txt file. I've been able to export smaller portions of it (e.g. selecting and exporting the first 100 records), but the sheer number of rows slows the export process to a crawl or just crashes it.

Clarification: I have a raster that covers about 90% of Venus with ~4 km resolution, and I need to extract the latitude, longitude, and pixel value for each pixel. I thought the best way would be to use Raster to Point, then Add XY Coordinates, and export the table of the resulting shapefile. If there's a better way, I'd love to learn!

Using the ArcPy script described in the answers worked on a similar, but smaller shapefile (~10% of Venus, same resolution), but when run on the larger shapefile of 90% of Venus, the IDLE shell quit without warning after an hour of running. Any ideas of what caused/could prevent this?

|improve this question
  • 1
    Please edit the question to specify your exact procedure. I've exported 100m rows to ASCII in minutes. Also specify your source format. – Vince Dec 6 '16 at 22:27
  • If your data is being held in a RDBMS, have you tried one of their tools rather than piping it through Arc? For example SQL Server's SSIS or BCP would pump through this in short order. For example on my machine BCP will do approx 120,000 rows a sec without any tuning. That's just over 4 mins for 30,000,000 rows. – MickyT Dec 8 '16 at 1:30
  • Unfortunately, these shapefiles/rasters are not in any databases. – Avram J Stein Dec 8 '16 at 4:00
6

I would suggest pure python in combination with the arcpy.da.SearchCursor. A basic export to text file code sample looks like this:

out_file = r'C:\path_to_data\output.txt'
input_feature_class = r'C:\path_to_data\your_input_data.shp'
with open(out_file, 'w') as w_file:
    with arcpy.da.SearchCursor(input_feature_class, '*') as s_cur:
        w_file.write(','.join(s_cur.fields) + '\n')
        for row in s_cur:
            output_data = ','.join(map(str, row))
            w_file.write(output_data +'\n')

This is a simplified version of the code, that simply converts the field values to a single string separated by comma and writes it to file. Without knowing more about your data it is difficult to provide custom formatting, such as handling the SHAPE field and other aspects of the export/file writing. The documentation for arcpy.da.SearchCursor is here. The above code snippet should get you on your way.

If you can provide more specifics about your data I can certainly clarify and give tips to handle certain types of fields.

|improve this answer
  • This looks like it's working perfectly, but I'll report back when it has finished running. Thanks for your help so far! – Avram J Stein Dec 6 '16 at 23:14
  • 1
    Nice. You could also add a line to write the field names as a header row. – Fezter♦ Dec 6 '16 at 23:48
  • @Fezter good call, see updated answer that includes getting and writing the headers. – GeoSharp Dec 7 '16 at 0:06
  • 1
    You can access the fields returned by the cursor as s_cur.fields. – Paul Dec 7 '16 at 0:15
  • 3
    Better to always use arcpy.ListFields, since a * query is going to return meta columns like the length and area, and rowid. You also want to be careful if/how you query the geometry column, so that it comes back in WKT. And if you do return geometry, you certainly don't want to make the output comma-delimited -- tab or pipe are the only ways to fly. – Vince Dec 7 '16 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.