Timeline for Speeding up Python code to convert multiple csv to shapefile?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:34 | history | edited | CommunityBot |
replaced http://gis.stackexchange.com/ with https://gis.stackexchange.com/
|
|
| Nov 27, 2015 at 8:00 | answer | added | Surya | timeline score: 1 | |
| Nov 26, 2015 at 11:32 | answer | added | PolyGeo♦ | timeline score: 1 | |
| Nov 25, 2015 at 14:09 | comment | added | Giacomo | Thank you all for your suggestions. I finally came back to the office today and tried to move the files on the local machine and re-run the script, it worked! What before was taking hours with the data on the network drive now it took only a couple of minutes. Thanks again! | |
| Nov 23, 2015 at 23:21 | comment | added | PolyGeo♦ | Would you be able to revise your presented code to include performance timing around any steps you think may be slow and the output of a few iterations from that, please? I suspect that this is not an arcpy/python problem but simply one of looking more closely at the MakeXYEventLayer then CopyFeatures workflow. If you have thousands of CSV files to process then you will need to iterate thousands of times so your focus should be what is inside each iteration. | |
| Nov 21, 2015 at 8:18 | comment | added | Tom | You could try writing to a gdb feature class instead of a shapefile. | |
| Nov 21, 2015 at 3:22 | comment | added | Paulo Raposo | I too tend to agree, with @jon_two - your code doesn't look inefficient on it's own. I/O is almost always the bottle-neck, and I've seen serious speed-ups when I switched from network to local drives on my local university machines. Another possibility is to use a RAM disk (i.e., a temporary hard-drive in RAM). You can work there, then save your work somewhere persistent (since RAM drives disappear upon shutdown!). This one (softperfect.com/products/ramdisk) has worked well for me. Need admin rights to use it. Also, local USB drives are slow, but maybe faster than your network. | |
| Nov 20, 2015 at 19:42 | history | edited | PolyGeo♦ | CC BY-SA 3.0 |
deleted 4 characters in body; edited tags; edited title
|
| Nov 20, 2015 at 17:45 | comment | added | Ben S Nadler | I tend to agree with Jon. Test by moving the files to the local machine, and output to the local drive too. Try adding debug timestamps to each step and see how long each specific process is taking. | |
| Nov 20, 2015 at 17:30 | comment | added | Giacomo | It is a network drive, unfortunately I cannot use my local drive but it is still incredibly slow. In 4 hours it converted 7 tables, each table is about 80,000 rows. At the moment I am running this code on just the London metropolitan area but I hope to run it for the entire country, but this would mean having to loop through thousands of csv...It would take forever at this pace. I guess my approach is entirely wrong. | |
| Nov 20, 2015 at 17:06 | comment | added | jon_two | Your script doesn't have many steps, so it's hard to see where any time savings can be made, but I would suspect that CopyFeatures is fairly slow. Is G:\ a network drive? Moving the shapefile to your local machine would speed things up a bit. | |
| Nov 20, 2015 at 16:16 | history | edited | Giacomo | CC BY-SA 3.0 |
added 127 characters in body
|
| Nov 20, 2015 at 16:08 | history | asked | Giacomo | CC BY-SA 3.0 |