“XY to Line” not converting all 58,000 records

Question

I have a dbf (and xls) that contains ~58,000 records of latitude and longitude for both start and end points. I would like to connect each XY coordinate pair using the XY to Line tool found inside the Data Management toolbox. I will be using the Great Circle method fyi.

The resultant shapefile contains ~16,000 records/polylines. As far as I can tell, there is no difference between processing the table inside or outside a file geodatabase.

Anyone have any thoughts as to why it may not be processing the entire table? This issue persists with both the dBaseIV and XLS file.

Cheers,

Michael

Answer 1

"I don't believe there is an issue due to the fact that my routes cross the 180/-180... I believe ArcMap understands how to handle those circumstances."
Not in WGS84, no. A feature cannot cross the international date line. You must break it at the international date line or it will throw a bad geometry. Too many bad geometries, and the tool bails out.

Convert to a projected coordinate system and that goes away. This is a very long standing bug that I thought was handled at 10 SP2 (or might be planned for SP3/10.1).

If you are still getting all of your features, even the ones that cross the dateline, then this bug was probably fixed. If you are missing the dateline crossing features, then the batching is probably just allowing the tool to continue past bad geometries when the whole set could not. I know the bug definitely still existed at 9.3.1.

Answer 2

blord-castillo,

I've solved it by making a few changes to my workflow as I think it may have been some physical memory trouble.

1. Turned of background geoprocessing (ArcSOCP ate up all my ram) - Not Solved
2. Split table into 10,000 record chunks (6 tables). 2b. Worked the first time, but didnt compute all records on the 2nd table... hmm
3. Closed/Re-opened ArcMap (refreshed memory usage at the same time?) 3b. Worked and computed the 2nd table properly
4. Went about closing and reopening ArcMap each time I processed a table.

Now... I am left wondering if turning off background processing (ArcSOCP) was counter-intuitive. Does ArcSOCP clear its memory usage when a new geoprocessing task occurs such that my 2nd, 3rd, n'd table would process without having to close/re-open ArcMap?

The paths range in length because they are flight routes. I don't believe there is an issue due to the fact that my routes cross the 180/-180... I believe ArcMap understands how to handle those circumstances. There is no id field (that would be easy to test with a little python to fill in a column of sequential numbers).

Projection is the default WGS1984.

Anyways, I have completed what I have set out to do but I am still stumped as to why I couldn't just process the table as a whole.

Cheers, www.spatialanalysis.ca

Answer 3

Have you tried using a CSV (Comma Separated Values) file ? I find that it works better as input to ArcGIS than DBF or XLS.

Answer 4

I found a solution to what was my problem using the "XY to line" - tool, I thought I could post it here to help others.

I dragged and dropped my feature set containing my points, to the tool, and the tool gave a wrong result. However, I realized that I also had another copy of the feature set but with another query definition, laying above my "intended to use" layer. It looks like, even though I dragged and dropped my layer, the "XY to line" - tool picked the uppermost layer. When I deleted this layer, the tool worked perfectly!

Cheers from Norway! /Tore

Answer 5

I had a similar problem involving a one to many relatioinship with a feature class of a little over 175000 and a feature class of over 500000 records. The XY to line tool was crashing after creating anywhere from 10000 to 15000 lines. After designing a complex and slow python script that tiptoed around the records limit by using a custom made index layer and using it to select by location within a loop and outputing these into a temporary folder only to merge all of the temporary line subsets (contact me if you would like to see this clunky code) into the final line feature class.

Shortly after debugging this script I found that by converting your feature class to dBase Tables the records limit on the XY to line tool is avoided. The tool worked to create over 500000 lines in the new feature class. It worked for me and I am curious if these is true for others who have faced the same issue?