3

I need to run a "one-to-many" spatial join between two large and complex polygon layers. The problem is that the output saved to a file GeoDB results in a layer that is almost 1 TB in size and it takes hours to run. However, the only thing I really care about is the output of that attribute table itself, which has a separate row for each record in the join layer that each record in the target layer intersects with, and that is the only thing I need to generate various tabular summaries I need to run.

Does anyone know if there's a way to basically run a one-to-many spatial join, but only have it write the tabular results in the output, without any shapes? The shapes in this output are not useful for anything, and just take up a massive amount of space and probably add a lot to the processing time to write them all (hundreds of thousands).

Improve this question
10
  • Do need to retain any information about that the intersections, such as the acreage of overlap? If so, then you're probably best off leaving it like it is, and then exporting the attribute table and deleting the spatial output. If not, then I've got some ideas that would require ArcPy. Would a coding solution work for you?
    – Tom
    Commented Aug 3, 2017 at 14:43
  • No, this is a straight-up intersect, so do not care about any information about the intersections except that the records touch.
    – Jason M
    Commented Aug 3, 2017 at 14:59
  • The problem is that the 1 TB output I'm getting is just for a pilot run for data in a handful of states. I'm going to have to run this nationally eventually which could be an output of 1 TB x 10 = holy cow! The table itself is only a fraction of that size however, so if there's a way to avoid having to write all those shapes in the first place that would be ideal.
    – Jason M
    Commented Aug 3, 2017 at 15:09
  • I'm pretty much a novice at Python, but I figured one way might be to code something that will crawl through a layer, record by record, and intersect it against another layer, and write a row for each intersection with the attributes from both layers. We used to have an avenue script that would do that way back in the day... Wasn't sure if maybe I'm overthinking this and there's already an out-of-the-box way to do this using existing ArcToolbox utilities...
    – Jason M
    Commented Aug 3, 2017 at 15:11
  • After performing the Spatial Join, open the attribute table and export just the attributes.
    – klewis
    Commented Aug 3, 2017 at 15:49

2 Answers 2

Reset to default
1

Just adding an update to my own question that I think I may have figured out a couple of different existing tools that can accomplish this based on feedback above and from other folks I've reached out to.

One is the "Tabulate Intersection" tool, which basically outputs a table with a separate row for each type of value that records in the input layer intersect with in a second layer (or you can run a layer against itself). If you use the unique record ID for each layer, it will essentially give you a table with a separate row for each record that overlaps in each layer. The only problem is that it won't output records as an intersection if they only touch but do not cross like a spatial join will. This ran on my same input dataset in only about 1.5 hrs and the output table was only 35 MB.

The other tool that could potentially work as suggested above, is the Generate Near Table utility, which I haven't tried yet. Presumably if you set a search distance of zero, it would also generate a table with a separate row for each record with every other record that is within a distance of zero, which is essentially the set of all records that touch or cross each other. The difference with the Tabulate Intersection tool is that you wouldn't be able to discern whether records cross or just touch if that is important; you'd only know it's one or the other I think.

The coding solution could definitely be helpful if we decide to try and tackle this with a custom tool or script.

Improve this answer
0

To do this, what I would try starting from is:

  1. Use an arcpy.da.SearchCursor and @SHAPE token to write a Python dictionary for SHAPE and ID
  2. Use an arcpy.da.InsertCursor to have a table ready to write records into
  3. Use a second arcpy.da.SearchCursor to compare each geometry with every geometry in the dictionary, and whenever they intersect write a record to the table

If it provides adequate performance then fine, and if not you will have learned more about your problem, and may need to think about ways to divide and conquer your data.

If Python and ArcPy do not provide the performance that you seek then at least you will have a quickly written prototype from which to write lower level code using ArcObjects.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.