4

I have been looking at multiprocessing and tools like Find Duplicates in Big Data and they seem to simply split the data into sections based on rows/number of records. I am trying to do an intersect on datasets ranging from 30 million to 100 million records see this link for details on the project) and for that the data (say for aspect, slope and vegetation) needs to be split into zones which are geographically the same. Any idea on how to do this?

Ideally in ArcGIS 10.

Improve this question
7
  • 2
    I wonder if you might be able to use the existing spatial index to grab N grids worth of features at a time to send to your N threads (you'd of course have to decide on which grid size to iterate, spatial indices can have between 1 and 3 grid sizes.
    – blah238
    Nov 9, 2011 at 5:59
  • 2
    I hope it won't seem repetitive of me to indicate again that the grid-based solution is the way to go. It's horribly slow and inefficient to perform grid-based analyses (which is what this is) by using a vector format for the data.
    – whuber
    Nov 9, 2011 at 6:21
  • 1
    Might be of interest: Python Multiprocessing – Approaches and Considerations
    – radek
    Nov 9, 2011 at 14:09
  • 1
    thanks Blah. Would the spatial index from multiple datasources match up...do you think we can code it to maybe take a select by within or a clip to new temp file for each asp/slp/veg file and then intersect the clipped regions seperately on n cores and finally merge the n areas? can this be told to run on different networked machines rather than just cores on a single machine? It would be a cool issue to solve for stackexchange members and I am willing to offer a bounty to get a working process.
    – GeorgeC
    Nov 9, 2011 at 22:47
  • 2
    I think it would be cool if Esri would support some way to issue a command against a raster catalog layer that would allow processing of each tile in the catalog to be done in a separate process or computer. Kinda like the VISIT command they supported in ARC/INFO LIBRARIAN (kids: capitalization did not imply shouting in the 80's, it's just the way they spelled things back then). Ironically, I don't see any spatially enabled mapper for MapReduce.
    – Kirk Kuykendall
    Feb 18, 2012 at 23:42

1 Answer 1

Reset to default
2

I don't know if it will work with ArcGis 10 but the Districting tool might meet your requirements. It can be downloaded free from the Esri website

Districting for ArcGIS is an add-on that allows you to create defined groupings of geographic data, such as census tracts, ZIP Codes, and precincts, by creating a districting plan.

With Districting for ArcGIS, you can

Redraw political boundaries. Set up territories to streamline sales efforts. Define districts for voting, schools, and fire services. Analyze statistics and data concerning population densities, housing breakdowns, income, and race.

http://www.esri.com/software/arcgis/extensions/districting/index.html

Improve this answer
1
  • 1
    Can you expand a little on this answer to explain exactly how it might work? I also doubt it will be performant for the data sizes the OP is talking about.
    – blah238
    Feb 18, 2012 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.