9

I am trying to run an Intersect process in arcgis 10 sp 3 with 2 file sets (aspect and slope) from up to a 1m DEM across an area of 65,000sq km. The aspect has 9,930,384 records and the slope has 31,435,462 records (approx 12GB total in 2 file geo-databases).

I have run repair geometry about 3 times and now the datasets do not report any errors (each time took over 30h).

Now I get

Executing (Intersect): Intersect "D:\SCRATCH\Projects\106\data\7_asp_Merge.gdb\asp_HghstRez_M_rep #" D:\SCRATCH\Projects\106\data\working\working.gdb\AsSl_Int ALL # INPUT Start Time: Sun Oct 23 02:19:10 2011 Reading Features...

Processing Tiles...

ERROR 999999: Error executing function.

Invalid Topology [Too many lineseg endpoints.]

Failed to execute (Intersect).

Failed at Sun Oct 23 04:09:12 2011 (Elapsed Time: 1 hours 50 minutes 2 seconds)

Is this really a topology issue or a file size issue?

I have tried to use the ArcINFO SPLIT tool but it fails even with over 1TB of free space on the drive and on smaller file set causes jagged edges. I can’t use DICE as the areas to intersect between the asp and slope must be exactly the same. I understand that on large datasets ESRI cracks (automatically tiles) the datasets –can this be introducing issues? Is there any more info I can provide to problem solve.

The spec of the machines is more than ESRI minimum –we have 16GB RAM, Intel Xeon, Windows 7, 64-bit, 2 x One TB disks and more than 1.2TB free on the drives. All files used in the process are on the local drives.

just found this explanation (July 2nd, 2012) that gives a lot of helpful hints on resolving the issues.

http://blogs.esri.com/esri/arcgis/2010/07/23/dicing-godzillas-features-with-too-many-vertices/

Improve this question
7
  • 1
    The file size limit for the Windows operating system is 2 GB. (3GB with /3GB on XP). Try the SPILT tool in ArcGIS with Large Datasets 'Tiling' resources.esri.com/help/9.3/arcgisdesktop/com/gp_toolref/…
    – Mapperz ♦
    Commented Oct 24, 2011 at 14:24
  • 1
    An important piece of info from the link Mapperz sent: "Enterprise and file geodatabases do not have this limitation so they are recommended as the output workspace when using very large datasets."
    – RyanKDalton
    Commented Oct 24, 2011 at 14:47
  • 1
    Do you have slope and aspect rasters ? If so, do you have spatial analyst?
    – Kirk Kuykendall
    Commented Oct 24, 2011 at 18:11
  • @Mapperz, it depends on the File System. FAT is limited to 2GB, FAT32 is 4GB and NTFS is unlimited according to: microsoft.com/resources/documentation/windows/xp/all/proddocs/…
    – blah238
    Commented Oct 24, 2011 at 23:05
  • 1
    For a raster calculation, George, you can either resample to a common cellsize (such as 1m) or process the different patches separately. It deserves some thought, because a slope or aspect computed at 30m resolution is not exactly comparable to one computed at 1m resolution. It's hard to give general advice in the absence of information about the purpose of this calculation.
    – whuber
    Commented Oct 25, 2011 at 4:46

2 Answers 2

Reset to default
9

Very few contiguous cells in a detailed DEM will have identical values of both slope and aspect. Therefore, if the input features represent contiguous areas of common slope and common aspect, we should expect the result of this intersection procedure to have, on average, almost one feature per cell.

There were originally 65,000 * 1000^2 = 6.5 E10 cells in the DEM. To represent each of these requires at least four ordered pairs of either 4-byte integer or 8-byte floating coordinates, or 32-64 bytes. That's a 1.3 E12 - 2.6 E12 byte (1.3 - 2.5 TB) requirement. We haven't even begun to account for file overhead (a feature is stored as more than just its coordinates), indexes, or the attribute values, which themselves could need 0.6 TB (if stored in double precision) or more (if stored as text), plus storage for identifiers. Oh, yes--ArcGIS likes to keep two copies of each intersection around, thereby doubling everything. You might need 7-8 TB just to store the output.

Even if you had the storage needed, (a) you might use twice this (or more) if ArcGIS is caching intermediate files and (b) it's doubtful that the operation would complete in any reasonable time, anyway.

The solution is to perform grid operations using grid data structures, not vector data structures. If vector output absolutely is needed, perform the vectorization after all grid operations are complete.

Improve this answer
2
  • Accepted with much sadness. Instead of combining the 30m, 10m and 1m datasets I am instead running the asp+slp+veg intersects/scoring on the each dataset separately and then merging them.
    – GeorgeC
    Commented Nov 4, 2011 at 6:18
  • Using the spatial splitting strategy allowed us to complete the project. A dataset which took 7 hours to process (and crashed sometimes), processed in about 100 mins when split into 6 parts and then took 10 mins to merge. To this add approx 40mins to modify the models to efficiently process the multiple parts with minimum inputs (for each iteration) and basically it’s a saving of half the processing time (at least). So a process that otherwise took close to 200h took less than 50h and with only about 15h of "real" work (in deciding how to split data, entering the variables into the models etc
    – GeorgeC
    Commented Dec 2, 2011 at 2:51
1

My experince with using split tool and repair geomtry. It works for me because the one I worked on was using vector layer that I did conversion from raster to vector. I tried to split tool first and gave me the error. So, I had to use the repair geomtry and it depends how long it running. I had do this twice because whenever you made any changes or edit, you still have to re-run repaire geomtry before you do the split. it worked for me.

By the way, I did run repair geomtry on both layers: shapefile and file geodatabase. I suggest you run repair geomtry overnight.

Improve this answer
6
  • 1
    One more thing I forgot. May I suggest is whenever you do something like this, I recommend to try open a new ArcMap and run these tools ? To clear out the temp files you already open, and closed it and open ArcMap. It cleans out the temp. That is my one cent suggest.
    – PROBERT
    Commented Oct 24, 2011 at 16:38
  • Thanks. I have run repair geom 3-4 times and now the datasets don't report any errors. This usually works but I think the datasets are just to large as per whuber's explanation...
    – GeorgeC
    Commented Oct 24, 2011 at 23:19
  • George, I am glad it work for you. Yeah I read what whuber's explanation but my question to you did you merge slope and aspect ? If so, then when you use the split tool, which feature layer did you use to split these layer you merged with ? For example, I had to use 24 quad (about 24 of them which is not that big) to split with my slope ane elevation merged layer. Maybe you could try and narrow down to smaller layer that can split with your merged layer ?
    – PROBERT
    Commented Oct 25, 2011 at 14:23
  • I did merge the slope and aspect and it worked but was not the right process...we needed to intersect and that doesn't work. To split I got a copy of the national 100k topo map grid and used that on the asp and slope seperately. The zone is covered by 30 map sheets.
    – GeorgeC
    Commented Oct 26, 2011 at 10:17
  • Did you run the 100k topo map grid to clean geomtry ? Because I asked, I have had mine detected some errors and had to do clean repair. So it worked on mine. If you are still encounter more problems, can you try and make them divide the national 100k into smaller ones ? Like divide them to three ?
    – PROBERT
    Commented Oct 26, 2011 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.