2

I'm a novice GISer who needs to do some basic computations on very large (say ~20GB) files. Nothing complicated, merge, intersect, add XY columns kind of thing. It's a necessary step for a project, but it won't be something I do frequently. From what I understand, ArcMap refuses to work with files over 2GB and I think QGIS would crash with these files.

As I can see it, my options are:

  1. Knuckle down and learn GIS python scripting just for this one computation
  2. Break my computation down into 2GB chunks and run it 10 times, hoping none of the intermediates exceed 2GB.
  3. Find software or file formats that can handle large files.

Number 3 would be the easiest if such software exists but I'm not sure it does. If not, do I have any options other than 1 and 2? Both are unpalatable (being new at scripting for 1, tedium and possibility that it doesn't work for 2). Are there other options?

3
  • 2
    Use a database like PostgreSQL/PostGIS. Jun 3 '15 at 17:10
  • 8
    Your understanding of ArcMap is flawed. Files over 2GB are not a problem for Arc. "The default maximum size of datasets in file geodatabases is 1 TB. The maximum size can be increased to 256 TB for large datasets (usually raster)." The issue is format, as in shapefiles have a size limit of 2GB. It's also somewhat tied to your system specs. So what data do you have in what format?
    – Chris W
    Jun 3 '15 at 17:11
  • If I can ask a stupid question, is it > 2TB for a single class or is that all your data? How did the data come to you in a size larger than 2TB? As multiple shapefiles/text files that needed merging perhaps.. Python isn't anything special, it doesn't have magic powers to do anything that you can't do in ArcGis interactively, so learning scripting is something you should put on your 'to do' list because it's a good thing to know not because it has amazing super powers. Jun 3 '15 at 21:13
10

I agree with @ChrisW in that ArcGIS can work with massive datasets in file geodatabases (1TB), yet shapefiles have size limitations (2GB). A few recommendations:

  1. Make sure your vector data are in a file geodatabase.
  2. Ensure that 64-bit background geoprocessing is enabled.
  3. Simplify your data in any way you can prior to processing. If your objectives allow, 1) simplify the geometry, 2) delete duplicate features, and/or 3) dissolve.
  4. Make sure your spatial data are all in the same projection, otherwise ArcGIS will be forced to project-on-the-fly, which can be computationally expensive.
  5. If your workflow allows, switch to a raster-based analysis. Raster-based processing can be much more efficient than vector in many cases.
  6. Make sure you have a computer with suitable processing power. These days, 32GB RAM and 8 cores are common. Often, large organizations such as Universities have high powered systems available for use in, for example, the Computer Science or Engineering department--these folks are usually willing to help you out and let you use the system.
5
  • 2
    I agree 100% with everything you and Chris say, however the possibility of breaking down the data (dicing) and running in multiple operations has merits (and detriments).. for someone who isn't a novice (sorry Cory, this isn't for you) splitting the data and reassembling the results can turn a 48 hour geoprocessing operation into a few (dozen) two hour operations - but the user must be aware of how the absence of other features affect the results and be capable of removing/resolving duplication. Note also that a lot of temp files are shapefiles hence the issue ArcGis has with large files Jun 3 '15 at 21:08
  • Good points @MichaelMiles-Stimson. These recommendations also apply to processing diced data. This post could benefit from having another answer that details the points you touch on (e.g. methods to break up datasets into smaller pieces and the pros/cons of performing these actions with vector data).
    – Aaron
    Jun 4 '15 at 2:13
  • 1
    Thanks for all the tips, guys. It seems like Geodatabasing is the way to go if I want to do this cleanly. This is completely new to me, but if my efforts to slice & dice fail, I'll go there. Very helpful. I was asked how the data got to be so large. I'm trying to map a normal-sized shapefile (0.8GB) with very finely gridded data. If I tried to create a single grid with the range I'd need (as a shapefile), it would be around 20GB. I'd intersect this grid with the original, normal-sized shapefile.
    – Cory Smith
    Jun 4 '15 at 7:51
  • 1
    @CorySmith Also, if you can convert any of your workflow to a raster-based analysis, you will also see performance gains.
    – Aaron
    Jun 4 '15 at 11:22
  • @CorySmith FYI, I work with a large grid on a regular basis - the PLSS for the entire state of Colorado. It's an irregular grid, and the dataset contains multiple levels/feature classes. The finest is 1.54 million cells, next level up is 105k, top level 3k. Plus a 31k feature class of special areas. All this is in a single geodatabase that's about 425MB. Don't know how that compares to your data/grid, but file geodatabase is much more efficient than shapefiles.
    – Chris W
    Jun 4 '15 at 18:31
4

I've had the same issue with ArcMap and even though it nowadays probably can handle that size, I still find that using a geographical database such as PostGreSQL+PostGIS is the easiest and most reliable way. Plus, it connects nicely to QGis, so you can visualize your results quickly after you've done your updates.

The downside is of course that you'd need to install, setup, and learn to use a PostGIS database (by using the SQL language) - so that might be a drawback. In my experience, it's a good investment if you want to continue with GIS - once you know geographical databases, things just tend to take a lot less time.

There are lots of guides for getting started with PostGIS+QGis on the web: I've used this one. Hope this helps!

1
  • 1
    ArcGIS will also work with PostgreSQL, with or without PostGIS.
    – Vince
    Jun 3 '15 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.