Time to open 3.5GB File.GDB directory vector layer in QGIS

Question

I am a new GIS user, and have started with QGIS 2.01 Dufour. I am trying to open an ESRI-style File.GDB (v10) directory that has a 3.5GB gdbtable file that is of Texas state hypsometry/topography. The machine is clearly busy as it tries to open this layer because the hard disk is spinning, and one of the cpu cores shows elevated activity at ~20%. But even after waiting up to 25 minutes the layer has still not opened successfully (one small exception below). As I am new to GIS, I simply don't know what to expect.

Should I expect opening such a layer to take seconds, minutes, or hours?

I know this is not a straightforward question given so many details are involved, but I provide more info below.

I am using a fairly capable system. It is a notebook with an Intel Core i7 3632QM Quad Core cpu, 8GB ram, and Windows 8.1 (64-bit). I am running the 64-bit version of QGIS 2.01 that was installed with the standalone installer. The File.GDB directory vector layer that I am trying to open is on the local hard drive (so it is not constrained by USB/other transfer limits).

I mentioned above that there was one small exception to opening this 3.5GB File.GDB directory vector layer successfully, and it opened essentially instantly (well, within probably seconds). This was the very first time I tried to open it. I was zoomed in to an area of Texas just a few kilometers wide, and I went through the process of adding this layer to my project. It opened fine within seconds, and I could see the topography in the area of my viewpoint. Then I had the "bright" idea of zooming to the extents of the 3.5GB topography layer (i.e. the whole state of Texas), and that's where the problem started. The machine tried to refresh the view, and after five minutes of a blank screen and spinning blue ball, I thought there was a problem so I force-closed QGIS via the Task Manager.

Now, every time when I attempt to open this 3.5GB topography layer, the machine puts itself to work, and even after waiting up to 25 minutes, the layer has still not opened. This happens whether I try to open the layer directly, and thus, I should be at the default full extents. And it also happens when I start with a zoomed in view of several kilometers width (like the first and only time it opened successfully), and try to add it as a layer so the machine should not try to render the full extents of the layer.

I have twice deleted this File.GDB directory, and unzipped a completely new, fresh copy, and this still happens.

I have deleted the .qgis2 directory in my home directory, and also the main QGIS key in the Windows registry. But again, this still happens.

I have completely reset my QGIS defaults with no success.

---

I am adding some comments to after seeing responses from @GetSpatial and @Burton449.

It seems like I must have been positioned and zoomed just right when I opened this layer originally, it happened quickly. And that waiting many minutes to hours to potentially days might be in order for a data set of this size.

From the various suggestions that were made like chopping the layer into quadrants, simplifying the lines via point reduction, filtering on a subset of the contours, etc., it seems like I would have to open the layer first to perform any of these operations. So this would mean waiting it out at least once.

Or is there a way that I could perform some of these operations before actually loading the layer, esp. for something like the contour filtering operation?

In layman's terms, I'm asking if I can set up something like a filter rule first (open only every fifth contour line, for example), and then open the layer, and have this filter applied on-the-fly.

I'm not averse to command line tools so I am wondering if there is something like a cookie-cutter operation (or Python script) that I could use offline to extract a subregion of this larger layer, and spit it out to a new layer. And preferably in a format that is not FileGDB since that seems to be proprietary. My whole goal is to avoid this lengthy opening process (even if it is only once) because I simply have no idea if we are talking about hours or days. Indeed, since writing my original question, I have let QGIS chomp on this layer, and after 1.5 hours it is still thinking.

Answer 1

There are a few issues that are probably coming into play here. Let's walk through them and see if there are workarounds.

1. The layer type is inherently very difficult to process. Topographic layers are made up of contour lines that tend to be very long, miles in many cases, and depending on the method used to create them, these lines can be extremely dense in terms of points. You may have a 2 mile long line, with points every 1 to 2 feet, or even closer. This means that 1 line may have 5-10,00+ vertices to render every time it regenerates.
   • The potential workaround for this is to attempt to simplify your lines. I've taken contour lines and simplified to even 1 point every foot, and this reduced the density significantly, and reduced the file size by over half. I was concerned about how this would change the smoothness of the lines themselves, so I overlaid the before and after layers. The result in most cases was close enough to almost be indistinguishable. This will, of course, depend on your contour interval. The smaller the interval, the more likely your features are to be close together in some areas. Smoothing excessively in these areas can cause overlaps, which may then present processing issues in the future. Try the simplify geometry tool in QGIS.
   • The other option, assuming you only need this for processing and not display purposes, is to load it into your project and have the layer turned off. This will allow you to work with it, but not cause the overhead on the computer to try to display. One way to do this is load the layer in. Once it appears in the table of contents on the left side, hit the Escape key. This will cancel the regeneration. The other, easier way, is to open up a new session of QGIS, and before you load any layers in, uncheck the Render box at the bottom of the window. This will disable rendering, but still allow you to do other tasks.
2. The fact that the layer takes as long to display when you are zoomed in as when you are zoomed to the full extent could be the result of a couple factors:
   1. The File GDB format is a bit tricky since it is not completely exposed through the API. This means that it may not be as optimized for display outside of ESRI software, as other spatial formats are. This affects you in the way the features are accessed from the GDB. It could be that QGIS has to read through the entire layer before it is able to display a small portion. The reason that you were able to load in a small area previously could be because that area was near the top of the list of features in the layer. This depends on how the spatial indexing of the layer is utilized by the OGR software that QGIS uses to access this file format.
   2. The other factor could again be the length of the features in the layer. Even though you are trying to view a small area, QGIS likely has to render the entirety of the features that intersect your area. This includes the portion of the features that are not in your zoom area. This relates back to the first item about how densely packed the vertices are in the features. The more vertices, the longer it takes to regenerate each feature to be able to determine which portions fall inside your area, and thus should be displayed. As an example, this photo is the attribute table of contours from a small city. The layer is stored in a File GDB. The length of the features shown, is in feet. Note that some of these features are over 5 miles long.
3. The way you are trying to display the layer definitely affects how long it takes to process it. If you have a very dense contour interval, for the state of Texas, even a 10 meter interval could be considered dense if you try to display the entire state. If it regenerates, you are likely to end up with a solid mass of lines covering the entire state, because it is so dense. I would suggest limiting how much is displayed, for example showing every other, or every 3rd contour, for a 20 or 30 meter interval.
   • This example shows the same contour attribute table above. The contour interval is 2 ft, but for display purposes, I only wanted to show 10ft contours. I added a field and calculated it to show a value of 1 for each contour that was a multiple of 10 ft elevation.
     I then added a feature subset based on that value
     This photo shows the 2ft contour display.
     This photo shows the 10ft contour display.
   • As you can see, this significantly decreases the density of features, and makes the display much clearer. Even though you are adding the overhead of processing a query on the layer, this is less intensive than the processing required to display the features, so it is still quicker.
4. The last way to address this problem is looking at the sheer size of the dataset, and where it is stored.
   • If it is feasible, I would try to break this dataset up into sections. Even 4 quadrants are going to lessen the amount of time necessary to render.
   • If it is possible, moving this dataset to a different format for storage may be beneficial. With something this large, putting it into a server based database, like Postgresql with PostGIS, may make sense. The benefit to doing this is that you may be able to set up spatial indexes to speed up the display, and potentially also processing of the data.

For this project, my first action would be to try to simplify the contours. This has always given me a huge performance boost, with negligible loss of data quality due to the smoothing.

Whatever you end up doing, good luck with the project, and welcome to the world of GIS. I would recommend that you look at the documentation for QGIS, and search for similar questions on GIS.se. Both are great resources.