Processing Problems with the Difference (QGIS) and Erase (ArcGIS) Tool with big data

Question

I am struggling with processing a huge data set.

I downloaded it from a German geodata site (https://download.geofabrik.de/europe/germany.html). I downloaded the ZIP of Bavaria and clipped the eight layers (landuse, water, traffic, transport, pois, pawf, natural and building) with a vector shapefile which has the boundary shape of Bavaria. Afterwards I merged all the layers together. The total ZIP has 581MB after clipping and merging the layers the final shapefile has around 4,8GB.

Someone has an idea what happend here?

I used QGIS 2.18.18.

My problem in particular is that I want to get the difference of the clipped and merged layer within the boundary Layer of Bavaria to get a new shapefile with all the empty areas but nothing works. Both GIS programs are calculating for hours and in the end I get an error message. I think the dataset is too big but I also tried to use the difference tool with only the smallest layer and the same happens. So it does work with the huge merged/clipped layer and not with the individual layers.

I worked with the Difference Tool on QGIS and with the Erase Tool on ArcGIS, I also tried the Symmetrical Difference Tool on ArcGIS. The problem is that there is no other possibility for me to get the data of open areas on a different way.

Has someone an idea what could solve my problem?

Is my dataset to big to be proceed with QGIS?

I don't think that my laptop is the problem (I have an Lenovo T470 with 8GB RAM, 64bit QGIS and Core i5 of the 7th Gen) because the computers which I can use at the University with ArcGIS are struggling too.

Here a three pictures as an example. The first one shows the Bavarian Boundary Layer

The second one shows the clipped an merged Layers

The third picture has the Boundary Layer under the clipped and merged Layer

The yellow dots are my open areas which I need to work with and which I tried to cut out with the Difference, Erase or Symmetrical Difference tools.

Answer 1

Based on your description and the images of the data, it looks like it's either a size limit issue or a geometry issue. For size limit, the issue would stem from the fact that shapefiles have a size limit of 2 Gb for component parts. Processing for very large shapefiles can be as time intensive as you're describing - and if you hit the upper memory limits in either RAM or even hard disk for temporary files, your process will fail after running for a long time.

Much better to convert your shapefiles to feature classes stored in a file geodatabase in ArcMap. This will also allow you to more easily check and if necessary correct geometry. To do this, create a file geodatabase in ArcMap, and then import your shapefiles into that geodatabase as "multiple feature classes". You can also simply import the merged shapefile you've already created. Run a Dissolve (Analysis Toolbox) process on your merged feature class, because it looks like the areas you've merged are overlapping, and this could potentially cause issues with the Erase process. After running through these steps, the Erase function should work for you, even if it takes a little while to run.

If the erase fails again, even after converting to feature classes, you can run a Repair Geometry process in ArcMap to see if there are issues with the source files you downloaded, or resulting from the merge process.

You can also run the Dissolve in QGIS, which might solve your problem quickly and simply.