3

I have 21 separate shapefiles containing data for Ireland as various polygons on QGIS 3.10. I'm trying to intersect each shapefile onto a 250m² grid of Ireland so that I'll have 21 separate shapefiles in a grid format. I'll then spatially join each dataset using the corresponding grid "ID"s to create one shapefile containing all the data in a 250m² grid format for Ireland.

My problem is that the intersections onto the grid are taking days to complete. One intersection is currently on 56% after 31 hours, and the intersected shapefile only contains 26 features and 1 column of data, although there will be 1,000,000+ features in the 250m² grid output. Other datasets that will also need to be intersected onto the grid are much larger (4,000,000+ features) as they were converted from rasters.

I have enabled "Render layers in parallel using many CPU cores" and have created spatial indexes for the two layers and don't know how else to speed up the processing. I'm running QGIS 3.10 on a PC with 8GB RAM and an i5 processor. Is the lack of processing power on my PC the only reason the intersection is taking so long, or is there another way to increase the speed of the intersection?

I'm performing the intersections by Vector > Geoprocessing Tools > Intersection. With the Input layer as my 250m grid, and the Overlay layer as the shapefile to be intersected onto the grid.

5
  • Welcome to Geographic Information Systems! Welcome to GIS SE! We're a little different from other sites; this isn't a discussion forum but a Q&A site. Your questions should as much as possible describe not just what you want to do, but precisely what you have tried and where you are stuck trying that. Please check out our short tour for more about how the site works.
    – Ian Turton
    Sep 25, 2020 at 15:54
  • 2
    Please edit your question to tell us how you are doing the intersection?
    – Ian Turton
    Sep 25, 2020 at 15:54
  • Try to keep the data in raster format as long as possible.
    – BERA
    Sep 25, 2020 at 17:08
  • 1
    It's not clear what you mean by "create one shapefile containing all the data in a 250m2 grid format ". A grid usually refers to raster, not shapefile. To continue BERA's comment, if your goal is to have geospatial data on a 250 m2 grid, want definitlly want to be using rasters, not vector layers with millions of small polygons.
    – Micha
    Sep 26, 2020 at 8:58
  • @micha "definitely" is strong. Rasters are not good with attribute data, so there is I think still a good case for using vector data. My out-of-left-field advice would be rather to learn about and use a DGGS. Apr 12, 2021 at 0:05

1 Answer 1

2

When processing big data sets the biggest cost is usually related to moving your data from storage space to processing. If your processing takes a long time while your CPU performance is still low you, should start looking at how your data is accessed. Working on data placed on a network drive is typically something which will lead to long processing time. A slow disk or fragmented disk could also give you similar problems.

I suggest two alternative approaches to be able to do this more efficiently:

  1. Load the files into a PostGIS instance and process it there using a spatial SQL query. It is relatively easy to set up a job using spatial queries. You can install PostGIS locally on your computer. Remember to index your data.
  2. Avoid using Shapefiles. Convert your data to Geopackage. Then try your processing again.
  3. If fiddling with PostGIS sounds overwhelming, FME Desktop from Safe software provides an excellent option for processing data in a visual way. This is commercial software, but they have a free option for non commercial use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.