2

I have 21 separate shapefiles containing data for Ireland as various polygons on QGIS 3.10. I'm trying to intersect each shapefile onto a 250m² grid of Ireland so that I'll have 21 separate shapefiles in a grid format. I'll then spatially join each dataset using the corresponding grid "ID"s to create one shapefile containing all the data in a 250m² grid format for Ireland.

My problem is that the intersections onto the grid are taking days to complete. One intersection is currently on 56% after 31 hours, and the intersected shapefile only contains 26 features and 1 column of data, although there will be 1,000,000+ features in the 250m² grid output. Other datasets that will also need to be intersected onto the grid are much larger (4,000,000+ features) as they were converted from rasters.

I have enabled "Render layers in parallel using many CPU cores" and have created spatial indexes for the two layers and don't know how else to speed up the processing. I'm running QGIS 3.10 on a PC with 8GB RAM and an i5 processor. Is the lack of processing power on my PC the only reason the intersection is taking so long, or is there another way to increase the speed of the intersection?

I'm performing the intersections by 'Vector > Geoprocessing Tools > Intersection'. With the Input layer as my 250m grid, and the Overlay layer as the shapefile to be intersected onto the grid.

5
  • Welcome to Geographic Information Systems! Welcome to GIS SE! We're a little different from other sites; this isn't a discussion forum but a Q&A site. Your questions should as much as possible describe not just what you want to do, but precisely what you have tried and where you are stuck trying that. Please check out our short tour for more about how the site works. – Ian Turton Sep 25 '20 at 15:54
  • 1
    Please edit your question to tell us how you are doing the intersection? – Ian Turton Sep 25 '20 at 15:54
  • Try to keep the data in raster format as long as possible. – BERA Sep 25 '20 at 17:08
  • It's not clear what you mean by "create one shapefile containing all the data in a 250m2 grid format ". A grid usually refers to raster, not shapefile. To continue BERA's comment, if your goal is to have geospatial data on a 250 m2 grid, want definitlly want to be using rasters, not vector layers with millions of small polygons. – Micha Sep 26 '20 at 8:58
  • @micha "definitely" is strong. Rasters are not good with attribute data, so there is I think still a good case for using vector data. My out-of-left-field advice would be rather to learn about and use a DGGS. – alphabetasoup Apr 12 at 0:05
0

When processing big data sets the biggest cost is usually related to moving your data om pace for processing. If your processing takes a long time while your CPU performance is still low you, should start looking at how your data is accessed. Working on data placed on a network drive is typically something which will lead to long processing time.

I would suggest two alternative approaches to be able to do this more efficiently:

  1. Load the files into PostGIS and process it there using spatial SQL query. It is relatively easy to set up a job using spatial queries. You can install PostGIS locally on your computer. Remember do index your data.
  2. Avoid using Shapefiles. Convert your data to Geopackage. Then try your processing again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.