2

I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing.

Once I've got my final shapefile, i have to make further calculations/scripting for making a model, and i don't have that time.

So, in order to accelerate the process I've been thinking in:

  • Exporting my shapefiles to R
  • Converting them to SQL and process them by spatialite
  • Maybe gdal in shell?

But i don't know what is faster/better option, because I'm a beginner with big data and scripting, so python is not yet an option for me. I have a few experience in R, but I do not know if it's the better option.

I have a database for Brazil. I'm interested in two shapefiles: "landuse" and "roads". I have another with the state of São Paulo "sp". So, i just need to intersect the landuse with sp = landuse_sp, roads with so = roads_sp, and later roads_sp with landuse_sp. That's in order to have every the roads of the state of São Paulo with the landuse. Then I'll intercept with the municipalities and with another dataset with vehicle count I'll generate a model for vehicle count.

With this final shapefile, i need to perform case calculations adding fields. With field calculator, creating the field "count" as an example, would be something like this:

CASE WHEN roads IS 'primary' AND landuse IS 'residential' THEN exp(8 + 0.0033*2)

ELSE 0

END

This is just an example, but it's quite the idea

Improve this question

2 Answers 2

Reset to default
1

Perhaps you can try this in GRASS.

The GUI is perhaps less intuitive than QGIS, and you need extra steps to import (v.in.ogr), perform processing, and export (v.out.ogr) the shapefile. But once you're past the initial hurdle it is a great supplement to QGIS, since it lets you come at tougher problems with an alternative approach.

For a once-only processing effort, you won't need to get into scripting the commands either.

use v.in.ogr for each shapefile to import them to the GRASS environment:

http://grass.osgeo.org/grass64/manuals/v.in.ogr.html

Most of your clipping and intersections can be done through the v.overlay command with 'and' operator:

http://grass.osgeo.org/grass64/manuals/v.overlay.html

To explore those commands through the GUI:

File -> Import Vector Layer -> common import formats [v.in.ogr]

Vector -> Overlay Vector Maps [v.overlay]

Figuring out how to run those steps shouldn't take too long, and will give you a good idea whether GRASS is an useful approach for your dataset.

v.overlay should be able to handle everything else in your processing up to the 'field calculator' - I'm not sure what GRASS has to offer for this step. In the worst case you could export your product back to shapefile using v.out.ogr, then do the field-calculator step in QGIS.

Improve this answer
7
  • 1
    after using v.overlay, you might also want the attribute commands for renaming and deleting the slightly-awkward names of attributes: database->vector database connections -> v.db.*
    – WileyB
    Oct 9, 2013 at 5:51
  • It worked perfectly, and so much faster in grass! i did the following: v.in.ogr dsn=/home/sergio/SIG/roads.shp output=roads v.in.ogr dsn=/home/sergio/SIG/landuse.shp output=lu v.overlay ainput=roads atype=line binput=lu out=roadslu operator=and Thank you WileyB and @underdark
    – Sergio
    Oct 10, 2013 at 4:37
  • Excellent! I've had great luck with grass for bigger overlay operations.
    – WileyB
    Oct 10, 2013 at 21:52
  • Well, i got stucked again. I could perfectly create v.overlay ainput=roads atype=line binput=lu out=roadslu operator=and then i created a grid in qgis and imported to GRASS. Now i want to do v.overlay ainput=grid binput=roadslu out=gridroadslu operator=and and it says: "ERRO:No area features found in vector map <roadssp@PERMANENT>. Verify 'btype' parameter." Is there any way to do v.overlay with ainput: area and binput: lines??? Thank you so much
    – Sergio
    Oct 11, 2013 at 12:25
  • It should work out, as long as everything's been imported without major errors. See the last example on the v.overlay manual page: v.overlay ainput=grid atype=area binput=roadslu btype=line operator=and
    – WileyB
    Oct 11, 2013 at 15:13
0

200Mb is not big data. have you considered using PostGIS for this. two tables , X query windows to run all queries at same time (every query run on own thread). Just remember but something like 2G RAM for shared memory in postgis and those queries should take few minutes. Loading data takes probably longer than processing it

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.