I have 200 Mb shapefiles of polylines downloaded from OSM. I've been working with QGIS 2.0 in Ubuntu 64 Bit 8 Gb RAM to make some geoprocessing tasks and it's taking 2 days for processing.
Once I've got my final shapefile, i have to make further calculations/scripting for making a model, and i don't have that time.
So, in order to accelerate the process I've been thinking in:
- Exporting my shapefiles to R
- Converting them to SQL and process them by spatialite
- Maybe gdal in shell?
But i don't know what is faster/better option, because I'm a beginner with big data and scripting, so python is not yet an option for me. I have a few experience in R, but I do not know if it's the better option.
I have a database for Brazil. I'm interested in two shapefiles: "landuse" and "roads". I have another with the state of São Paulo "sp". So, i just need to intersect the landuse with sp = landuse_sp, roads with so = roads_sp, and later roads_sp with landuse_sp. That's in order to have every the roads of the state of São Paulo with the landuse. Then I'll intercept with the municipalities and with another dataset with vehicle count I'll generate a model for vehicle count.
With this final shapefile, i need to perform case calculations adding fields. With field calculator, creating the field "count" as an example, would be something like this:
CASE WHEN roads IS 'primary' AND landuse IS 'residential' THEN exp(8 + 0.0033*2)
ELSE 0
END
This is just an example, but it's quite the idea