1

I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (dimension X: 639760 Y: 452420 ). I am implemented a multiprocessing starting from this script this script (from @Perrygeo!!) but the process run forever; after two weeks it haven't finished yet.

The process was reading the entire shapefile and the entire raster and after this processing using the code.

So I would like to implement this process:

  • reading the shapefile piece by piece from a PostGIS database. I am not able to read a shapefile piece by piece with geopandas or fiona.
  • applying the zonal statistic only on the part intersecting the raster, a very big raster (6.3 GB also with LZW compression). I am assuming the processing will be done only on the intersection area.
  • printing the final output on the database

Can I use asyncio to implement a concurrency to speed the process and applying at the same time a multiprocessing on the zonal statistic itself? I need to write a fast code...

  • Why can't you do it inn fiona or geopandas. Which zonal statistics, intersecting what raster (very big is not very helpful). What is the input? There are lots of ways to use concurrency depending on the algorithm, input, etc, but your question as it stands is a bit unclear. – John Powell Feb 18 at 11:13
  • you are right. I will try to write my question better – Glori P. Feb 18 at 11:18
  • Are you planning on loading the the raster to Postgis also? If you load it with tiling, raster2pgsql -t, then vector/raster lookups with us a spatial index and can be very fast. If you are not, then I assume you are opening the raster using something like rasterio in Python? – John Powell Feb 18 at 13:09
  • Yes I am planning to use rasterio. If I use raster2pgsql -t can I read the raster with python ? postgis on zonal statistic is realy slow. – Glori P. Feb 18 at 13:15
  • Yes, you can still read the raster from Postgis, but, if you are using raster2pgsql then there would be no point. My guess is that the reason you are saying that zonal statistics is really slow in Postgis is that you have not tiled your raster. Why can you not read a shapefile bit by bit in Python, btw? As I see it, you could do this either completely in Python or completely in Postgis, it, is the combination that seems to be causing confusion/speed problems. – John Powell Feb 18 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.