I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (dimension X: 639760 Y: 452420 ). I am implemented a multiprocessing starting from this script this script (from @Perrygeo!!) but the process run forever; after two weeks it haven't finished yet.
The process was reading the entire shapefile and the entire raster and after this processing using the code.
So I would like to implement this process:
- reading the shapefile piece by piece from a
PostGISdatabase. I am not able to read a shapefile piece by piece with
- applying the zonal statistic only on the part intersecting the raster, a very big raster (6.3 GB also with LZW compression). I am assuming the processing will be done only on the intersection area.
- printing the final output on the database
Can I use
asyncio to implement a concurrency to speed the process and applying at the same time a multiprocessing on the zonal statistic itself?
I need to write a fast code...