Global Human Settlement Layer-Rasterio and Fiona

Question

I've downloaded the Global Human Settlement Layer data from here. They are huge files, too big for ArcMap or QGIS to handle comfortably so for that reason I decided to mask them. I was only interested in Ireland so I made sure my coordinate system was the same for both the tif and the polygon (Ireland-admin area) before I begun.

I tried to the following code on one of the tifs that I'm interested in.

import fiona
import rasterio
from rasterio import mask

with fiona.open(r"D:\GIS\Global Human Settlement Layer\Built_Eire\IRL_adm0_WEB.shp", "r") as shapefile:
    geoms = [feature["geometry"] for feature in shapefile]

with rasterio.open(r"D:\GIS\Global Human Settlement Layer\Built_Eire\GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0\GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_p1.tif") as src:
    out_image, out_transform = mask(src, geoms, crop=True)
    out_meta = src.meta.copy()

out_meta.update({"driver": "GTiff",
                 "height": out_image.shape[1],
                 "width": out_image.shape[2],
                 "transform": out_transform})

with rasterio.open(r"D:\GIS\Global Human Settlement Layer\Built_Eire\GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0\GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_p1_Masked.tif", "w", **out_meta) as dest:
    dest.write(out_image)

The original .tif is 358.5MB, as soon as I try and run this code it started creating a huge file that even after only 9% completion it is 50GB.

Can anybody point me in the right direction as to what I may be doing wrong? I presume the file I downloaded was compressed?

Answer 1

Dunuts, yes, your source could be compressed. As a sanity check, make sure the footprint of your out_image (out_image.shape[1:]) is smaller than the footprint of src. The mask() function should only read pixels from src that are within the cumulative spatial extent of your shapes and if your output footprint is larger than you expect there may be some odd features in the shapefile or a bug in rasterio.