cannot read raster of 14 MB

Question

I would like to start using pure python to handle geospatial data since POSTGIS is not so optimized for rasters.

I starting using rasterio and with this code :

with rasterio.open(clc_r) as src:
    print(src.width, src.height)
    print(src.crs)
    print(src.transform)
    print(src.count)
    print(src.indexes)
    print(src.profile)
    array = src.read()
    array.shape

I got this output:

63976 45242
+ellps=GRS80 +lat_0=52 +lon_0=10 +no_defs +proj=laea +units=m +x_0=4321000 +y_0=3210000
| 100.00, 0.00, 918900.00|
| 0.00,-100.00, 5440600.00|
| 0.00, 0.00, 1.00|
1
(1,)
{'driver': 'GTiff', 'dtype': 'float32', 'nodata': 65536.0, 'width': 63976, 'height': 45242, 'count': 1, 'crs': CRS({'proj': 'laea', 'lat_0': 52, 'lon_0': 10, 'x_0': 4321000, 'y_0': 3210000, 'ellps': 'GRS80', 'units': 'm', 'no_defs': True}), 'transform': Affine(100.0, 0.0, 918900.0,
       0.0, -100.0, 5440600.000000001), 'tiled': False, 'compress': 'deflate', 'interleave': 'band'}
Traceback (most recent call last):
  File "raster_calculation.py", line 23, in <module>
    array = src.read()
  File "rasterio/_io.pyx", line 316, in rasterio._io.DatasetReaderBase.read
MemoryError

I am getting crazy trying to understand why with:

RAM 7,8 GiB
 os 64-bit

I am not able to read a raster of 14 MB... What am I doing wrong? If even with such a small raster I got an issue like that, the processing will be even more tricky. (and with larger files even worst..)

I am running on Ubuntu 18.04 using python 3 64 bit..

---

UPDATE

---

The file I was trying to read had a lot of NoData compressed with the LZW compression (i dunno much about this: just figured it out ...). I didn't notice this on QGIS.. So it's the extension of the entire Europe! I clipped it better and it is ok now!

Answer 1

You'll notice in the dict there that the file has deflate compression, so on the disk it may be 14MB but rasterio is trying to uncompress it as you read the whole band.

We can figure out the uncompressed size if we know the dimensions and datatype, which you've also listed:

63976 * 45242 = 2894402192 total pixels
2894402192 pixels * 32 bits/pixel = 92620870144 bits
92620870144 bits * (1/8e+9) = 11.57 GB

I'm guessing you have a ton of nodata in the image if it compressed down that well.