I am having a tiff slide with the dimension of 90112 x 100352. The slide size is 1.3GB . To process this slide i am creating a numpy memmap file with above dimension like below.

If I check the size of the "test.memmap", its showing nearly 27GB. I dont know why its this much bigger than original tiff file.

I am reading the original slide patch by patch and copying its RGB values with "memMapRef". After reading whole slide i am drawing a rectangle on "memMapRef" at some X,Y coordinates. After drawing the rectangle, now I am trying to create image using below code from "memMapRef" numpy array. Here i am getting the Error "size does not fit in an int".

I am using the "imsave" method from scipy.misc.

My Entire code is below:

import cv2
import numpy as np
from scipy.misc import imread, imresize, imsave
import openslide
level = 0
read_size = (10000, 10000) # reading size per patch
tumor_file = "Test_Tumor.tif" # 1.3 GB file 
tumor_slide = openslide.open_slide(tumor_file)
max_OX,max_OY = tumor_slide.level_dimensions[level] # max_OX = 90112 max_OY = 100352
memMapRef = np.memmap('test.memmap', dtype='uint8', mode='w+', shape=(max_OY,max_OX,3))
def process_slide():
    for start_OX in xrange(0,max_OX,read_size[0]):
        for start_OY in xrange(0,max_OY,read_size[1]):
            tile = np.array(tumor_slide.read_region((start_OX,start_OY),level,(size_OX,size_OY)), dtype="uint8")[:,:,:3]
            yield tile, start_OX, start_OY

for img_tile, X, Y in process_slide():
    h,w,d = img_tile.shape
    memMapRef[Y: Y+h, X: X+w] = img_tile

x1 = 500
y1 = 1000
x2 = x1+256
y2 = y1+256
imsave("output.png", memMapRef) # Error "size does not fit in an int".

Where I did mistake and How to fix it?

  • Next time you ask a question, please limit it to a single question. Here you ask two questions - "why is the memmap file size bigger the tif" and "why does my code raise an exception". I suggest you take the tour to find out how this site works. Also, when posting python exceptions, post the entire traceback, formatted as code, not just the last bit of the error message. By just posting "size does not fit in an int", you lose very important information, i.e. that it's an OverflowError and where the error occurred in the underlying packages. – user2856 Feb 5 '17 at 0:05
  • Sure @Luke i will do it properly from next time. – Murali S Feb 7 '17 at 13:30

Your tiff is likely using internal lossless compression, such as LZW or deflate. A file size of 27GB is right for an 8 bit uncompressed 3 band raster of those dimensions.

File size in bytes = rows * cols * bands * bit depth / 8

I calculate your file should be about 27GB

90112 x 100352 * 3 / 1000^3 = 27.1GB

Or in GiB

90112 x 100352 * 3 / 1024^3 = 25.3GiB

As for your 2nd question, your array is too big to be written to png using imsave. It's a limitation in the PIL/Pillow library that scipy is using.

  • Thanks @Luke. 1. How can I reduce that file size? Using cv2.imwrite() method i am able to solve my second question. – Murali S Feb 7 '17 at 13:26
  • If this answered your question, please accept it - gis.stackexchange.com/help/someone-answers You can ask a new question about how to reduce file size. I would give quite a bit of detail about how you want to use the output (not just that you want a png, but what do you want to do with it) as that will determine the best method. – user2856 Feb 8 '17 at 0:33
  • I accepted your answer and raised another question about how to reduce the file size. The link is gis.stackexchange.com/questions/227540/… – Murali S Feb 8 '17 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.