I wrote a python script that download a .zip file from internet, then opens it in memory, and write all files I need in a new .zip stocked on hard drive.

But a MemoryError is raised with the biggest file, (300MB uncompressed), could someone help me find out what is wrong?

QGIS version : 2.18.16 - 32 bits

The script :

import requests, zipfile

def getZipFromUrl(url):
    data = requests.get(url)
    unzipped = zipfile.ZipFile(StringIO.StringIO(data.content))
    del data
    return (unzipped)

def copyZFile(name, origin, dest):
    data = origin.open(name, "r")
    dest.writestr(name, data.read())

def getFile(sdep):
    dir_out = "PLACE OUTPUT PATH HERE"
    dest = zipfile.ZipFile(dir_out+u"/01.zip", "w")
    source = self.getZipFromUrl("http://bano.openstreetmap.fr/BAN_odbl/BAN_odbl_01-shp.zip")
    for filename in bano.namelist():
        if (filename[-4:] != '.txt'):
            self.CopyZFile(filename, source, dest)

The error :

in copyZFile
    dest.writestr(name, data.read())
File "C:\PROGRA~2\QGIS2~1.18\apps\Python27\lib\zipfile.py", line 626, in read
    data = self.read1(n)
File "C:\PROGRA~2\QGIS2~1.18\apps\Python27\lib\zipfile.py", line 680, in read1
    max(n - len_readbuffer, self.MIN_READ_SIZE)

I don't understand why I get it, as it works well using python2.7, or smaller files.


The easiest solution is to forget about 32 bits version that is limited by Windows with 2GB, and use 64 one instead.

I don't have the error anymore, and thanks to OsGeo4W, I can have both versions installed.

More info on why this limit exists: https://superuser.com/questions/1163749/why-do-32-bit-processes-have-a-2-gb-ram-limit

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.