4

I am writing a small program that runs through a ton of different shape files, tab files and geodatabases and gets their extent (about 10000- 100000 files).

I am opening these using the GDAL python library. I am using Python 2.7 and GDAL 2.2.3 on a windows 7 platform.

Unfortunately, some of these layers are completely corrupted, which causes GDAL to crash with a system error code.

ERROR:root:Could not read layer g:\31\23922\Tech\SWMM\Heatherton\Data\Heatherton_NS_draped.TAB

Process finished with exit code -1073741819 (0xC0000005)

My code looks like:

import ogr, logging, osr
path = r'g:\31\23922\Tech\SWMM\Heatherton\Data\Heatherton_NS_draped.TAB'
def check_path(path):
    try:
        logging.debug(path)
        ds = ogr.Open(path)
        if ds is None:
            return None
        for i in range(ds.GetLayerCount()):
            print ds.GetLayerByIndex(i).GetExtent()
    except:
        import traceback
        traceback.print_exc()
check_path(path)
print 'amIhere?'

I am fine with skipping this file. I just want to reach the print 'amIhere?' part. Or best case scenario: reach the except block.

But the fact that it uses an exit code, makes the entire thing crash. Is there a way to capture this?

I have tried catching SystemExit, BaseException....

2

I have managed to do a workaround by using subprocesses. Not ideal, but it works.

import ogr, logging, osr
from multiprocessing import Process, Queue

path = r"C:\pgtest\test\1.shp"
def check_path(path, newQueue):
    try:
        logging.debug(path)
        ds = ogr.Open(path)
        if ds is None:
            return None
        for i in range(ds.GetLayerCount()):
            newQueue.put(ds.GetLayerByIndex(i).GetExtent())
    except:
        import traceback
        traceback.print_exc()


if __name__ == '__main__':
    newQueue = Queue()
    p = Process(target=check_path, args=(path,newQueue,))
    p.start()
    p.join()
    if not newQueue.empty():
        print newQueue.get()
    print 'amIhere?'

The solution does slow the process down a lot! I am open to any suggestions.

0

Lennert, there's no exception to catch in your case. The gdal and ogr modules don't raise Python exceptions by default. I believe that if you add import gdal; gdal.UseExceptions() at the top of your script, you'll get a normal Python exception that you catch and handle in a single Python process (no multiprocessing needed). The situation with GDAL and exceptions is explained here: https://pcjericks.github.io/py-gdalogr-cookbook/gdal_general.html#enable-python-exceptions.

  • It's because of this and other gotchas that I don't recommend using the GDAL modules and am working on replacements. – sgillies Dec 6 '17 at 14:52
  • Hi, Thanks for the comment, but it doesn't solve the error. Iit is a straight up GDAL bug which causes a rapid unscheduled disassembly. The bug report to gdal dev worked and the bug won't be there for 2.2.4. – Lennert De Feyter Dec 6 '17 at 22:00
  • Good to know! Glad it was fixed. – sgillies Dec 7 '17 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.