0

I have a simple process to make some edits to all the contour files in a directory. It runs fine on the test of 100 odd files but in a directory with about 500 it stops after about 120 files.

Is there any way to release memory after each file is processed?

import processing, os, glob
path=r'F:\Input\Contours\25_cm'
outdir=r'F:\Output\Contours'
os.chdir(path)
for file in glob.glob("*.TAB"):
 filename=file[:-4]
 processing.runalg("modeler:contour_edits",filename+".tab",outdir+"\\contour_5m\\"+filename+"_5m.tab",outdir+"\\contour_25cm\\"+filename+".tab",outdir+"\\contour_1m\\"+filename+"_1m.tab")

UPDATED ---

import processing, os, glob, gc
gc.enable
path=r'F:\Douglas\Input\Contours\25_cm'
outdir=r'F:\Douglas\Output\Contours'
os.chdir(path)
for file in glob.glob("*.TAB"):
 filename=file[:-4]

 processing.runalg("modeler:contour_edits",filename+".tab",outdir+"\\contour_5m\\"+filename+"_5m.tab",outdir+"\\contour_25cm\\"+filename+".tab",outdir+"\\contour_1m\\"+filename+"_1m.tab")
 gc.collect()
 print filename

Output from console

{u'OUTPUT_LAYER_ALGQGISFIELDCALCULATOR_1': 'F:\Douglas\Output\Contours\contour_25cm\pt000443_contour.tab', u'OUTPUT_ALGQGISEXTRACTBYATTRIBUTE_2': 'F:\Douglas\Output\Contours\contour_1m\pt000443_contour_1m.tab', u'OUTPUT_ALGQGISEXTRACTBYATTRIBUTE_1': 'F:\Douglas\Output\Contours\contour_5m\pt000443_contour_5m.tab'} pt000443_contour 107 {u'OUTPUT_LAYER_ALGQGISFIELDCALCULATOR_1': 'F:\Douglas\Output\Contours\contour_25cm\pt001062_contour.tab', u'OUTPUT_ALGQGISEXTRACTBYATTRIBUTE_2': 'F:\Douglas\Output\Contours\contour_1m\pt001062_contour_1m.tab', u'OUTPUT_ALGQGISEXTRACTBYATTRIBUTE_1': 'F:\Douglas\Output\Contours\contour_5m\pt001062_contour_5m.tab'} pt001062_contour 107 ... pt001063_contour 0 ... pt001064_contour 0 ...

When I split the files into batches of 30 in a directory and run this it works fine, when the batch is 50 it fails. If I run it without first restarting qgis (not just the python window) it processes the first 30 and may process some of the smaller files from the next 30 but not all of them. So I really think it is a memory issue???

  • 1
    Can you expound on the stopping? Does it hang up or crash out? I believe Python already releases memory in for loops, but if you want to manage it yourself, you can import gc and then gc.collect() in your loop. See this question. – wchatx Nov 22 '15 at 12:37
  • Does it always fail on the same file? Perhaps there's something amiss in one of your input files. – ndawson Nov 22 '15 at 21:19
  • @ndawson no it just fails after about 120 files, even when I run it on separate directories. – GeorgeC Nov 22 '15 at 22:41
  • @wchatx it doesn't crash or hang up, it just stops processing files. I tried gc but it doesn't seem to make any difference. See updated question – GeorgeC Nov 22 '15 at 22:43
  • gc.enable should be gc.enable() – Paulo Raposo Nov 23 '15 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.