3

I'm trying to automate geoprocessing by editing simple python scripts generated from "Save as Python" and running them within the QGIS script editor framework. I am using QGIS v2.18.7 my Mac (OS X Yosemite 10.10.5).

I've noticed that when I process multiple files (currently using gdalogr:warpreproject, with 12 layers run sequentially), I always get errors of the type "ERROR 1: Output dataset /path/file.tif exists" for at least some of the operations. The files in question do not actually exist as far as I can tell (I have tried using the temporary file option as well as writing to specific files using various strange file names to be sure). Furthermore, when I rerun, it tends to alter which layers are generating the error (e.g. months 01, 04, 05 on one run and months 03, 08, 09 for the next run). There is also an error on the last layer merge operation, which I am at this point guessing is a consequence of the errors in the input layers.

What might be causing these apparently spurious tif exists errors? Is there a fix or work-around? Is there a way to specifically flag -overwrite within the runalg() framework? (as suggested here: Can't clip vector layer from a raster layer I created)

Example code:

##WarpStack12=name
##m01=raster
##m02=raster
##m03=raster
##m04=raster
##m05=raster
##m06=raster
##m07=raster
##m08=raster
##m09=raster
##m10=raster
##m11=raster
##m12=raster
##YearDat =output raster

outputs_GDALOGRWARPREPROJECT_1=processing.runalg('gdalogr:warpreproject', m01,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None,None)
outputs_GDALOGRWARPREPROJECT_2=processing.runalg('gdalogr:warpreproject', m02,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_3=processing.runalg('gdalogr:warpreproject', m03,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_4=processing.runalg('gdalogr:warpreproject', m04,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_5=processing.runalg('gdalogr:warpreproject', m05,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_6=processing.runalg('gdalogr:warpreproject', m06,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_7=processing.runalg('gdalogr:warpreproject', m07,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_8=processing.runalg('gdalogr:warpreproject', m08,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_9=processing.runalg('gdalogr:warpreproject', m09,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_10=processing.runalg('gdalogr:warpreproject', m10,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_11=processing.runalg('gdalogr:warpreproject', m11,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_12=processing.runalg('gdalogr:warpreproject', m12,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRMERGE_1=processing.runalg('gdalogr:merge', [
    outputs_GDALOGRWARPREPROJECT_1['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_2['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_3['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_4['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_5['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_6['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_7['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_8['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_9['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_10['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_11['OUTPUT'],
    outputs_GDALOGRWARPREPROJECT_12['OUTPUT'],
    ],False,True,1,YearDat)

Example error output (in this example, error shows up on 1st, 4th, and 11th layers. If I rerun, it may show up for different layers):

2017-06-15T13:15:05 0 GDAL execution console output ERROR 1: Output dataset /var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/558b56639752426697b3345fd326a7c3/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

       2017-06-15T13:15:05    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001032.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001060.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           ERROR 1: Output dataset

/var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/7fa79d45d41e40cdb0651f2242a00699/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001121.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001152.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001182.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001213.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001244.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:06    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001274.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:07    0   GDAL execution console output           ERROR 1: Output dataset

/var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/ee42542a5c0e434ea3f5c6156f9935bf/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

       2017-06-15T13:15:07    0   GDAL execution console output           Creating output file that is 1003P x 991L.
                  Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001335.Win04.006.burndate.tif.
      
      0...10...20...30...40...50...60...70...80...90...100 - done.
       2017-06-15T13:15:07    2   Uncaught error while executing algorithm            Traceback (most recent call last):
        File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/core/GeoAlgorithm.py",

line 203, in execute self.processAlgorithm(progress) File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/script/ScriptAlgorithm.py", line 378, in processAlgorithm exec((script), ns) File "", line 42, in NameError: name 'YearDat' is not defined

Improve this question
2
  • 1
    not sure if a typo, but there's a spurious space in your ##YearDat line. Try removing that in case the extra space has been added to the input name. Might explain the error you're getting (" NameError: name 'YearDat' is not defined")
    – Steven Kay
    Jun 15, 2017 at 20:59
  • 1
    @StevenKay Eliminating the space fixed all the problems. The code now executes correctly. I have to admit to being surprised that this was the cause, but I'm often surprised when debugging anything. I'm happy to post an answer for the community's benefit, but invite you to since you answered the question in a comment.
    – Tom
    Jun 15, 2017 at 21:32

1 Answer 1

Reset to default
1

there's a typo in this line...

##YearDat =output raster

the extra space before the equals sign seems to be significant. It looks as if processing includes the space in the input name, which is why you see the error

NameError: name 'YearDat' is not defined

Processing is looking for a field called "YearDat " (with a trailing space) rather than "YearDat"

I'd also recommend not using spaces in filenames or layer names, and using "_" (underscore) instead, when using GRASS or SAGA algorithms, for the same reason :)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.