OGR error "is not a directory" when moving to Windows from UNIX

Question

I have to write a script that works either on Windows or macOS.

I want to run these lines :

processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///D:/extract_TSX/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no', 'OUTPUT' : 'D:/extract_TSX/SHP/input.shp' })
processing.run('native:buffer',{ 'DISSOLVE' : False, 'DISTANCE' : 2.5, 'END_CAP_STYLE' : 2, 'INPUT' : 'D:/extract_TSX/SHP/input.shp', 'JOIN_STYLE' : 1, 'MITER_LIMIT' : 2, 'OUTPUT' : 'D:/extract_TSX/SHP/windows_1px/windows_1px.shp', 'SEGMENTS' : 1 })       

When I run it from QGIS it works perfectly with both Windows and macOS. When I run it directly from python3 it works on macOS but I get the error below when I run it on Windows.

ERROR 1: D:/extract_TSX/SHP/input.shp is not a directory.
Traceback (most recent call last):
  File "D:\extract_TSX\extract_TSX_w.py", line 292, in <module>
    windows(1,3,11)
  File "D:\extract_TSX\extract_TSX_w.py", line 51, in windows
    processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///D:/extract_TSX/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no', 'OUTPUT' : 'D:/extract_TSX/SHP/input.shp' })
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\tools\general.py", line 108, in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\core\Processing.py", line 189, in runAlgorithm
    ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\gui\AlgorithmExecutor.py", line 71, in execute
    results, ok = alg.run(parameters, context, feedback, {}, False)
_core.QgsProcessingException: Could not create layer D:/extract_TSX/SHP/input.shp: Creation of data source failed (OGR error: D:/extract_TSX/SHP/input.shp is not a directory.)

Since the error is :

OGR error: D:/extract_TSX/SHP/input.shp is not a directory.

I do not think it is cause by the inversion / and \. But why not ?

As you can see the file input.shp has not been written properly :

D:\extract_TSX\SHP>dir
 Le volume dans le lecteur D s’appelle Photos
 Le numéro de série du volume est 6260-5FA6

 Répertoire de D:\extract_TSX\SHP

20/04/2022  18:32    <DIR>          .
20/04/2022  18:32    <DIR>          ..
20/04/2022  21:56    <DIR>          windows_1px
20/04/2022  21:56    <DIR>          windows_11px
20/04/2022  21:56    <DIR>          windows_3px
               0 fichier(s)                0 octets
               5 Rép(s)  953 589 825 536 octets libres

D:\extract_TSX\SHP>cd ..

D:\extract_TSX>python3 run_w.py

# Starting QGIS...

# Exporting D:\extract_TSX/XLSX/input.xlsx to D:\extract_TSX/CSV/input.csv...

# 1px windows were created
ERROR 1: D:/extract_TSX/SHP/input.shp is not a directory.
Traceback (most recent call last):
  File "D:\extract_TSX\extract_TSX_w.py", line 291, in <module>
    windows(1,3,11)
  File "D:\extract_TSX\extract_TSX_w.py", line 50, in windows
    processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///D:/extract_TSX/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no', 'OUTPUT' : 'D:/extract_TSX/SHP/input.shp' })
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\tools\general.py", line 108, in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\core\Processing.py", line 189, in runAlgorithm
    ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\gui\AlgorithmExecutor.py", line 71, in execute
    results, ok = alg.run(parameters, context, feedback, {}, False)
_core.QgsProcessingException: Could not create layer D:/extract_TSX/SHP/input.shp: Creation of data source failed (OGR error: D:/extract_TSX/SHP/input.shp is not a directory.)

D:\extract_TSX>cd SHP

D:\extract_TSX\SHP>dir
 Le volume dans le lecteur D s’appelle Photos
 Le numéro de série du volume est 6260-5FA6

 Répertoire de D:\extract_TSX\SHP

20/04/2022  18:32    <DIR>          .
20/04/2022  18:32    <DIR>          ..
20/04/2022  21:56    <DIR>          windows_1px
20/04/2022  21:56    <DIR>          windows_11px
20/04/2022  21:56    <DIR>          windows_3px
23/04/2022  09:17             1 276 input.shp
23/04/2022  09:17             2 608 input.dbf
               2 fichier(s)            3 884 octets
               5 Rép(s)  953 589 563 392 octets libres

The error come from the line native:buffer because when I run the script without this line no errors are raised.

Based on this topic I tried to export the output as something else than .shp. Then I get a new error :

Could not create layer D:/extract_TSX/SHP/input.gpkg: Creation of data source failed (OGR error: A file system object called 'D:/extract_TSX/SHP/input.gpkg' already exists.)

But as you can see below it does not exist...

D:\extract_TSX\SHP>dir
 Le volume dans le lecteur D s’appelle Photos
 Le numéro de série du volume est 6260-5FA6

 Répertoire de D:\extract_TSX\SHP

20/04/2022  18:32    <DIR>          .
20/04/2022  18:32    <DIR>          ..
20/04/2022  21:56    <DIR>          windows_1px
20/04/2022  21:56    <DIR>          windows_11px
20/04/2022  21:56    <DIR>          windows_3px
               0 fichier(s)                0 octets
               5 Rép(s)  953 586 941 952 octets libres

D:\extract_TSX\SHP>cd ..

D:\extract_TSX>python3 run_w.py

# Starting QGIS...

# Exporting D:\extract_TSX/XLSX/input.xlsx to D:\extract_TSX/CSV/input.csv...

# 1px windows were created
ERROR 1: A file system object called 'D:/extract_TSX/SHP/input.gpkg' already exists.
Traceback (most recent call last):
  File "D:\extract_TSX\extract_TSX_w.py", line 291, in <module>
    windows(1,3,11)
  File "D:\extract_TSX\extract_TSX_w.py", line 50, in windows
    processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///D:/extract_TSX/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no', 'OUTPUT' : 'D:/extract_TSX/SHP/input.gpkg' })
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\tools\general.py", line 108, in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\core\Processing.py", line 189, in runAlgorithm
    ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
  File "D:\QGIS 3.22.6/apps/qgis-ltr/python/plugins\processing\gui\AlgorithmExecutor.py", line 71, in execute
    results, ok = alg.run(parameters, context, feedback, {}, False)
_core.QgsProcessingException: Could not create layer D:/extract_TSX/SHP/input.gpkg: Creation of data source failed (OGR error: A file system object called 'D:/extract_TSX/SHP/input.gpkg' already exists.)

Reference :

• Fixing "Error: creation of data source failed (OGR error:Failed to create directory .shp for shapefile datastore.)"
• Failure creating output file: (OGR error: file location not a directory)
• OGR error: file location not a directory [closed]
• Getting error while adding few layers to GeoPackage using PyQGIS?

Answer 1

Explanations

ERROR 1: D:/extract_TSX/SHP/input.shp is not a directory.

A very common interpretation of this error through a lot of forum is the following :

You have chosen a file, where a folder is expected.

But this do not solve the error at all and even created a new error because it is expecting a file and not a folder... In fact, it seems that the right interpretation is the one that has been proposed here from @user2856 :

It's happening because file already exists when you try to run

Let's be more clear about this. Here is my code (working on macOS but not on Windows) :

    for px in format_tuiles:

    processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///Users/pugliesipc/Desktop/extract_TSX.nosync/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no','OUTPUT':pwd+'/SHP/input.shp'})
    processing.run('native:buffer',{'DISSOLVE':False,'DISTANCE':px*2.5,'END_CAP_STYLE':2,'INPUT':pwd+'/SHP/input.shp','JOIN_STYLE':1,'MITER_LIMIT':2,'OUTPUT':pwd+'/SHP/windows_'+str(px)+'px/windows_'+str(px)+'px.shp','SEGMENTS':1})

As you can see this code is not optimized because the line native:refactorfields could be outside the loop since it does not depend on the px variable. Because it was working I did not pay attention to this and the file input.shp is overwrite each loop.

Solution

In short, here what the code should has been (working on Windows and macOS):

processing.run('native:refactorfields',{ 'FIELDS_MAPPING' : [{'expression': '"Name"','length': 10,'name': 'Name','precision': 0,'type': 10},{'expression': '"UTM32_E"','length': 0,'name': 'UTM32_E','precision': 0,'type': 6},{'expression': '"UTM32_N"','length': 0,'name': 'UTM32_N','precision': 0,'type': 6}], 'INPUT' : 'delimitedtext://file:///Users/pugliesipc/Desktop/extract_TSX.nosync/CSV/input.csv?type=csv&maxFields=10000&detectTypes=yes&xField=UTM32_E&yField=UTM32_N&crs=EPSG:32632&spatialIndex=no&subsetIndex=no&watchFile=no','OUTPUT':pwd+'/SHP/input.shp'})

for px in format_tuiles:
    
    processing.run('native:buffer',{'DISSOLVE':False,'DISTANCE':px*2.5,'END_CAP_STYLE':2,'INPUT':pwd+'/SHP/input.shp','JOIN_STYLE':1,'MITER_LIMIT':2,'OUTPUT':pwd+'/SHP/windows_'+str(px)+'px/windows_'+str(px)+'px.shp','SEGMENTS':1})

What does that mean ?

I guess that is not allow to overwrite files you just create from QGIS standalone on Windows and it is on macOS. Maybe you can change this set up by using the command QgsVectorFileWriter.SaveVectorOptions() but I have not try. More details on this here.

Concerning / and \ in path

Because you move from UNIX to Windows you might think the error come from this. Luckily things are going well and as you can read here and here Windows understands almost every time when / is used rather than \. You can even mix it !

Here an example :

pwd=os.path.dirname(os.path.realpath(__file__))
processing.run('native:buffer',{'DISSOLVE':False,'DISTANCE':px*2.5,'END_CAP_STYLE':2,'INPUT':pwd+'/SHP/input.shp','JOIN_STYLE':1,'MITER_LIMIT':2,'OUTPUT':pwd+'/SHP/windows_'+str(px)+'px/windows_'+str(px)+'px.shp','SEGMENTS':1})

For UNIX the input would be write as below :

/Users/pugliesipc/Desktop/SHP/input.shp

While on Windows the input would be like this :

C:\Users\pugliesipc\Desktop/SHP/input.shp

Then you do not need to find and replace all the / with \ each time you from one operating system to another.