Geopandas script won't perform overlay calculations

Question

I have a txt that contains in each line a sentence like this:

INT_ASTIK_EAS
ER_ASTOM_ASTIK
and so on.

I want to ask for a directory of the main folder that contains the subfolders that each contain the ASTIK.shp, then the EAS.shp, then the ASTOM.shp, etc

INT means to perform an intersection with this code where in this case should involve ASTIK and EAS shapefiles.

inte_s=gpd.overlay(ASTIK,EAS,how='intersection')

the code:

rootdir = r'C:\Users\user\Desktop\a' # path to your root directory you walk
sfiles = {} # a dictionary with all the .shp files
for entry in os.listdir(rootdir):
    dirpath = os.path.join(rootdir, entry)
    if os.path.isdir(dirpath): 
        for file in os.listdir(dirpath): # Get all files in the subdirectories
            if file.endswith('.shp'): # If it's an .shp.
                filepath = os.path.join(dirpath, file)
                sfiles[filepath] = gpd.read_file(filepath)


action_dict = {'INT': 'intersection', 'ER': 'difference'}


directory=input('Insert dir of the main folder')
with open(input()) as txtfile: #insert directory of txt
    x = txtfile.readlines()
for line in x:
    action, shape1, shape2 = line.split('_')  # here line is ER_ASTOM_ASTIK or whatever line in the txt file
    if shape1 in sfiles and shape2 in sfiles:
        gpd.overlay(sfiles[shape1], sfiles[shape2], how=action_dict[action])

It doesn't probably perform the intersection or difference. How can i store its results in a variable because i have to do further process with the outcomes of every operation of every line.

After checking what's the issue with the sfiles variable and what it stores: it gives this:

for x in sfiles:
    print (x)

C:\Users\user\Desktop\a\ASTTOM\ASTTOM.shp
C:\Users\user\Desktop\a\PST\PST.shp

and finally:

if shape1 in sfiles:
    print('success')
else:
    print('failed')

failed

which means it doesn't communicate what it should in the dictionary because the shapefiles happen to exist in the folder. How can i make it do the overlay calculations?

print(sfiles)

{'C:\\Users\\user\\Desktop\\a\\ASTTOM\\ASTTOM.shp':      OBJECTID  FID_Asttom TheOTA_Tmp  aa    SHAPE_Leng    SHAPE_Area  \
0           1           1      20008   7   4206.346569  5.796467e+05   
1           2           2      20008  11   3311.521073  6.892735e+05   
2           3           3      20008   6   2131.153007  2.196689e+05   
3           4           4      20008   2   1738.249605  4.731495e+04   
4           5           5      20008   1   1075.903789  5.337986e+04   
5           6           6      20008  10   3255.422426  3.628100e+05   
6           7           7      20008   3   2469.427102  2.267493e+05   
7           8           8      20008   5   2466.593786  2.042149e+05   
8           9           9      20008   8   3095.473108  5.384131e+05   
9          10          10      20008   4   1856.659243  1.136976e+05   
10         11          11      20008   9   2293.865083  2.063858e+05   
11         12          12      20104   4   4445.338873  4.455547e+05   
12         13          13      20104   7   3659.621424  2.834771e+05   
13         14          14      20104   2   2098.126160  1.981298e+05   

it prints the attribute tables of all the shps inside the dictionary

if attempt this:

print(sfiles[shape1], sfiles[shape2], action_dict[action])

KeyError                                  Traceback (most recent call last)
<ipython-input-11-9f435f40293e> in <module>()
----> 1 print(sfiles[shape1], sfiles[shape2], action_dict[action])

KeyError: 'ASTTOM'


and each every time:
print(sfiles[shape1])
KeyError: 'ASTTOM'

print(sfiles[shape2])
KeyError: 'PST'

print(action_dict[action])
difference

---

What it gets printed with the new code:

print(group)
['C:\\Users\\user\\Desktop\\a\\ASTTOM\\ASTTOM.shp']

print(grouped_shapefiles)
[['C:\\Users\\user\\Desktop\\a\\ASTTOM\\ASTTOM.shp'], ['C:\\Users\\user\\Desktop\\a\\PST\\PST.shp']]

print(command)
INT_ASTTOM_PST


print(commands)
['INT_ASTTOM_PST', 'ER_ASTTOM_PST']

command.split('_')
['INT', 'ASTIK', 'EAS']

and finally df1 prints the attribute table of the first shp

Answer 1

It is still unclear exactly what you are asking, for example what output do you want - one for each row in the textfile or are the rows steps in an algorithm to produce one output? .

I think your problems are in the way you are listing paths to shapefiles and using dictionaries. Try:

import os
import geopandas as gpd

commandfile = open("C:\Test\commands.txt","r")
rootfolder = r'C:\Test\Root_folder'

def splitpath(path):
    folders = []
    while 1:
        path, folder = os.path.split(path)

        if folder != "":
            folders.append(folder)
        else:
            if path != "":
                folders.append(path)

            break
    folders.reverse()
    return folders

#List all shapefiles as full path and filename
shapelist = []
for path, subdirs, files in os.walk(rootfolder):
    for name in files:
        if name.endswith('.shp'):
            shapelist.append(os.path.join(path, name))

#List all subfolders
subfolders = set(map(lambda x:splitpath(x)[-3], shapelist))
#Create list of sublists where each sublist are the shapefiles in that subfolder
grouped_shapefiles = [[y for y in shapelist if splitpath(y)[-3]==x] for x in subfolders]

#List commands
commands = commandfile.read().splitlines()

action_dict = {'INT': 'intersection', 'ER': 'difference'}

results = []
for group in grouped_shapefiles:
    for command in commands:
        #print(command)
        df1 = gpd.read_file([g for g in group if command.split('_')[1] in g][0])
        df2 = gpd.read_file([g for g in group if command.split('_')[2] in g][0])
        df = gpd.overlay(df1,df2,how=action_dict[command.split('_')[0]])
        results.append(df)
#Do something will the results...

It is important the folder structure is exactly like this (rootfolder\subfolder (which is used to group together shapefiles to overlay)\shapefolder1,2..n) or the code will have to be modified: