Multiplying files from different folders based on similar date in filename using ArcPy?

Question

I am fairly new to Python and ArcPy.

I've been trying to get files originating from different directories and multiplying files from directory a only if directory b had files containing exact same dates within the filename. Example of file a: 20020101_105433.img Example of file b: 20020101.img

So I wrote this code thinking it might be able to recognise that 20020101_105433.img can be used to multiply against another file from the other directory that contains the same date within i.e. 20020101.img

import arcpy
import string
from  math import *
from arcpy import *
from arcpy.sa import *
import os
from os import listdir
from os.path import join

arcpy.CheckOutExtension("spatial")

BP = "H:\\NorthSea\\Images4AllDates\\BPnoEo\\"
Eo = "C:\\Users\\robyn\\Desktop\\NCEPre2\\Watts2Eo\\"
output = "C:\\Users\\robyn\\Desktop\\FinalPPresults\\"
arcpy.env.cellSize = "MINOF"

netSW = os.listdir(Eo)
chlKd = os.listdir(BP)

for files in netSW and chlKd:
    SW = os.path.join(Eo, files)
    filenameSW = SW[40:48]
    CC = os.path.join(BP, files)
    filenameCC = CC[35:43]
    if files.endswith(".img"):
        if (files in SW[40:48]) == (files in CC[35:43]):
        #or if filenameSW == filenameCC:
            Output = output + files
            BPI = Raster(SW) * Raster(CC)
            BPI.save(Output)
            print Output + " done"

The error number is 000732 (Dataset does not exist).

I've had a look at the re.compile module but also don't really understand it enough, essentially it should have this pattern: re.compile("[0-9]{8}") where there can be numbers from 0-9 and 8 characters in a sequence to match the pattern?

I have possibly got this wrong too.

Answer 1

When looping over multiple directories, you can't do it with an and like you do here:

for files in (netSW and chlKd):

(parenthesis added for clarity, it means the same as in your code).

What this does is assign netSW to files, but chlKd is never used. You can see that by placing a print files in that loop, it will show just files from one of the two folders. Here is another explanation on Stack Overflow about using and.

You are splitting your file names with indexes, e.g. here SW[40:48]. This can result in wrong/broken file names when paths change, it is indeed better to use the os.path module or regular expressions. I added those to the code below.

I might have mixed up the file names and I don't have arcpy to test right now, but I hope you get the idea of what I tried to fix.

import arcpy
import string
from  math import *
from arcpy import *
from arcpy.sa import *
import os
from os import listdir
from os.path import join
import re

arcpy.CheckOutExtension("spatial")
arcpy.env.cellSize = "MINOF"

BP = "H:\\NorthSea\\Images4AllDates\\BPnoEo\\"
Eo = "C:\\Users\\robyn\\Desktop\\NCEPre2\\Watts2Eo\\"
output = "C:\\Users\\robyn\\Desktop\\FinalPPresults\\"

netSW = os.listdir(Eo)
chlKd = os.listdir(BP)

netSW_regex = re.compile("([0-9]{8}).img") # matches 20020101_105433.img
chlKd_regex = re.compile("([0-9]{8})_([0-9]{6}).img") # matches 20020102.img

for fileSW in netSW:
    for fileCC in chlKd:
        SW = os.path.join(Eo, fileSW)
        CC = os.path.join(BP, fileCC)

        netSW_matcher = netSW_regex.match(fileSW)
        chlKd_matcher = chlKd_regex.match(fileCC)
        if netSW_matcher and chlKd_matcher and netSW_matcher.group(1) == chlKd_matcher.group(1):
            print "Multiplying {} and {}".format(fileSW, fileCC)
            target_path = os.path.join(result, fileSW)
            BPI = Raster(SW) * Raster(CC)
            BPI.save(target_path)
            print "Done {}".format(target_path)