I have 13000+ CSV files, all containing around 13000 rows. These CSV files are arranged in 146 different folders. All CSV files have the same schema with
; as separator. These CSV files contain about 175 million rows of data combined. The folders are named as seen in the picture below (for example ...\CSV_folders\5785xxx)
CSV files are named like these:
I already tried to copy all CSV files to same directory and run them to one GDB with FME. This went well... but resulted too heavy ESRI GDB that takes forever to search through. (175 million rows, around 20GB)
Now I would need to make things a bit differently. I would like to work with FME to create one ESRI Geodatabase. It should contain tables named like these folders containing CSV files. A table in the GDB should have all the CSV files loaded in the table, that are inside the original folder. This would split my 175 million rows to 146 tables, making it a bit faster to search through with Arcmap.
So my problem is, how can I make FME read CSV files from all of the folders and write a GDB with tables named like the folder, each table containing the same information the CSV files have in the original folder combined? I'm not familiar with PostGIS etc. (and it would not work with ArcGIS), so this needs to be done with some kind of ESRI GDB workaround..