3

I have ~7,000 2 line dbf files (zonal stats output) that I would like merged into one table, all the same format. I have used the arcpy merge tool, as seen below.

    #merge tables together

    outfile = "C:\\Pivot\\hourly.dbf"
    arcpy.env.workspace = "C:\\Pivot\\dbfs"
    listTable = arcpy.ListTables()
    print("...")
    arcpy.Merge_management(listTable, outfile)
    print("All done!")

and it works but INCREDIBLY slowly. Adding about one file every 10 minutes. Is there a way to concatenate these files using a dbf module? Or is there a reason this is functioning so slowly? I've already processed all of the dbfs a couple times with other tools and they weren't nearly this slowly.

closed as off-topic by PolyGeo Jan 23 '17 at 10:56

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • Is it same slow using GUI? – FelixIP Jan 26 '16 at 19:44
  • Do all of your tables have the exact same schema (field names, types)? – CStarbird Jan 26 '16 at 20:57
3

If your 7000 files have the exact same number of columns you can easily do that without arcpy in a few steps:
- Fetch a list with the name of your files
- Open a first file as binary
- Fetch the length of the header of this file (it will be used for each other files, assuming they have the same number of records/columns, as you say it is an output of statistical information) and read its data
- Read the data of each other file..
- Export the merged file (as .csv for example)

import numpy as np
import struct
import os

# Fetch the name of each one of your files
os.chdir('/your_directory/with/dbf/')
file_list = [i for i in os.listdir('.') if '.dbf' in i]

# Use a first file to fetch the header length and the number of fields:
first_file = file_list.pop()
with open(first_file, 'rb') as f:
    # num_rec, len_header and num_fields are read
    #  thanks to a recipe written by Raymond Hettinger:
    num_rec, len_header = struct.unpack('<xxxxLH22x', f.read(32))
    num_fields = (len_header - 33) // 32
    f.seek(len_header)
    data = np.array(f.read().split()).reshape((num_rec, num_fields))

# Loop on the other to fetch their values and concatenate them:
for file_name in file_list:
    with open(file_name, 'rb') as f:
        f.seek(len_header)
        data = np.concatenate((
            data, np.array(f.read().split()).reshape((num_rec, num_fields))
            ))

np.savetxt("merged_stats.csv", data, delimiter=",")

Note that it will only work if your files have the same number of columns and records (and the same data-type in it, in fact due to the split method it assumes that your values aren't strings with spaces in it).
Anyway I guess it can easily be adapted to better fit your need, by reading the number of records on each file for example.

Not the answer you're looking for? Browse other questions tagged or ask your own question.