Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

TIGER FTP Site

Hello everyone! I've been downloading TIGER data per county from the FTP site one at a time, with some states having hundreds of files. I was curious if there was a faster way to download maybe all the files for each county for each state at once? Thanks for your time and help!

Attached is a picture of what i'm dealing with, would like to eventually download all the files for each state at once versus one at a time.

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

Use an FTP client (e.g. FileZilla), which will allow you to download multiple files at once. After installing and starting your FTP client software:

  1. Connect to ftp2.census.gov
    1. username = anonymous
    2. password = your email address
  2. Navigate to the folder where you want to save files on your local system
  3. Navigate to the geo/tiger/TIGER2012/EDGES directory (or other directory if necessary) on the remote site
  4. Select the file(s) you want to download.
    1. depending on your software, you may be able to click one file then use a combination of shift-click or ctrl-click to select multiple files, or drag a box around the desired files
  5. Drag the selected files from the remote system to your local system, or click the software's transfer/download button
share|improve this answer
    
Of course this is fine for getting the data, but you still have to unzip each of those files, one-by-one. The solution I proposed automates the entire task from data acquisition, through unpacking, reprojecting via ogr2ogr (if desired), and finally ingest into your DB instance. –  elrobis Apr 24 '13 at 19:04
    
Im having an issue where the connection is timing out, and unable to connect to ftp2.census.gov. –  Maksim Apr 24 '13 at 19:13
    
@elrobis - your answer says that you can write a program to do all that, but doesn't explain how to. Some sample code would be beneficial. –  user3461 Apr 24 '13 at 20:10
    
@Maksim - try a couple of files, one at a time, first to see if you're still getting a timeout issue. You may have to set the transfer mode to passive in order to download multiple. The software's Help will show you how to do that. –  user3461 Apr 24 '13 at 20:12
    
im unable to connect to the ftp2.census.gov server at all. Says it times out, so unable to try files one at a time or at all. –  Maksim Apr 24 '13 at 20:23
show 3 more comments

I've done what you're attempting with the Tiger road data, so it stands to reason it would work for what you're doing here. The bold value in this filename (the last one in your list, above)..

tl_2012_04005_edges.zip

..is a so-called FIPS code. Specifically, this one is Coconino County, AZ. So all you need is something that provides a nice list of FIPS codes, then you can write a fun one-off program to concatenate URLS for each FIPS code, then download the corresponding tiger file.

In my case, I got the corresponding Tiger US counties shapefile (use the same year as your other data so the FIPS have the best 1-to-1 match), then I used Python and the OGR bindings to iterate over my shapefile and concat together a url for each county record in the shapefile, then download them one at a time (I further used ogr2ogr calls in the same automation to ingest each download into MySQL so it all happened in the same pass).

It was pretty cool. And this probably goes without saying, but you should test against only a few features, then once you have the kinks worked out, run the whole thing overnight.

share|improve this answer
add comment

I have a small python script that I cobbled together from several sources. I use it to download all the files from a directory on an FTP site, extract the files and copy them to a file geodatabase. All the zipped and extracted files are deleted at the end of the script.

It is especially useful when the files in the directory change names or you are stuck using only python.

In this instance, the files are downloaded into a permanent file called WRKSPCE, and a text file called 'Lands_data_readme.doc' is deleted.

import arcpy, ftplib, os, socket, sys, zipfile, traceback

HOST = r'ftp.geomaticsyukon.ca'
DRN = r'\CSW\land_tenure\land_tenure\'
WRKSPCE = r'c:\users\athom\documents\GeomaticsYukonFTP'

print 'The current working directory is %s' % os.getcwd()

os.chdir(WRKSPCE)

print 'The workspace has been changed to %s' % WRKSPCE

try:
    f = ftplib.FTP(HOST)
    print "CONNECTED TO HOST '%s'" % HOST
except (socket.error, socket.gaierror) as e:
    print 'Error: cannot reach "%s"' % HOST

try:
    f.login()
except ftplib.error_perm:
    print 'Error: cannot login annonymously'
    f.quit()
print 'Logged in'


try:
    f.cwd(DRN)
except ftplib.error_perm:
    print 'Error: cannot CD to "%s"' %DRN
    f.quit()
print 'Changed to "%s" folder' %DRN

try:
    filenames = f.nlst()
    print filenames

    for ftt in filenames:
        with open(os.path.join(WRKSPCE, ftt), 'wb') as local_file:
            f.retrbinary('RETR '+ ftt, local_file.write)

except ftplib.error_perm:
    print 'Error cannot read file "%s"' % FILE
    os.unlink(FILE)

f.quit()

os.remove (r'c:\users\athom\documents\GeomaticsYukonFTP\Lands_data_readme.doc')

dest_dir = WRKSPCE
try:
    s_file = os.listdir (dest_dir)
    for s_files in s_file:
        with zipfile.ZipFile(s_files) as zf:
            for member in zf.infolist():
                words = member.filename.split('/')
                path = dest_dir
                for word in words[:-1]:
                    drive, word = os.path.splitdrive(word)
                    head, word = os.path.split(word)
                    if word in (os.curdir, os.pardir, ''): continue
                    path = os.path.join(path, word)
                zf.extract(member, path)
        print 'File extracted'

except:
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    pymsg = "PYTHON ERRORS:\nTraceback Info:\n" + tbinfo + "\nError INfo:\n    "+\
    str(sys.exc_type)+ ": " + str(sys.exc_value) + "\n"
    arcpy.AddError (pymsg)
    msgs = "GP ERRORS: \n" + arcpy.GetMessages(2) + "\n"
    arcpy.AddError(msgs)

try:
    wk = arcpy.env.workspace = r'c:\users\athom\documents\GeomaticsYukonFTP'
    outloc = r'c:\users\athom\documents\arcGIS\UTM_ZONE_7.gdb\LandUse'
    arcpy.env.overwriteOutput = True
    fcs = arcpy.ListFeatureClasses()
    print fcs
    for fc in fcs:
        newfc = fc.replace('.shp', '')
        print newfc
        arcpy.FeatureClassToFeatureClass_conversion(fc, outloc, newfc, '')

except Exception as e:
    print e.message

    arcpy.AddError(e.message)

WRKSPCE = r'c:\users\athom\documents\GeomaticsYukonFTP'

os.chdir(WRKSPCE)

delfile = os.listdir (WRKSPCE)

for delf in delfile:
    os.remove (delf)
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.