Applying the censusgeocode Package to an Entire Dataframe of Geocoded Data

Question

I have a data frame of latitudes and longitudes. The goal is to apply the censusgeocode package to the entire data frame where our x is our latitude column and our y is our longitude column. After this, I am hoping to get a data frame with the geoid, state, county, tract, and block as columns.

Is it possible to do this so that I do not have to iterate over every single geolocation in a for loop? I could try the apply function, however, the apply function would not necessarily be able to be adaptable to this kind of problem. Below is my code for defining the dataframe and adding the solution to:

import pandas as pd
import censusgeocode as cg
data = pd.DataFrame({'latitude':[40.811765, 40.30019, 39.2464], 'longitude':[-81.528874, -81.83743, -83.60476]})

And what I have tried so far that takes too long to do this with thousands of locations:

geo_list = []
for f, g in zip(data['x'], data['y']):
    try:
        census = cg.coordinates(x=g, y=f)['2010 Census Blocks'][0]
        data = [census['GEOID'], census['STATE'], census['COUNTY'], census['TRACT'], census['BLOCK'], f, g]
        geo_list.append(data)
    except:
        pass

Answer 1

Unfortunately you are going to be at the mercy of how fast their API can return results, since it does not support batch requests for lat/lng (it does for addresses). To push that as fast as your connection will allow, I would make a function to fetch a single location, and then map that over the data frame in parallel.

import pandas as pd
import censusgeocode as cg
from random import uniform
from concurrent.futures import ThreadPoolExecutor
from tqdm.notebook import tqdm

locations = pd.DataFrame({'latitude' :[uniform(30., 40.,)   for _ in range(10000)], 
                          'longitude':[uniform(-100., -72.) for _ in range(10000)]})

def geocode(row):
    index, lat, lng = row
    try:
        census = cg.coordinates(lng, lat)['2010 Census Blocks'][0]

        data = dict(geoid=census['GEOID'], 
                    state=census['STATE'], 
                    county=census['COUNTY'], 
                    tract=census['TRACT'], 
                    block=census['BLOCK'], 
                    lat=lat, 
                    lng=lng)

    except Exception as e:
        data = dict(lat=lat, 
                    lng=lng)

    return data

with ThreadPoolExecutor() as tpe:
     data = list(tqdm(tpe.map(geocode, locations.itertuples()), total=len(locations)))
df = pd.DataFrame.from_records(data)