Bulk geocoding 20 million US addresses [closed]

Question

Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?

Answer 1

For that many records, don't even consider a web service. They will throttle or cut you off before you can finish your task.

So then your option becomes to run it locally, and for that you have several commercial or free options.

The free options will use the census TIGER dataset which you will need to load into a spatial database. You can find libraries that geocode against TIGER for PostGIS or even sqlite. Heck you can even use ArcGIS to geocode against TIGER. Of course, ArcGIS is not free, which brings me to the next commercial options. If you do have an ArcGIS license chances are you have StreetMap DVD with a TeleAtlas (I mean Tom Tom) or Navteq dataset. That depends if you got StreetMap Premium bundled. Any of those two datasets will probably give you more consistent results than TIGER.

Do yourself a favor and make several copies of the street database once your data is loaded and run the geocoding process on several machines with a subset of the input data. Dont try to run it on just one machine or you will be waiting for days for it to finish not to mention that most likely whatever process you run will probably leak memory and crash several times before it finishes. This means that you want to have different checkpoints for your process.

Answer 2

I work at SmartyStreets (an address verification company). Our service is free for everyone (up to the basic level). Startups can also request to use our service completely free for the first year. So if you fit that classification, there's no charge for our unlimited service for a year.

Ragi recommends against a web-service, however, our API can easily clean, standardize and geocode 20 million addresses for you in about 5 hours (approximately 1000 per second). Some of that time will depend on the speed of your machine (how many cores you have) and your network connection (don't try it over 3G, but a standard broadband connection will do just fine).

Just wanted to point out that it is certainly possible with a web service.

---

As of 1 Aug 2017, I have remotely tested our web service and gotten a sustained 70,000 lookups per second using only a single 2015 MacBook Pro on a wireless network. Yeah, it's pretty fast. That means a small list like 20 million addresses would only take about 5 minutes.

Answer 3

I used this walkthrough describing how to build a postgis geocoder using 2010 TigerLine data. I'm running it right now - it's not fast, as it's going to take 3 weeks to geocode 2 million addresses.

However, it's free, unthrottled, and took someone with minimal coding and postgres skills less than 2 days to set up and load with one (large) state's data to begin geocoding. I've also done absolutely no postgres tuning for the system and it's running over NFS mounts, so I suspect there's one or two orders of magnitude worth of performance gains I could get out of it if I needed to.

Rather than using web services, I loaded all my addresses into the postgres database, and then I'm running a quick and dirty perl script to geocode them all one at a time:

perl -e for ($i=1; $i<[max_key_value]; $i+=1) 
   {printf "UPDATE source_addresses
               SET (rating, new_address, lon, lat) 
                     = (g.rating, pprint_addy(g.addy), 
                       ST_X(g.geomout), ST_Y(g.geomout) ) 
              FROM (SELECT DISTINCT ON (address_id) address_id, (g1.geo).* 
                      FROM (SELECT address_id, (geocode(address)) As geo 
                              FROM source_addresses As ag 
                             WHERE ag.rating IS NULL and address_id = $i 
                           ) As g1 
                     ORDER BY address_id, rating LIMIT 1
                   ) As g WHERE g.address_id = source_addresses.address_id;\n"
  } | psql -d geocoder 

(line breaks solely for readability)

So that generates a "geocode the address with this ID value and use the best match" update statement, and pipes it to psql to do it. It only attempts to geocode address with no rating - i.e. ones it's not already geocoded. So it's restartable, and each one is done independently.

Answer 4

I'm geocoding 18 millions addresses right now so I want to share my setup.

Basically I used an ansible playbook to setup Amazon EC2 instance as a PostGIS Tiger Geocoder server, then use script to batch geocode addresses, map them to census block.

The cost of Amazon EC2: minimum requirement 180G SSD cost about $18/month. My t2.large instance cost about $90/month in total.

If you have a linux box with SSD > 180G, you can also use it basically free.

My average performance is about 170 ~ 300 ms / good quality address, 400 ~ 600 ms / bad quality address. By bad quality I mean lots of them have missing or wrong city, zipcode, or even just wrong address. This kind of address take much longer time to geocode, so your performance depend on the quality of the input. The Amazon EC2 instance I used can geocode about 4 million really bad formatted addresses in one month.

For more details, see my blog post about the system setup and scripts

Answer 5

The location of your data is very important because the quality of web-services change (precision, scale etc...) I geocode my adress from google api, facebook api and ex-simplegeo with my geodatabase.

http://blog.programmableweb.com/2012/06/21/7-free-geocoding-apis-google-bing-yahoo-and-mapquest/

Answer 6

You could check out my software called "Easy Georeferencer" which is independently created, easy to use, and yet powerful (see screenshot at the bottom of the post).

The program is simple and straight-forward to use, and is run directly from an exe file requiring no installation. You can choose to geocode between the GNS or GeoNames datasource, and you can do what no other geocoder so far can do, geocode provinces based on the GADM administrative units database, as well as geocode historical country borders from the CShapes dataset. The only caveat is that it does not geocode address data. All outputs come as shapefiles ready for immediate visualization/analysis in a GIS.

As far as regards efficiency and handling of large data, the program has been tested to geocode 100 000 records in only 3 hours. For larger datasets the expected increase in processing time should drop curvilinearly because much of the processing time goes only to the initial phase when the country reference datasets are loaded, but picks up afterwards. Also, one does not have to worry about internet bottle-necks or connectivity issues when geocoding large datasets because the software, reference datasets, and processing are all based on the local computer. Match rates can get up to 80-90 percent because it is based on fuzzy-name matching accounting for spelling differences.

More details, including an introductory paper and beginner's guide are included in the download package. No need to be hesitant about trying it, the program is just a simple file that you can place and run on your desktop without any commitment or cluttering of your computer.

The software can be downloaded from: http://geocodeanything.wordpress.com/

Answer 7

There are a bunch of services that you can geocode 20 million records on, but it will cost you. Esri, Pitney Bowes and other offer these services via subscription or on a cost per x geocodes. 20 million isn't trivial, but I'm assuming there is a business case for this.

Answer 8

As you'll be geocoding US addresses, I think the Street Address to Coordinates tool from the Data Science Toolkit should work well for you.

This API takes either a single string representing a postal address, or a JSON-encoded  
array of addresses, and returns a JSON object with a key for every address. The value 
for each key is either null if no information was found for the address, or an object 
containing location information, including country, region, city and latitude/longitude 
coordinates.

You might want to download the virtual machine though and run it from your own hardware. That way,you don't have to worry about API limits plus you control it too. Oh and did I say that it's free? ;-)

Answer 9

I have not seen any reference to the level of precision you need but I am assuming that you want rooftop long lat or close to that. The quality of the input address can be a factor as well. A well cleaned list of addresses will code better and faster than a list of incomplete or bad addresses. Also, is a 90% hit rate acceptable or do you need all 20 million addresses coded? I do not have a free solution, but there is a relatively inexpensive one that I know of and use. ZP4 from Semaphore Corp, http://www.semaphorecorp.com/ offers a address scrubber/geocoder and add ons which will process the addresses and return a cleaned address, a flag which indicates if the address is USPS deliverable, and the long lat for the ZIP+4. ZIP+4 precision is generally close to rooftop precision in builtup areas (correct side of the street and on the correct block) and not close in rural areas. The cost for a 30 day license is $120. After that time the address scrubber will still function, but Delivery Point Validation (DPV) and the geo returns will not function. With a relatively quick computer, built within the last two years, and all of hte data stored and accessed locally, the 20 million records should be done in about 10 days. I have been geocoding addresses for the last 15 years and I have beeen using ZP4 for most of that time. Before they offered long-lat or DPV, I used it to pre clean addresses prior to geocoding.

Answer 10

If your workflow is in Python, geopy is a great solution. You can write your basic geocoding logic abstractly, and then choose from one of many providers (ArcGIS, Baidu, Bing, DataBC, GeocodeFarm, GoecoderDotUS, GeoNames, Google, IGN France, LiveAddress, NaviData, Nominatim (OSM), OpenCage, OpenMapQuest, Yahoo! BOSS Place Finder, What3Words, Yandex... phew). You could even use all of them and ultimately choose the point that has the highest confidence of being correct through validation by multiple geocoding services. Many of these services require registration (but not all). They may not be equally suitable for use in the USA, but the benefit of doing it with a neat Python function is that you should be able to control for that.

Here's a short example:

from geopy.geocoders import Nominatim
geolocator = Nominatim(country_bias='New Zealand', timeout=4)
geocoded = geolocator.geocode('Raglan, Waikato', exactly_one=True)
if geocoded is not None:
    # success
    lat = geocoded.latitude
    lon = geocoded.longitude

https://geopy.readthedocs.org/en/1.10.0/

Really the only hard part is ensuring that your addresses aren't poorly formed. It will probably still take a considerable amount of time to geocode 20 million locations, however... and that kind of load will probably violate someone's terms of service. Hope this helps someone though.

Answer 11

AWS has a number of geocoding servers available now. Get your own server and dump your data there. https://aws.amazon.com/marketplace/search/results/ref=dtl_navgno_search_box?page=1&searchTerms=geocode

https://aws.amazon.com/marketplace/pp/B013CW6HOA

https://aws.amazon.com/marketplace/pp/B0112MQ63Y

Answer 12

if your geocode request does not exceed 2,500 per day, you can use The Google Geocoding API . you should take a glance to api, it can return to results as json or xml.

Usage Limit:

Use of the Google Geocoding API is subject to a query limit of 2,500 geolocation requests per day. (User of Google Maps API for Business may perform up to 100,000 requests per day.)

Example:

http://maps.google.com/maps/geo?key=yourkeyhere&output=json&q=520+3rd+Street+San+Francisco+CA

Example Result:

{
  "name": "520 3rd Street San Francisco CA",
  "Status": {
    "code": 200,
    "request": "geocode"
  },
  "Placemark": [ {
    "id": "p1",
    "address": "520 3rd St, San Francisco, Kaliforniya 94107, Amerika Birleşik Devletleri",
    "AddressDetails": {
   "Accuracy" : 8,
   "Country" : {
      "AdministrativeArea" : {
         "AdministrativeAreaName" : "CA",
         "SubAdministrativeArea" : {
            "Locality" : {
               "LocalityName" : "San Francisco",
               "PostalCode" : {
                  "PostalCodeNumber" : "94107"
               },
               "Thoroughfare" : {
                  "ThoroughfareName" : "520 3rd St"
               }
            },
            "SubAdministrativeAreaName" : "San Francisco"
         }
      },
      "CountryName" : "USA",
      "CountryNameCode" : "US"
   }
},
    "ExtendedData": {
      "LatLonBox": {
        "north": 37.7826364,
        "south": 37.7799384,
        "east": -122.3942267,
        "west": -122.3969247
      }
    },
    "Point": {
      "coordinates": [ -122.3955757, 37.7812874, 0 ]
    }
  } ]
}

and you can check some example link for geocoding from google:

1. Single Code

2. Reverse GeoCoding

3. Starting with Google GeoCoding