For a one-time project, I need to geocode a few thousand addresses. In the past I've used various online resources for this kind of thing (e.g., Google Maps API), but the addresses I'm working with have to be kept confidential - which means no sending it over the Internet, unless there's some iron-clad guarantee of privacy.
What other options do I have?
Have a look at the Geocoder::US 2.0, the successor to geocoder.us:
http://github.com/geocommons/geocoder
It's a ruby port of the perl module by the same author.
If using the Google Geocoding API or another online source is your preference rather than local options, I'd suggest looking into the Tor Project (easily installed through the bundle called 'the Vidalia Bundle').
Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents ... the sites you visit from learning your physical location.
Along with injection of random addresses and using ssl (https) to encrypt communications to their endpoints (make sure you're also doing this), I can't think of a more secure way to geocode remotely. Whatever geocoding service you're using won't ever be able to identify where the requests ultimately came from, and with https no one else will, either. Note: don't use a geocoding service that requires an api key for this, or you'll no longer be anonymous. (Google doesn't require an api key anymore).
A side 'benefit' of this procedure is that you'll no longer be restricted to any number of geocoding requests, as your requests will look like they're coming from multiple ip addresses. However, I do not recommend or endorse abusing these lovely free APIs! Rate will still be limited if the API limits rate (though the speed of transmission using Tor is quite a bit slower than connecting directly).
Case study in Python -- Once you've installed the Vidalia Bundle and have the proxy running on 127.0.0.1:8118 (the default), in Python 2.7 or higher you can set up an https urllib2 proxy using:
import urllib2
proxy = urllib2.ProxyHandler({'https': '127.0.0.1:8118'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
response = urllib2.urlopen("https://maps.google.com/maps/geo?q=Los+Angeles&sensor=false&gl=us")
Note that urllib2 proxies don't work with https until at least Python 2.7 or so, so this method only works with recent Python versions. Make sure you've got 'https' (not 'http') in both places in the example above. I've only tested it with Python 2.7.1.
Vidalia changes your identity / apparent IP address origin every 10 minutes, but if you run into slow rates or other problems (quota exceeded errors), or if you are especially paranoid and want to change your identity more frequently, you can change your Tor identity using the python code here (slightly modified below). You'll need to change the Tor password to a static one (rather than a randomly generated one) by entering the Vidalia settings. Might also need to restart Vidalia after all changes.
p = "MySuperSecurePassword"
def renewTorIdentity():
success = False
try:
s = socket.socket()
s.connect(('localhost', 9051))
s.send('AUTHENTICATE "' + p + '"\r\n')
resp = s.recv(1024)
if resp.startswith('250'):
s.send("signal NEWNYM\r\n")
resp2 = s.recv(1024)
if resp2.startswith('250'):
success = True
except:
success = False
return success
One option is to use Geo-Coder-US, which is an open-source Perl module that uses the US Census' Tiger/Line data to geocode. I haven't used it personally, but it looks excellent. The link above includes a nice overview and a link to a version that already has the necessary Census files assembled.
The Geokit library can use any of Google, Yahoo, Geocoder.us, Geocoder.ca, and Geonames. It is written in Ruby, and there's also a sister library for your Ruby on Rails projects:
To conserve privacy, you could spread queries to all providers by separating them into sets that are less likely to be linked to your activities. You could also inject noise in your addresses by adding real addresses from an online phone directory. And I suggest you run this script from various places, such as internet cafés, combining the results at the end.
The only way to truly preserve your privacy is to download the full set of data and run your script against it. There's the Nominatim system from OpenStreetMap. It is not complete for all the cities, but you could use that to reduce the list of addresses sent to other providers.
Although still in early stages of development http://openaddresses.org/ aims to provide an open database of worldwide addresses, and associated geocoding services.
Whilst not private, the nature of an open address database could mean it is available to download in its entirety (or at least for selected regions) to allow offline geocoding.
Most of the answers are steering you toward a local database. While that would certainly work, you must also consider whether geocoding is your core domain. (Is that what you are good at? If so, you probably already have the data they are recommending. If not, AND YOU WANT IT TO BE, then you should download the data and just do it locally. However, if you just need to solve a problem and don't want to put in countless hours ramping up for production, there are still options to do it through an API without compromising security.
First, insist on HTTPS because you need the data to be secure on the way to the API and then on the way back to you. Second, ensure that you are doing a POST request instead of a GET request to the API. Using POST, you are just passing a URL request with a payload and the only results that would hit the server log is the fact that an address verification and geocoding request was made at a certain time and from a certain IP. Neither the address submitted nor the address returned would be stored to disk or written to a server log. It doesn't get much more secure than that.
So, while a local box would definitely be secure, it could require a lot of development to do what you need. Since the security concerns can be pacified, you might want to consider (again) the option of using an API.
I work for an address verification company that specializes in secure API geocoding & reverse geocoding -- SmartyStreets.
Depending of where the address are located, you could download and use OpenStreetMap features.
Check it out: http://www.openstreetmap.org/
Also, if you are in the USA there is the TIGER project along with PostGIS TigerGeocoding API. http://svn.osgeo.org/postgis/trunk/extras/tiger_geocoder/
I thought that the code behind http://geocoder.us/ was available for download such that you could get it and a TIGER data file and more-or-less set up your own local install. I don't see that immediately upon revisiting that site, but you may want to look around a bit.
Why not use the same geocoders you've used before, just remove all the other meta data?
Don't send over "Secret Location; 123 Main Street, Some City", just send over "123 Main Street, Some City"? The addresses are public information anyways. Just don't tell the geocoder that you have a list of nuclear bases or all NSA locations. The results will be in table format, you can then re-attached all your other secret meta-data.
The search on the OpenStreetMap homepage is an system called Nominatim. You can call it as a geocoding service (if you're gentle) but it's all open source, so you can set it up on your own server too.
This is using OpenStreetMap data loaded into postGiS database. It's relatively new and under development still, and the process of setting up and loading with data is not all that straightforward, and quite resource hungry. ...but it's free and open!
Old thread but worth mentioning it. http://www.tigergeocoder.com/ using TIGER 2013 data, ready to run your own server instance in Amazon EC2 cloud.
Set up PostGIS Tiger Geocoder in your local linux box. It is definitely more cumbersome than online API, but maybe the best bet for your situation. And it can scale to millions of addresses if needed.
With the help of ansible playbook, setup the server in linux is much easier than before. Wrting SQL queries probably will take more time if you are not familiar with SQL or PostGIS.
You can check my system setup and my script for more details. It should include all the information you needed.
