Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Are there any free or reasonably priced databases for the US which can be searched and return latitude and longitude information?

share|improve this question
    
Fast is fine, but accuracy is everything. - Wyatt Earp When dealing with a geocoder, there are some (such as one referenced in this thread) that offer a low price to geocode many addresses, but if you are looking for very fine accuracy in many different areas, you need to go with a premium geocoder. The results will be more consistent and extraordinarily accurate, especially when the postal match does not occur. There's not much you can compete against when it comes to the most accurate geocoding, otherwise your calculations aren't going to be up to bar with other technologies offering users t –  user15251 Feb 16 '13 at 20:35
    
You offer your opinion about the need for high quality, but no suggestions. Please provide some of your suggestions, otherwise it does not help answer the users question. –  RyanDalton Feb 16 '13 at 23:27

9 Answers 9

up vote 20 down vote accepted

For that many records, don't even consider a web service. They will throttle or cut you off before you can finish your task.

So then your option becomes to run it locally, and for that you have several commercial or free options.

The free options will use the census TIGER dataset which you will need to load into a spatial database. You can find libraries that geocode against TIGER for PostGIS or even sqlite. Heck you can even use ArcGIS to geocode against TIGER. Of course, ArcGIS is not free, which brings me to the next commercial options. If you do have an ArcGIS license chances are you have StreetMap DVD with a TeleAtlas (I mean Tom Tom) or Navteq dataset. That depends if you got StreetMap Premium bundled. Any of those two datasets will probably give you more consistent results than TIGER.

Do yourself a favor and make several copies of the street database once your data is loaded and run the geocoding process on several machines with a subset of the input data. Dont try to run it on just one machine or you will be waiting for days for it to finish not to mention that most likely whatever process you run will probably leak memory and crash several times before it finishes. This means that you want to have different checkpoints for your process.

share|improve this answer
1  
I have found it beneficial to split up the input and address data into smaller portions (such as states). This makes the locators faster to use and the processes less likely to lock up after running for too long. It is usually not too difficult to script a state-based loop to do the locator creation and geocoding in one fell swoop. –  Nathanus Mar 7 '12 at 19:45

This post is probably too late to help the original poster. However, for others looking to georeference large amounts of data for free you could check out my software called "Easy Georeferencer" which is independently created, easy to use, and yet powerful (see screenshot at the bottom of the post).

The program is simple and straight-forward to use, and is run directly from an exe file requiring no installation. You can choose to geocode between the GNS or GeoNames datasource, and you can do what no other geocoder so far can do, geocode provinces based on the GADM administrative units database, as well as geocode historical country borders from the CShapes dataset. The only caveat is that it does not geocode address data. All outputs come as shapefiles ready for immediate visualization/analysis in a GIS.

As far as regards efficiency and handling of large data, the program has been tested to geocode 100 000 records in only 3 hours. For larger datasets the expected increase in processing time should drop curvilinearly because much of the processing time goes only to the initial phase when the country reference datasets are loaded, but picks up afterwards. Also, one does not have to worry about internet bottle-necks or connectivity issues when geocoding large datasets because the software, reference datasets, and processing are all based on the local computer. Match rates can get up to 80-90 percent because it is based on fuzzy-name matching accounting for spelling differences.

More details, including an introductory paper and beginner's guide are included in the download package. No need to be hesitant about trying it, the program is just a simple file that you can place and run on your desktop without any commitment or cluttering of your computer.

The software can be downloaded from: http://geocodeanything.wordpress.com/

Hope that helps.

enter image description here

share|improve this answer
    
You should always disclose that are the author/developer and/or work for the company that produces a software product. –  RyanDalton Oct 21 '13 at 4:50
    
@RyanDalton I did disclose that I was the author when I wrote "software I created", though I can see how it might have gone unnoticed because it was not given much emphasis in the sentence. Have changed my wording to hopefully make it clearer that I am condoning my own software. –  Karim Bahgat Oct 21 '13 at 13:07

The location of your data is very important because the quality of web-services change (precision, scale etc...) I geocode my adress from google api, facebook api and ex-simplegeo with my geodatabase.

Geocoding Time

http://blog.programmableweb.com/2012/06/21/7-free-geocoding-apis-google-bing-yahoo-and-mapquest/

share|improve this answer

I have not seen any reference to the level of precision you need but I am assuming that you want rooftop long lat or close to that. The quality of the input address can be a factor as well. A well cleaned list of addresses will code better and faster than a list of incomplete or bad addresses. Also, is a 90% hit rate acceptable or do you need all 20 million addresses coded? I do not have a free solution, but there is a relatively inexpensive one that I know of and use. ZP4 from Semaphore Corp, http://www.semaphorecorp.com/ offers a address scrubber/geocoder and add ons which will process the addresses and return a cleaned address, a flag which indicates if the address is USPS deliverable, and the long lat for the ZIP+4. ZIP+4 precision is generally close to rooftop precision in builtup areas (correct side of the street and on the correct block) and not close in rural areas. The cost for a 30 day license is $120. After that time the address scrubber will still function, but Delivery Point Validation (DPV) and the geo returns will not function. With a relatively quick computer, built within the last two years, and all of hte data stored and accessed locally, the 20 million records should be done in about 10 days. I have been geocoding addresses for the last 15 years and I have beeen using ZP4 for most of that time. Before they offered long-lat or DPV, I used it to pre clean addresses prior to geocoding.

share|improve this answer

As you'll be geocoding US addresses, I think the Street Address to Coordinates tool from the Data Science Toolkit should work well for you.

This API takes either a single string representing a postal address, or a JSON-encoded  
array of addresses, and returns a JSON object with a key for every address. The value 
for each key is either null if no information was found for the address, or an object 
containing location information, including country, region, city and latitude/longitude 
coordinates.

You might want to download the virtual machine though and run it from your own hardware. That way,you don't have to worry about API limits plus you control it too. Oh and did I say that it's free? ;-)

share|improve this answer
    
That's pretty amazing! I'm surprised I've never heard of it before. Plus its great that you can download the VM and have it already ready-to-run on your own system. –  RyanDalton Mar 8 '12 at 16:45
    
It's open source too. :) github.com/petewarden/dstk –  R.K. Mar 9 '12 at 2:41

I used this walkthrough describing how to build a postgis geocoder using 2010 TigerLine data. I'm running it right now - it's not fast, as it's going to take 3 weeks to geocode 2 million addresses.

However, it's free, unthrottled, and took someone with minimal coding and postgres skills less than 2 days to set up and load with one (large) state's data to begin geocoding. I've also done absolutely no postgres tuning for the system and it's running over NFS mounts, so I suspect there's one or two orders of magnitude worth of performance gains I could get out of it if I needed to.

Rather than using web services, I loaded all my addresses into the postgres database, and then I'm running a quick and dirty perl script to geocode them all one at a time:

perl -e for ($i=1; $i<[max_key_value]; $i+=1) 
   {printf "UPDATE source_addresses
               SET (rating, new_address, lon, lat) 
                     = (g.rating, pprint_addy(g.addy), 
                       ST_X(g.geomout), ST_Y(g.geomout) ) 
              FROM (SELECT DISTINCT ON (address_id) address_id, (g1.geo).* 
                      FROM (SELECT address_id, (geocode(address)) As geo 
                              FROM source_addresses As ag 
                             WHERE ag.rating IS NULL and address_id = $i 
                           ) As g1 
                     ORDER BY address_id, rating LIMIT 1
                   ) As g WHERE g.address_id = source_addresses.address_id;\n"
  } | psql -d geocoder 

(line breaks solely for readability)

So that generates a "geocode the address with this ID value and use the best match" update statement, and pipes it to psql to do it. It only attempts to geocode address with no rating - i.e. ones it's not already geocoded. So it's restartable, and each one is done independently.

share|improve this answer
    
Did you find the 2010 TigerLine data to be inaccurate for some addresses? –  Chris Muench Mar 9 '12 at 17:53
    
It's not perfect data by any means; however, the address data I have is septic from a quality perspective. The geocoder as written does provide a "rating" field, where lower is better. I'll add an example to my answer. –  Adam Musch Mar 9 '12 at 18:00
    
@ChrisMuench: If you give me one or two California addresses, I'd be glad to shoot them through my geocoder to show you - that way I'm not picking them. –  Adam Musch Mar 9 '12 at 18:11
    
Well I tried the postgis geocoder for some address in NY, and they could be off by 500 ft to 1 mile –  Chris Muench Mar 9 '12 at 18:36

I'm guessing you want to Geocode but not pay anything for it? There are a bunch of services that you can geocode 20 million records on, but it will cost you. Esri, Pitney Bowes and other offer these services via subscription or on a cost per x geocodes. 20 million isn't trivial, but I'm assuming there is a business case for this.

share|improve this answer

I work at SmartyStreets (an address verification company). Our service is free for nonprofit organizations (school, library, church, charity...), so if you fit that classification, there's no charge for our unlimited service.

Ragi recommends against a web-service, however, our API can easily clean, standardize and geocode 20 million addresses for you in about 5 hours (approximately 1000 per second). Some of that time will depend on the speed of your machine (how many cores you have) and your network connection (don't try it over 3G, but a standard broadband connection will do just fine).

Just wanted to point out that it is certainly possible with a webservice.

share|improve this answer
1  
Well, if you are going to let him geocode 20 million records without charging him a dime and can do 1000/requests per seconds (extremely impressive), then of course this is a better solution than building your own stack from scratch. –  Ragi Yaser Burhum Mar 7 '12 at 20:39
1  
Absolutely. 1000/second not the max. It is just a good standard. Multithreading, multiple cores, and faster network connections could even increase that throughput. We have a recent nonprofit customer that just used our service to process 180 million addresses. With national elections looming, lots of groups are trying to clean and geocode their addresses. –  Jeffrey Mar 7 '12 at 20:49
1  
@RagiYaserBurhum Note that Chris is also seeking "reasonably priced" solutions. Sure, building your own stack from scratch might be "free," but is it really, when you consider your opportunity cost? The benefit you gain by doing it all yourself: potentially no costs ("free"). The benefit of using an existing service: Saving many hours and hours of time. Plus, a thousand requests/second is probably routine these days with modern web architectures (parallelization) to deal with constant traffic loads. In the end, I think for this matter, "free" and" reasonably priced" is a matter of preference. –  Matt Mar 7 '12 at 20:51
1  
@Matt Hands down "free" is not always free.I will not argue against that (your time is worth money). I am well aware that the cost for most open source solution is actually a shift in the cost model from usage licenses to support services. I was not trying to be ironic, I do think that if Jeffey can offer the solution for free (or even reasonably priced) this is the better solution. At the best listed price from SmartyStreets, 20m points would cost $100,000. I am sure even for commercial use cases they can do a fraction of the cost. –  Ragi Yaser Burhum Mar 7 '12 at 21:02
    
Also 1000/req per second, which is very doable in contemporary architectures, would still require some awesome engineering for geocoding. They are getting my props for doing that :) –  Ragi Yaser Burhum Mar 7 '12 at 21:04

if your geocode request does not exceed 2,500 per day, you can use The Google Geocoding API . you should take a glance to api, it can return to results as json or xml.

Usage Limit:

Use of the Google Geocoding API is subject to a query limit of 2,500 geolocation requests per day. (User of Google Maps API for Business may perform up to 100,000 requests per day.)

Example:

http://maps.google.com/maps/geo?key=yourkeyhere&output=json&q=520+3rd+Street+San+Francisco+CA

Example Result:

{
  "name": "520 3rd Street San Francisco CA",
  "Status": {
    "code": 200,
    "request": "geocode"
  },
  "Placemark": [ {
    "id": "p1",
    "address": "520 3rd St, San Francisco, Kaliforniya 94107, Amerika Birleşik Devletleri",
    "AddressDetails": {
   "Accuracy" : 8,
   "Country" : {
      "AdministrativeArea" : {
         "AdministrativeAreaName" : "CA",
         "SubAdministrativeArea" : {
            "Locality" : {
               "LocalityName" : "San Francisco",
               "PostalCode" : {
                  "PostalCodeNumber" : "94107"
               },
               "Thoroughfare" : {
                  "ThoroughfareName" : "520 3rd St"
               }
            },
            "SubAdministrativeAreaName" : "San Francisco"
         }
      },
      "CountryName" : "USA",
      "CountryNameCode" : "US"
   }
},
    "ExtendedData": {
      "LatLonBox": {
        "north": 37.7826364,
        "south": 37.7799384,
        "east": -122.3942267,
        "west": -122.3969247
      }
    },
    "Point": {
      "coordinates": [ -122.3955757, 37.7812874, 0 ]
    }
  } ]
}

and you can check some example link for geocoding from google:

1. Single Code

2. Reverse GeoCoding

3. Starting with Google GeoCoding

i hope it helps u

share|improve this answer
11  
That would only take about 22 years (8,000 days) to geocode 20 million locations. Totally reasonable solution. –  Andy W Mar 7 '12 at 15:01
    
i know this and I have expressed if his geocode request does not exceed 2,500 per day, he can use it...this is an option if he cant find any solution –  Aragon Mar 7 '12 at 15:52
4  
This violates the terms of service. It is ilegal unless you display the results on a Google Map. Even if you do, I find the "caching argument" that some people use a little bit stretched out for 20 million features! –  Ragi Yaser Burhum Mar 7 '12 at 16:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.