2

Suppose you use various geocoding facilities (like Mapquest's API, Texas A&M's, etc.) to get a latitude-longitude pair for a given address in the United States. They give you differing answers, sometimes miles apart from one another. How should you choose among them?

Well, I suppose you can inspect the various answers on Google Maps or similar and see which makes the most sense. But now suppose you're doing this for a large dataset, where it is very impractical to inspect all the answers; you want an automated way to do it. What should you do?

I will gladly accept answers of any of the following forms:

  • a way to choose among the answers obtained
    • which might simply be a single authoritative data source to believe even if others disagree (in which case there's no need to check multiple sources in the first place, unless the good one has missing data)
  • a way to find some sort of mean value among the answers obtained
  • other
1

I'll use Texas A&M and Mapquest as examples but you can use whatever you want...

Basically, assuming you have clean data, we have to look at a couple things.

1) Can you find a pattern? (Say the difference between Texas A&M and Mapquest is usually around 2 miles. Look up Exploratory Data Analysis)

2) The case where you can't find a pattern.

3) Can we just use the average of the two?

1 is unlikely, and it usually doesn't end up being that easy for us. 3 is inaccurate in most cases, so we'll have to use some machine learning algorithms (sounds scary but it's not too bad). We can use the k-Nearest Neighbors algorithm to figure out if we should use Texas A&M or Mapquest data. Basically, In k-NN classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor. (nearest neighbors being the, sort of, training data). If we give it some good training data the system will start to weigh it pretty accurately. It will recognize patterns. This is (in my opinion) one of the best ways to do it as you will get very accurate results and the algorithm is really easy to implement.

So in a way that's easy to interpret:

It will weigh (classify) which data set it should go to based on what happened in the test data

Read more about it here

  • I don't understand. kNN classification uses a metric on the data points. That is, we assign a value (e.g. "Mapquest") to a data point (e.g. "350 5th Avenue, New York NY 10118") based on the values assigned to that point's nearest neighbors, where "nearest" is measured by some metric. I assume here you want to use the usual (great-circle) Euclidean metric; but (1) I don't understand why that is sensible. Why should Mapquest be more accurate for "350 5th Avenue, New York NY 10118" just because it's more accurate for "365 5 Av, NY NY 10016"? ... continued – JQKP Aug 31 '15 at 14:13
  • continued ... And (2) I can't measure distance, to know which is nearer, until I have the latlongs, which is what I'm trying to find. Or did you mean some other metric? Levenshtein distance on the addresses perhaps? – JQKP Aug 31 '15 at 14:13
  • Yeah, I would definitely use the Levenshtein distance. That's gonna come out with a really accurate result for the data sets. – p32094 Aug 31 '15 at 14:29
  • @JQKP you don't have something to compare to, so whatever way you choose is never going to be 100% accurate, but since mapquest APIs usually give you very clean (standard) data it should be pretty accurate across the board in regards to "Av vs Avenue" – p32094 Aug 31 '15 at 18:25
  • My addresses don't come from Mapquest. – JQKP Sep 9 '15 at 15:23
0

If you have various data sources for address-to-geocode information, and you want to find a best value, a fair and simple solution is to determine and throw out outliers and do some kind of arbitrary selection among the other sources (simple average?).

Unless you have access to Google's data (not shared) or some other very expensive data, then you are not pulling from a source that provides certain latitude and longitude information. Google and a few other big companies (such as Nokia-owned HERE) use machine learning, image recognition, and on-the-ground work to pinpoint places.

Those companies have the means to image every place on every road and measure latitude and longitude along the way or via image recognition learning algorithms. For the data sources that do not have the means to do that, they mostly base their data off of USPS and the US Census Bureau road data. This data gives latitude and longitude information for intersections and turns (US Census TIGER Products). Most address-to-geoinformation data sources take that data and extrapolate the latitude and longitude of houses along the road.

For instance, some sources take the known fact that there are five houses between point A and point B in the US Census data and arbitrarily say there is one house every fifth of the total distance. If you have a lot of these sources and average them, you will have the best latitude and longitude information possible, assuming it is not feasible for you to do Google-level imaging and machine learning.

As a side note, in my experience, TAMU's data is wrought with bad results for address searches. Some of the latitude and longitude results are just really wrong, for whatever reason.

If you're willing to pay for a service, I recommend SmartyStreets. (They do have a free option providing a few hundred lookups per month.) They are an address validation company that focuses on things like verifying and autocompleting addresses, but also works in this area of getting geocode information from the addresses. For their geocode information, they do a similar process to the one described above that uses the US Census TIGER data and multiple other sources. To get a better look at what they do, see their product page.


Full disclosure: I work for SmartyStreets.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.