Removing duplicate data using FME

Question

There are three roadkill data sets from three different institutions. We are sure that a combined data set will have duplicates.

The three institutions will probably record their data differently:

• coordinates for roadkills with different tools - handheld GPS, GPS in phones etc. So we are looking at a probability of the same incidence being recorded with coordinates with up to 300 meters distance.
• time might be recorded different. Most probably the same day, but this can also be fuzzy. Up to 2 days difference. Where one institution records the incidence through field officers (police, rangers, etc), the other could record this by asking for week number and then set an arbitrary date within that week.

Thus we are looking at a two dimensional distance calculation consisting of time and physical distance. Duch closeness could be calculated based on relative closeness (establishing a distance list for each object related to all other objects for this attribute, or an absolute distance filter (categorizing distances).

I guess the way to do this is to run subsequent matching according to dates and coordinates. I have been looking at the different libraries in FME 2015.0 but have not found any which can use coordinate closeness and time closeness as filters.

The hard way to find neighboring coordinates is to pull the coordinates out as numbers and then do a comparison. It will of course mean I have to populate an array and work on that array. I could then export it and use a small python script to do the job. I was hoping to avoid that.

Does anyone have examples of such duplicate procedures and their parameters/constraints using FME?

Answer 1

FME does not provide the tools to do this with the available transformers. This solution describes how the coordinate check can be done using python (PythonCaller). It is based on an answer to a question at stackoverflow.

This is the proposed procedure which can be programmed in Python:

1. Coordinates are made an attribute
2. A list of candidate coordinates is established and written to disk

3. All object coordinates are compared to the candidates using the following python procedure called from PythonCaller (code draft):

   candList= Pulled from text file on disk

   coordinate = get from feature (feature.getAttribute)

   nearest = min(candList, key=lambda x: distance(x, coordinate))

4. Where distance distance(a, b) is a function which returns the distance between coordinates a and b. If the nearest coordinate pair is within 300 meters it is considered a coordinate duplicate.

5. An attribute with pair numbers are added.
6. The pairs are considered for date duplication/closeness (to be described).
7. Final candidates are then merged.

Other platforms could be considered. The model builder in QGIS is flexible and could represent a useful platform. GeoKettle could also be considered.