# Removing duplicate data using FME

There are three roadkill data sets from three different institutions. We are sure that a combined data set will have duplicates.

The three institutions will probably record their data differently:

• coordinates for roadkills with different tools - handheld GPS, GPS in phones etc. So we are looking at a probability of the same incidence being recorded with coordinates with up to 300 meters distance.
• time might be recorded different. Most probably the same day, but this can also be fuzzy. Up to 2 days difference. Where one institution records the incidence through field officers (police, rangers, etc), the other could record this by asking for week number and then set an arbitrary date within that week.

Thus we are looking at a two dimensional distance calculation consisting of time and physical distance. Duch closeness could be calculated based on relative closeness (establishing a distance list for each object related to all other objects for this attribute, or an absolute distance filter (categorizing distances).

I guess the way to do this is to run subsequent matching according to dates and coordinates. I have been looking at the different libraries in FME 2015.0 but have not found any which can use coordinate closeness and time closeness as filters.

The hard way to find neighboring coordinates is to pull the coordinates out as numbers and then do a comparison. It will of course mean I have to populate an array and work on that array. I could then export it and use a small python script to do the job. I was hoping to avoid that.

Does anyone have examples of such duplicate procedures and their parameters/constraints using FME?

• I don't have FME and have only used it very sparingly a long time ago, but 'coordinate closeness' is often analysed through the use of buffers so this could be the approach to take for that. Buffer a point, then check whether other points are within the buffer to determine whether they are close or not. As for the time issue, you should do some data cleansing prior to analysis and attempt to standardise the way that time data is stored. Commented Apr 21, 2015 at 13:04
• A NeighborFinder transformer is probably the way to go with this. It can store all neighbours within 300m in a list.
– Fezter
Commented Apr 21, 2015 at 23:01
• DuplicateRemover or Matcher Transformer docs.safe.com/fme/html/FME_Transformers/… docs.safe.com/fme/html/FME_Transformers/…
– Mapperz
Commented Apr 22, 2015 at 2:38
• @Mapperz DuplicateRemover and MatcherTransformer do not cut it. They do not allow for the proper fuzzy tolerance criteria. The transformer FuzzyDuplicateRemover does not support coordinates (directly) or dates. I guess if I pull out the coordinates as numbers I could make a transformer by doing comparisons. But it will necessitate iterating through the whole data set back and forth. As for the dates... not so sure. Commented Apr 22, 2015 at 4:44
• FME is about chaining and formatting your data to get the desired result - the timestamper will format ALL your dates/time into one standard. DuplicateRemover will work with fuzzy points ported out of the NeighbourFinder.
– Mapperz
Commented Apr 22, 2015 at 13:34

FME does not provide the tools to do this with the available transformers. This solution describes how the coordinate check can be done using python (PythonCaller). It is based on an answer to a question at stackoverflow.

This is the proposed procedure which can be programmed in Python:

1. Coordinates are made an attribute
2. A list of candidate coordinates is established and written to disk
3. All object coordinates are compared to the candidates using the following python procedure called from PythonCaller (code draft):

candList= Pulled from text file on disk

coordinate = get from feature (feature.getAttribute)

nearest = min(candList, key=lambda x: distance(x, coordinate))

4. Where distance `distance(a, b)` is a function which returns the distance between coordinates a and b. If the nearest coordinate pair is within 300 meters it is considered a coordinate duplicate.

5. An attribute with pair numbers are added.
6. The pairs are considered for date duplication/closeness (to be described).
7. Final candidates are then merged.

Other platforms could be considered. The model builder in QGIS is flexible and could represent a useful platform. GeoKettle could also be considered.