Remove duplicate values in an attribute using regex and fme

Question

I have an FME process that is duplicating some values in an attribute. When I try to use regex in FME to clean it up it is deleting the separator (see screengrab below). Any ideas on how I can change the regex or process in FME to just get the required values.

So from

164511|119958|164512|164511|119958|164512|164511|119958|164512
161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636
1114|1114|1114|1114|1114

I just want

164511|119958|164512
161634|161635|161636
1114

The value in AttributeManager from FME is @ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|",\1)

I have tried to replace \b),?(? with \b)\|(?to use the | seperator but that doesn't work either. This is based on https://superuser.com/a/1343524/639219

The regex tester is https://regex101.com/r/1YhC59/1

Answer 1

You are almost there, instead of

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|",\1)

you need

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|","")

Answer 2

You could use a PythonCaller as well to remove your duplicate values:

def processFeature(feature):
    ids = feature.getAttribute('Name_Ids')
    feature.setAttribute('Name_Ids', '|'.join(list(dict.fromkeys(ids.split('|')))))

Answer 3

Without regex you could always use:

• An AttributeSplitter transformer to turn the attribute into a list
• A ListDuplicateRemover transformer to remove duplicate list entries
• A ListConcatenator transformer to turn the list back into an attribute

But also, could we stop the duplication happening from the very start? Can you clarify how that string is being created?