2

I have an FME process that is duplicating some values in an attribute. When I try to use regex in FME to clean it up it is deleting the separator (see screengrab below). Any ideas on how I can change the regex or process in FME to just get the required values.

So from

164511|119958|164512|164511|119958|164512|164511|119958|164512
161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636|161634|161635|161636
1114|1114|1114|1114|1114

I just want

164511|119958|164512
161634|161635|161636
1114

The value in AttributeManager from FME is @ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|",\1)

I have tried to replace \b),?(? with \b)\|(?to use the | seperator but that doesn't work either. This is based on https://superuser.com/a/1343524/639219

The regex tester is https://regex101.com/r/1YhC59/1

enter image description here

2

You are almost there, instead of

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|",\1)

you need

@ReplaceRegEx(@Value(Name_Ids),"(?:^|\G)(\b\w+\b),?(?=.*\1)\|","")
2
  • Thanks but for my future knowledge can you let me know what the ` \1` at the end does as against ""? That's the only difference I noticed. – GeorgeC Aug 26 '20 at 22:54
  • 1
    I don’t know, fiddled around and it just worked :) – nielsgerrits Aug 26 '20 at 23:08
1

You could use a PythonCaller as well to remove your duplicate values:

def processFeature(feature):
    ids = feature.getAttribute('Name_Ids')
    feature.setAttribute('Name_Ids', '|'.join(list(dict.fromkeys(ids.split('|')))))

enter image description here

1

Without regex you could always use:

  • An AttributeSplitter transformer to turn the attribute into a list
  • A ListDuplicateRemover transformer to remove duplicate list entries
  • A ListConcatenator transformer to turn the list back into an attribute

But also, could we stop the duplication happening from the very start? Can you clarify how that string is being created?

2
  • Wow, 3 transformers for a simple task ;-) ... would be nice if Attribute Manager could support Python! – christoph Aug 25 '20 at 15:58
  • 1
    Or maybe we could ask Don and Dale for an expression engine with custom Python functions like QGIS offers us... only thinking loud. – christoph Aug 25 '20 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.