1

Every week, I download a CSV table from an FTP server, which contains a variable number of rows ranging from 700,000 to 800,000. This CSV table serves as the foundation for subsequent data processing tasks.

What I need is a mechanism to detect if either of two specific columns, namely "avg" and "stddev," contains only "NULL" values across all rows, or if any other column contains entirely "NULL" values in all rows. When such a condition is met, I want to be notified, and I also want the further data processing to halt.

I've attempted to achieve this using FME Desktop and FME Server, experimenting with transformers like "Tester," "TestFilter," and "AttributeValidator," but so far, I haven't been successful in implementing the desired functionality.

I'm seeking guidance and assistance on how to effectively set up this data validation and notification process using FME Desktop and FME Server.

1 Answer 1

1

For the logic part, there's a transformer on the FME Hub called AdvancedSorter. It lets you include nulls in your sorting. So set it up like this:

enter image description here

The point is to sort Nulls after any data. Then use a Sampler transformer to get the first feature. Test if the avg value for the first feature is <null>. If so, then you know the rest of the values will be too.

After that, you can send a notification in many ways. There's the Emailer transformer, but on Server (now Flow) you can trigger a notification or automation process in other ways too.

1
  • Thank you for you response. I will try your advise and let you know
    – Ram
    Sep 13, 2023 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.