I have a table which has over 100 columns, of which many columns have no values (or missing values) for every line. In FME, I would like to remove the entire column from the table in the case that no values are found in said column.
Example: | Column 1 | Column 2 | Column 3| | -------- | -------------- |-------------- | | First entry | missing value | First entry | | Second entry | missing value | Second entry |
In this case, I would like to remove column 2, but keep column 1 and 3.
So, there are multiple places you might want to remove the attribute.
If you want to remove it from the workspace itself, then you need to do it manually, say with an AttributeManager. You can't do it automatically, or when run, because FME won't update the workspace.
So to hide it in the workspace above is a manual task.
If you want to remove the attribute from the data itself, that's fairly easy. In fact, if the value is already set to <Missing>, then FME will consider it gone. If it's another value, you can use a NullAttributeMapper transformer to handle that:
So if the value is literally " missing value " - as it is in your data above - then this transformer turns it into a true, FME-style, missing value.
You'll still see it in the Visual Preview table (labelled as <Missing>) but not in the Feature Information window:
But, of course, that's not particularly useful if you want it removed from any output data as well. That takes a little more effort.
Firstly we add a SchemaScanner transformer (I've edited bits out of the image for clarity):
It's basically creating the schema (data structure) we want to write. I've said to ignore attributes beginning with csv_ or fme_ or multi_. I've also said to ignore any empty (missing) attributes.
Now I want to write the data; in my case, I'm going back to CSV:
Notice both SchemaScanner outputs are connected and that I've told it to get the schema from the schema feature (and nowhere else).
Under the User Attributes tab, it's set to Dynamic and I removed ALL of the attribute definitions:
Run the workspace, and this is what I get:
Column 1|Column 3
First entry | First entry
Second entry | Second entry
Hopefully, that's not too confusing. Basically, it depends on where you want to remove the attributes. If it's from the output data, then you need to use a Dynamic schema.
I answered a similar question during a live stream. You can find the recording here. The difference is that the user there wanted to fan out their data into multiple outputs, which is a complication that I don't think you need. But it follows the same dropping of empty fields.
