6

I am using FME to write all the MapInfo TAB files in a directory and it's subdirectories to a File Geodatabase. I'm using dynamic readers and dynamic writers as I don't know the geometry or schema of the input files beforehand. So far so good. Now, suppose that I have a file called "Pipeline.TAB" at the root of the directory and I have another file called "Pipeline.TAB" in a subdirectory. No problem. FME can merge the two files together and I get the geometry from both files in my new feature class called "Pipline_line".

My issue is, suppose those two original files have different attributes. Say, the file in the root folder has a field called "Name" and the second file has a field called "Pipe_Name". My resulting feature class gets the "Name" field but not the "Pipe_Name" field. The features that came from the file in the subdirectory have a NULL value in the "Name" field. Instead, what I would like is to have both fields come through. I'm happy if the "Name" field is NULL as long as the "Pipe_Name" field is there and has the correct values.

It appears that the schema is determined by the first feature that is being read, whereas, I need it to use the schema from all of the features with the same name. Is this possible?

EDIT

I have posted this question on FMEPedia as well. Here is some additional information which may help:

The example I gave was simple in that it only had two files called Pipeline.TAB. In reality, I'm looking at dozens of folders with thousands of files. there might be 20 files called "Pipeline" and another twenty called "Valves", etc. I won't know the schema of all of them at the start.

What I would like the final attribute table of my destination feature class to look like would be something like this:

Name       | Pipe_Name
Pipe1      | NULL
Pipe2      | NULL
Pipe3      | NULL
NULL       | Pipe 4
NULL       | Pipe 5

In the final file, there may actually be many fields. But I won't know all of them at the start:

Name      | Pipe_Name  | Diameter | Material | Type
Pipe1     | NULL       | 500      | Steel    | Gas
Pipe2     | NULL       | 1200     | Steel    | Gas
Pipe3     | NULL       | 800      | PVC      | Water
NULL      | Pipe 4     | NULL     |NULL      | Water
NULL      | Pipe 5     | NULL     |NULL      | Water

In response to @BradNesom's answer, I have looked at the SchemaMapper, but I have way too many files to look at for that to be feasible. I also have looked at the other link you mentioned and I tried adding a reader as a resource but that didn't seem to work either.

  • i have done something similiar using bulk/attributerenamer, using rexexp to rename similiar fields to one and then dropping old attributes. – simplexio Jan 9 '14 at 8:19
  • the new fme has an option to use a folder to bring in more than one file at a time. I don't know if this is a bug or as designed. But I worked with safe support and it would only bring in the attributes of the first file and then I got some twerky results after changing numbers and text fields to 000000000.000 values. My workaround was to add a separate reader and feature type for each file – Brad Nesom Feb 2 '15 at 22:53
1

Yes it is possible.
How would be the rest of your question.
Using the schema mapper transformer is a great direction but if you have "lots" of different files with no good way of cataloging all the variety.
In this fmepedia article there are multiple examples.
I like the attributerenamer method.
But there is another that allows the schema to be read at run time.
Specifically, the Dynamic Schema functionality provides two options for determining the destination schema information at run-time:
from a reader already in the workspace (Source Reader),
from a reader added as a Workspace Resource (Workspace Resource Reader).
EDIT:
I just found the collectors section on fme store.

featuremerger

UnconditionalFeatureMerger

This transformer merges sets of attributes onto all REQUESTOR features. Essentially it is an unconditional merge of feature attributes.
The user can choose between two modes:
One Supplier, or All Suppliers.

In One Supplier mode,
All REQUESTORS will receive a copy of the attributes of the first SUPPLIER feature,
and emerge from the COMPLETE output port.
The first SUPPLIER emerges from the REFERENCED output port; any others through the
UNREFERENCED port.

In All Suppliers mode,
All REQUESTORS will receive a copy of the attributes of all SUPPLIER features, and emerge from the COMPLETE output port. All SUPPLIER features emerge from the REFERENCED output port.

  • Thanks Brad, have a look at my edits above. SchemaMapper won't be feasible for me, and I have tried the other options too. Still no joy. – Fezter Jan 9 '14 at 4:45
0

I don't think what you're hoping for is possible. You will always need to have some special cases. Here's the two ways I suggest you do it:

  1. Don't use generic read/write and create all your input feature classes as distinct readers, and select them all and choose 'duplicate on writer' to set up the writer classes. This way you can modify each one individually and combine them together explicitly where you want to.
  2. Use the generic read/write but test for special cases. You can test for fme_dataset (or whatever one has the full path in it) and have a different set of rules there for the second Pipeline file. In this one, rename attributes to what you want them all to go in. Or use a special case for both and create attributes from both of the feature types on features from both.

Finally, how many duplicate names do you actually have? If you only have a dozen, why not run the first script to import everything, and then have a second that handles the special cases.

Sometimes you can't automate everything!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.