FME workflow not outputting tab file correct

Question

My FME workflow, combines property data sourced from three different datasets. It must start and end with the same number of features, so I have in place a statisticsCalculator at the start and finish of the workflow, added a tester to ensure they are the same, and added another tester to ensure the count is above 24,000 (which I know the output will never be less).

Now very sporadically neither of these safety nets work and the layer is still produced. Usually we can fix the issue in 10min the next day but its frustrating I can't figure this out or why it happens sporadically, and its hard to test any improvements.

I have reviewed the log file, and get these errors when I have an issue. The file gets overwritten every night to catch any updates to databases so there is adequate disk space, the file isn't in use by another application or workflow, and the errors regarding attributes - the file is written fine majority of the time.

Error 1

2016-07-24 19:27:45| 92.1| 0.0|ERROR |MapInfo file 'C:\etc' could not be created. Check that file path name is valid and there is adequate disk space. Also ensure that the total record length for all the attributes is less than 4000 bytes and number of attributes are less than 250

Error 2

MapInfo file 'C:\etc' could not be created. Check that file path name is valid and there is adequate disk space.

Error 3

Also ensure that the total record length for all the attributes is less than 4000 bytes

With the implemented precautionary steps not capable of capturing the failure, is there another way to add a safety net actually into the writer to ensure that it is not written if say it doesn't have more than 24,000. The image below shows the workflow.

Answer 1

OK, I've been puzzling over this one all day. It sounds like a simple writer issue, but you have the number of features thrown in.

One thing I would check is whether the files are being truly overwritten. It's hard to explain why, but I'd be tempted to add a startup script that wipes the output folder before translation. That way we can be sure we're starting with a blank slate.

But to try and debug this problem I would try this...

Add a Recorder transformer between the StatisticsCalculator and Tester_10. That will record the data in FME's "FFS" format, as it appeared at that point.

If the problem occurs, copy your workspace and in the copy wipe everything except the two Testers. Add a Player transformer to read back to data and pass it through the Testers.

That way we see whether the Testers are messing up.

If not, remove the Testers and add the FeatureWriter transformer back. Run the data through there to see if it will write correctly.

I suspect it won't. I suspect a feature of some form is causing a problem to the writer. If so, change the Player "Max Features to Read" to 1000. If that works, change it to 2000 and try again.

Keep at it narrowing it down until you find the offending feature. Or if the first feature causes an error, then the schema itself has a problem.

Or you could send the FFS file and workspace to our support team (or myself) and we can try that for you. See if we can figure out what the error is about.