1

My FME workflow, combines property data sourced from three different datasets. It must start and end with the same number of features, so I have in place a statisticsCalculator at the start and finish of the workflow, added a tester to ensure they are the same, and added another tester to ensure the count is above 24,000 (which I know the output will never be less).

Now very sporadically neither of these safety nets work and the layer is still produced. Usually we can fix the issue in 10min the next day but its frustrating I can't figure this out or why it happens sporadically, and its hard to test any improvements.

I have reviewed the log file, and get these errors when I have an issue. The file gets overwritten every night to catch any updates to databases so there is adequate disk space, the file isn't in use by another application or workflow, and the errors regarding attributes - the file is written fine majority of the time.

Error 1

2016-07-24 19:27:45| 92.1| 0.0|ERROR |MapInfo file 'C:\etc' could not be created. Check that file path name is valid and there is adequate disk space. Also ensure that the total record length for all the attributes is less than 4000 bytes and number of attributes are less than 250

Error 2

MapInfo file 'C:\etc' could not be created. Check that file path name is valid and there is adequate disk space.

Error 3

Also ensure that the total record length for all the attributes is less than 4000 bytes

With the implemented precautionary steps not capable of capturing the failure, is there another way to add a safety net actually into the writer to ensure that it is not written if say it doesn't have more than 24,000. The image below shows the workflow.

enter image description here

  • What version on FME with build number? – Mapperz Aug 17 '16 at 2:09
  • FME 2016.1.1.0 (20160722 - Build 16609 - WIN64) – Pat Aug 17 '16 at 2:12
  • Check the FME_TEMP location on some windows machines it can be a restricted location knowledge.safe.com/articles/176/… – Mapperz Aug 17 '16 at 2:12
  • So, can I just clarify the issue here? You have StatisticsCalculators. Presumably they calculate the Total Count of features at the start and end. A Tester follows the second one. Presumably it tests where EndCount < 24,000 AND EndCount != StartCount (ie the number of features are different). Sometimes features pass this test, even though they shouldn't. At the times this happens, strange messages appear in the log. Is that correct? If so, it would be helpful to see the parameters dialog for the Tester. After that, I could make some debug suggestions for next time it happens. – Mark Ireland Aug 17 '16 at 15:36
  • My first thought is the Tester is set to test "AND" when it should be "OR". ie something in the logic is causing the problem under special circumstances. – Mark Ireland Aug 17 '16 at 15:39
1

OK, I've been puzzling over this one all day. It sounds like a simple writer issue, but you have the number of features thrown in.

One thing I would check is whether the files are being truly overwritten. It's hard to explain why, but I'd be tempted to add a startup script that wipes the output folder before translation. That way we can be sure we're starting with a blank slate.

But to try and debug this problem I would try this...

Add a Recorder transformer between the StatisticsCalculator and Tester_10. That will record the data in FME's "FFS" format, as it appeared at that point.

If the problem occurs, copy your workspace and in the copy wipe everything except the two Testers. Add a Player transformer to read back to data and pass it through the Testers.

That way we see whether the Testers are messing up.

If not, remove the Testers and add the FeatureWriter transformer back. Run the data through there to see if it will write correctly.

I suspect it won't. I suspect a feature of some form is causing a problem to the writer. If so, change the Player "Max Features to Read" to 1000. If that works, change it to 2000 and try again.

Keep at it narrowing it down until you find the offending feature. Or if the first feature causes an error, then the schema itself has a problem.

Or you could send the FFS file and workspace to our support team (or myself) and we can try that for you. See if we can figure out what the error is about.

  • Also, if you can share your full log file here (or with me privately) it might help me to diagnose what is happening. – Mark Ireland Aug 18 '16 at 21:35
  • I have implemented your recorder transformer recommendation to try and catch this issue. I am yet to add the startup script to remove the file at the beginning, but will look at this next week. Its an error that occurs randomly, so i'll wait until it occurs again. Last couple of times it has finished with an error has occurred within two or three week period with the workflow running 5 times a week. When it does fail, and if I cant figure it out, I might take you up on the offer and send you the files. I will keep you posted with the outcomes. – Pat Aug 19 '16 at 5:46
  • Safe (FME) has knowledge base knowledge.safe.com/questions/index.html which might be a better location to 'debug' fme workspaces. – Mapperz Aug 22 '16 at 22:15
  • 1
    @MarkIreland I have been running this workflow for a month now and have not had the issue come back. I did however implement a Stop / Start server script to ensure no files were active which may of helped too. Thanks for your assistance – Pat Sep 22 '16 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.