Dividing/tiling input features in FME, processing independently and fanning out the output by tile

Question

I have two heavy-weight polygon feature classes that I want to union and then calculate some area-weighted average of one attribute for each aggregate (say parcel polygons, which is one of the inputs) by using FME (2018.1) on my Windows 7 machine. I developed a model with the sample data and it works fine (see below). However, when I use the official datasets, the workbench yields

"Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds."

error. As suggested I tried \3GB switch without any success (I have 32GB total RAM) and using 64bit FME is not an option at the moment.

If I was running the same process in ArcGIS, I would be able to divide my inputs into tiles and process them separately and then merge them to get my result feature class without any dramas. But in FME, I cannot do this since FME caches all inputs at once and pipe it to next process if it does not "hold" the overall model run.

I thought the looping as suggested elsewhere may help but still cannot fan-out the outputs from each loop. Any ideas?

UPDATE

The workbench works fine in 64-bit FME 2018.1.

Answer 1

I don't think a loop solution is going to help here. It just doesn't break it up into multiple processes as you would require. But you certainly can break your data into tiles and process it one tile at a time (or several tiles simultaneously).

One way to implement a tiling solution would be to use two workspaces. One workspace breaks the data into tiles, the second is your workspace in the screenshot. The first workspace initiates the second using the WorkspaceRunner transformer. So you'd have something like this:

Workspace 1:

1. Readers (to read data)
2. A Tiler (or similar, to tile the data)
3. A FeatureWriter (to write the tiled data to a file)
4. A WorkspaceRunner (to initiate Workspace2, passing the name of one tile file)

Workspace 2:

1. As above, but reading a tile of data, not the whole datasets.

Another tiling method would be to use Parallel Processing. In 2019 you'd need to use a Custom Transformer, but that's OK. It would actually be quite useful as you parallelize every transformer in the workspace. Basically put your workspace contents into a custom transformer and use the Parallel Process option (which could either be by a tile ID or another method of dividing the data).

That's one option. Another is to analyze your workspace in a bit more detail to see if there are any savings that can be made. To be honest, it's difficult to tell from the screenshot. One thing I do wonder about is the FeatureJoiner and whether it's needed. I assume it's joining the original attributes onto the processed features. But could the AreaOnAreaOverlayer not be set up in such a way as to keep those attributes? Like use the Merge Incoming Attributes option? Then you could avoid the FeatureJoiner.

In general memory gets chewed up when you're using a "group-based" transformer. In your workspace the AreaOnAreaOverlayer, the FeatureJoiner, and the Aggregator are all group-based. To reduce memory usage you need to avoid using these and/or to use options in them that reduce memory use. For example, if there is a group-by set then you can tell FME to process more efficiently when data is pre-sorted. So take a look for those.

I hope this helps. Other general advice is to avoid using lists, remove unnecessary attributes (use an AttributeManager), and put filter transformers as early as possible (basically get rid of excess data as soon as possible in the workspace).

I have also seen this error occur on a bad piece of data (instead of rejecting the data FME just crashes with this message). That was a very long time ago, and I don't know if you'd get the same issue today, but it's worth just monitoring memory use with a Process Monitor, just to confirm that it is running out of memory, not just throwing that error for a completely different reason.