2

I have two heavy-weight polygon feature classes that I want to union and then calculate some area-weighted average of one attribute for each aggregate (say parcel polygons, which is one of the inputs) by using FME (2018.1) on my Windows 7 machine. I developed a model with the sample data and it works fine (see below). However, when I use the official datasets, the workbench yields

"Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds."

error. As suggested I tried \3GB switch without any success (I have 32GB total RAM) and using 64bit FME is not an option at the moment.

enter image description here

If I was running the same process in ArcGIS, I would be able to divide my inputs into tiles and process them separately and then merge them to get my result feature class without any dramas. But in FME, I cannot do this since FME caches all inputs at once and pipe it to next process if it does not "hold" the overall model run.

I thought the looping as suggested elsewhere may help but still cannot fan-out the outputs from each loop. Any ideas?

UPDATE

The workbench works fine in 64-bit FME 2018.1.

  • Can you add operating system (sounds like windows) and the version of FME please. I have seen this issue on older versions. There can be some tweaking to make it work. – Mapperz Jun 10 at 13:37
  • @Mapperz, I have updated the question. – fatih_dur Jun 11 at 8:48
2

I don't think a loop solution is going to help here. It just doesn't break it up into multiple processes as you would require. But you certainly can break your data into tiles and process it one tile at a time (or several tiles simultaneously).

One way to implement a tiling solution would be to use two workspaces. One workspace breaks the data into tiles, the second is your workspace in the screenshot. The first workspace initiates the second using the WorkspaceRunner transformer. So you'd have something like this:

Workspace 1:

  1. Readers (to read data)
  2. A Tiler (or similar, to tile the data)
  3. A FeatureWriter (to write the tiled data to a file)
  4. A WorkspaceRunner (to initiate Workspace2, passing the name of one tile file)

Workspace 2:

  1. As above, but reading a tile of data, not the whole datasets.

Another tiling method would be to use Parallel Processing. In 2019 you'd need to use a Custom Transformer, but that's OK. It would actually be quite useful as you parallelize every transformer in the workspace. Basically put your workspace contents into a custom transformer and use the Parallel Process option (which could either be by a tile ID or another method of dividing the data).

That's one option. Another is to analyze your workspace in a bit more detail to see if there are any savings that can be made. To be honest, it's difficult to tell from the screenshot. One thing I do wonder about is the FeatureJoiner and whether it's needed. I assume it's joining the original attributes onto the processed features. But could the AreaOnAreaOverlayer not be set up in such a way as to keep those attributes? Like use the Merge Incoming Attributes option? Then you could avoid the FeatureJoiner.

In general memory gets chewed up when you're using a "group-based" transformer. In your workspace the AreaOnAreaOverlayer, the FeatureJoiner, and the Aggregator are all group-based. To reduce memory usage you need to avoid using these and/or to use options in them that reduce memory use. For example, if there is a group-by set then you can tell FME to process more efficiently when data is pre-sorted. So take a look for those.

I hope this helps. Other general advice is to avoid using lists, remove unnecessary attributes (use an AttributeManager), and put filter transformers as early as possible (basically get rid of excess data as soon as possible in the workspace).

I have also seen this error occur on a bad piece of data (instead of rejecting the data FME just crashes with this message). That was a very long time ago, and I don't know if you'd get the same issue today, but it's worth just monitoring memory use with a Process Monitor, just to confirm that it is running out of memory, not just throwing that error for a completely different reason.

  • Thanks Mark, enlightening as always. Here are my comments: I followed your advice calculating necessary attributes beforehand to avoid extra group-based transformer (Feature Joiner in my case). In fact it was quite obvious that I just missed. I have been using Merge Incoming Attributes from the outset. However my workbench cannot go any further than AreaOnAreaOverlay transformer. I followed up the memory consumption and at Setting up lines for phase# 2, the consumption leaps in big steps and reaches to 2.92GB ,then fails That implies it is not the bad piece of data problem. – fatih_dur Jun 11 at 8:26
  • I asked one of my colleagues who has 64-bit FME 2018 and it works just fine. I am carrying over only the lean part of my attributes. I used Parallel Processing option in the AreaOnAreaOverlay transformer but it did not help either. Yet, I could not manage to apply your two workbenches approach. When I use Tiler, it writes all tiles with _row and _column attributes. Are you suggesting placing a tester before WorkspaceRunner to pick only relevant tile part and apply union (Workspace 2)? Say if I have four tiles, should I run this configuration four times? – fatih_dur Jun 11 at 8:33
  • I greatly appreciate if you can give me a link to an example use. – fatih_dur Jun 11 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.