Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have a large FME FMW file which is using a lot of memory at runtime. I have been tasked with trying to find ways to reduce the memory footprint.

Early on in the workbench there is (what I assume is) a group-based transformer - LineOnAreaOverlayer, which looks for lines that intersect polygons. I suspect that the group-based nature of this transformer is pushing up memory usage, as the thousands of features which are fed into it need to be held in memory and processed all in one go (at least that's the theory). My aim is to halt processing when features are first output from this transformer to see how much memory is being used at this point. I'm using inspection points on the transformer's outputs.

However, it seems I either don't understand group-based transformers or inspection points. I break processing on the first feature that is output from the transformer, and the connector feature counts in the rest of the FMW all show features having already gone right through the transformation and being written. I thought that all input features were backed-up in front of the LineOnAreaOverlayer before anything else happened?

Can anyone help me clarify this please? Also, any other useful information on tracking memory usage in an FMW would be much appreciated - right now my only options are the log's peak memory usage numbers or trying to match a procmon log with the fme log, which will be a tedious task.

share|improve this question
    
You might cross-post on the FMETalk user group for a better chance of an answer: groups.google.com/group/fmetalk –  blah238 Oct 7 '11 at 18:21
    
What version of FME and what Operating System (+RAM) do you have? –  Mapperz Oct 7 '11 at 18:54
    
@blah238 I'm working on it, thanks –  tomfumb Oct 7 '11 at 19:33
    
@Mapperz I should have mentioned. FME 2011, Windows 7 Pro, 4 GB ram (but 32-bit system) –  tomfumb Oct 7 '11 at 19:34
1  
Ultimately the specific issue I was having was FMW-specific and due to the position of testers and multiple routes through the transformation, some of which did not involve the overlayer I was trying to break on. The discussion got slightly further at groups.google.com/group/fmetalk/browse_thread/thread/… –  tomfumb Oct 11 '11 at 19:08
add comment

1 Answer 1

The short answer is, don't use group-based transformers with large datasets. Due to their nature, any group based transformer acts as a FeatureHolder, which simply means that everything sent to it has to be stopped and held in RAM while it waits for everything to be read.

FMEPedia has a little information about Group-Based Transformers; of Memory Concerns it says:

A feature of group-based transformers is that they go against the general rule of FME, that features are processed one at a time, and require that a number of features be held in memory at one time. Because of this such transformers are more prone to consume large amounts of system resources (some such as the Clipper have extra options to help mitigate this effect) but can be used creatively to handle the flow of features through a workspace.

Without knowing specifics of your workspace I can only offer generic suggestions. But you may want to:

  • Don't use group-based transformers if at all possible and you have a large dataset.
  • Only send features that absolutely have to go through a group-based Transformer to said transformer.
  • SQLCreator is group-based, so if you're getting even hundreds of thousands of features with it, that will take lots of RAM. Try to use the appropriate FME Reader if possible.
  • Split up the workload. Some clever use of WorkspaceRunner (being sure to set "Wait for job to complete: Yes") could reduce memory use by only loading part of the dataset at a single time. Not all datasets can be split up and still give the desired output.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.