0

I'm new to FME. I have a process that intersects a line buffer from Oracle with a shp stored in local. In the end there is a dissolver to reduce the number of polygons and write it in a shp.

The problem is that from Oracle come 11 million records ... and in the dissolver I get an error.

My question would be if there would be a way to execute the process in the following way: that I read 500,000 records, process them, write them in the shp, then another 500,000, then others and so on until the end. It could be done?


This is an image of my workspace.workspace As you can see, lines buffers in workspace, not in oracle. The attribute to Group_by in the dissolver is inherited from oracle, and all the lines have same value on this attribute. I want the buffer as clean as possible, without overlaps, but I don't know if this is the best way to do that. If I only have a a single value for the exposed attribute, can I execute Parallel Processing? Or should I create artificial grouping to be able to run the dissolver through parallel processing?

To get the intersection I use Clipper and works fine.

1

There are a number of possibilities here, including just making what you have more efficient. And you don't mention what the error is, so it's hard to tell what is happening, and where. Let's assume it's an overload of data for the available resources.

Firstly, there is your setup. You could use FME 64-bit, if you aren't already. That would give you access to greater system resources than the 32-bit version.

What temporary folder are you using? It should say in the FME log. Is there a chance it could be running out of space? It's worth checking.

Secondly, you could check the efficiency of your current workspace. For example, if you have a group-by set in the Dissolver, does the data arrive in group order? If so, use the Group By Mode parameter (set to Process When Group Changes) to gain a performance boost.

How are you intersecting the Oracle and Shape polygons? Both the AreaOnAreaOverlayer and Clipper transformers can do that, and one might be faster than the other. The Clipper also gives the Clippers First mode, which will definitely help performance.

As for the Shapefile writer, I don't think there is a lot you can do there, although you might want to consider whether it is capable of storing 11 million polygon records (depends on how many vertices I would think).

So, finally, to process in batch is possible, but that's difficult because you want to ensure that adjacent features are sent to the same batch (otherwise they won't get Dissolved).

What I'd try is this:

  • Add a Tiler transformer to split your data up into tiles before it gets to the Dissolver. Create - say - a 2x2 grid of tiles so your data is split into four.
  • Select the Dissolver transformer and press Ctrl+T to turn it into a "Custom Transformer". In the custom transformer definition (click on the new tab) find the parameter for Parallel Processing in the Navigator window and turn it on (say Moderate processing).
  • Back in the main tab, under the parameters of the custom transformer, select _row and _column as the group by.

This way your data will be split into four parts, each of which is a separate (parallel) process.

Here's a very quick video demo: https://www.screencast.com/t/RHNy26FgQD

But overall, if you could post a screenshot of your workspace to the question, it would really help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.