There are a number of possibilities here, including just making what you have more efficient. And you don't mention what the error is, so it's hard to tell what is happening, and where. Let's assume it's an overload of data for the available resources.
Firstly, there is your setup. You could use FME 64-bit, if you aren't already. That would give you access to greater system resources than the 32-bit version.
What temporary folder are you using? It should say in the FME log. Is there a chance it could be running out of space? It's worth checking.
Secondly, you could check the efficiency of your current workspace. For example, if you have a group-by set in the Dissolver, does the data arrive in group order? If so, use the Group By Mode parameter (set to Process When Group Changes) to gain a performance boost.
How are you intersecting the Oracle and Shape polygons? Both the AreaOnAreaOverlayer and Clipper transformers can do that, and one might be faster than the other. The Clipper also gives the Clippers First mode, which will definitely help performance.
As for the Shapefile writer, I don't think there is a lot you can do there, although you might want to consider whether it is capable of storing 11 million polygon records (depends on how many vertices I would think).
So, finally, to process in batch is possible, but that's difficult because you want to ensure that adjacent features are sent to the same batch (otherwise they won't get Dissolved).
What I'd try is this:
- Add a Tiler transformer to split your data up into tiles before it gets to the Dissolver. Create - say - a 2x2 grid of tiles so your data is split into four.
- Select the Dissolver transformer and press Ctrl+T to turn it into a "Custom Transformer". In the custom transformer definition (click on the new tab) find the parameter for Parallel Processing in the Navigator window and turn it on (say Moderate processing).
- Back in the main tab, under the parameters of the custom transformer, select _row and _column as the group by.
This way your data will be split into four parts, each of which is a separate (parallel) process.
Here's a very quick video demo: https://www.screencast.com/t/RHNy26FgQD
But overall, if you could post a screenshot of your workspace to the question, it would really help.
As you can see, lines buffers in workspace, not in oracle. The attribute to Group_by in the dissolver is inherited from oracle, and all the lines have same value on this attribute. I want the buffer as clean as possible, without overlaps, but I don't know if this is the best way to do that. If I only have a a single value for the exposed attribute, can I execute Parallel Processing? Or should I create artificial grouping to be able to run the dissolver through parallel processing?
