8

I've got an FME Workbench (version 2015.0) which searches for shapefiles in a folder tree. The shapefiles are contours which are split into tiles for a particular area. The workbench is fairly simple, it filters based on certain attributes then combines the output into a merged shapefile containing all the contours.

However, the workbench fails because the resulting shapefile is too big. I get the following error:

Error - Cannot write out a file larger than 4.295GB. Split your data into two or more shape files

Of course, I didn't necessarily know beforehand that my output would be so large. Is there any way for FME to detect that the file will be too big and somehow automatically write the output to two or more shapefiles? Perhaps with the suffix "_1.shp", "_2.shp"?

Improve this question
2
  • 2
    not that i know , but fast & dirty way to fix problem is just take modulo count (5) from all feature and fanout using it. its not optimal solution but works , using 5 raises limit to ~20G data. Another option , use workspace runne ronto workspace which allways fanouts all data to x filse then in workspace runner check output directory and merge shapefiles to ~4G max size files.
    – simpleuser001
    Jan 27, 2015 at 9:49
  • You can clip your data with a vector grid and have each one write out as _1_0_1.shp etc.
    – Mapperz ♦
    Jan 27, 2015 at 15:04

2 Answers 2

Reset to default
4

A good guide on how to approximate when you are reaching a ShapeFile limit is mentioned in this documentation by Esri

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002t0000000m000000

"There is a 2 GB size limit for any shapefile component file, which translates to a maximum of roughly 70 million point features. The actual number of line or polygon features you can store in a shapefile depends on the number of vertices in each line or polygon (a vertex is equivalent to a point)."

Now in FME you could count the vertices but that would really slow down your process, and is not advisable.

Preferred way would be fanout by some attribute, or limit to a number of features per shapefile.

Improve this answer
3

No, there's no functionality that detects this and splits the file up for you. It's already registered as an enhancement request with our developers, and the reference number is PR#34128. I added a note linked to this question.

NB: Technically I believe the limit specified by the format is 2.14gb, so even 4.29 is stretching the format more than it should really be.

Of course, the question is whether a Shape dataset of that size is a good idea in the first place! Maybe you could use a different format?

If you are stuck on Shape, then I'd do as others have suggested. The ModuloCounter transformer is a good way to divide data (on a non-geographic basis) or you could split the data with either the Tiler transformer or the Clipper.

Improve this answer
1
  • Thanks Mark. Yes, I've decided to write to another format.
    – Fezter ♦
    Jan 27, 2015 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.