7

I've got an FME Workbench (version 2015.0) which searches for shapefiles in a folder tree. The shapefiles are contours which are split into tiles for a particular area. The workbench is fairly simple, it filters based on certain attributes then combines the output into a merged shapefile containing all the contours.

However, the workbench fails because the resulting shapefile is too big. I get the following error:

Error - Cannot write out a file larger than 4.295GB. Split your data into two or more shape files

Of course, I didn't necessarily know beforehand that my output would be so large. Is there any way for FME to detect that the file will be too big and somehow automatically write the output to two or more shapefiles? Perhaps with the suffix "_1.shp", "_2.shp"?

  • 2
    not that i know , but fast & dirty way to fix problem is just take modulo count (5) from all feature and fanout using it. its not optimal solution but works , using 5 raises limit to ~20G data. Another option , use workspace runne ronto workspace which allways fanouts all data to x filse then in workspace runner check output directory and merge shapefiles to ~4G max size files. – simplexio Jan 27 '15 at 9:49
  • You can clip your data with a vector grid and have each one write out as _1_0_1.shp etc. – Mapperz Jan 27 '15 at 15:04
3

No, there's no functionality that detects this and splits the file up for you. It's already registered as an enhancement request with our developers, and the reference number is PR#34128. I added a note linked to this question.

NB: Technically I believe the limit specified by the format is 2.14gb, so even 4.29 is stretching the format more than it should really be.

Of course, the question is whether a Shape dataset of that size is a good idea in the first place! Maybe you could use a different format?

If you are stuck on Shape, then I'd do as others have suggested. The ModuloCounter transformer is a good way to divide data (on a non-geographic basis) or you could split the data with either the Tiler transformer or the Clipper.

  • Thanks Mark. Yes, I've decided to write to another format. – Fezter Jan 27 '15 at 19:30
3

A good guide on how to approximate when you are reaching a ShapeFile limit is mentioned in this documentation by Esri

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002t0000000m000000

"There is a 2 GB size limit for any shapefile component file, which translates to a maximum of roughly 70 million point features. The actual number of line or polygon features you can store in a shapefile depends on the number of vertices in each line or polygon (a vertex is equivalent to a point)."

Now in FME you could count the vertices but that would really slow down your process, and is not advisable.

Preferred way would be fanout by some attribute, or limit to a number of features per shapefile.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.