3

I have inherited a project where part of the workflow is the execution of numerous FME Workbench files. It has worked flawlessly for several hundred runs, but it gets tripped up on a very large shapefile(actual shp is 1586981264 bytes, dbf is 51949349 bytes Type: Polygon
Number of Shapes: 775361 ).

Corrupt Shape file '<shapefilepath>' encountered -- record size 218972988 is invalid

ArcCatalog and shpinfo.exe have no trouble with it.

Based on some google searches, I think the issue is with the size of the shapefile but this might be outdated information: http://groups.yahoo.com/neo/groups/fme/conversations/topics/3031 http://evangelism.safe.com/tag/google/

Is there a way to configure FME to allow for larger shapefiles?

FME Information:
Edition: FME Professional Edition (node locked-crc)
Version: FME(R) 2012 SP2 (20120417 - Build 12238 - WIN64)

OS:
OS Version: 6.1.7601 Service Pack 1 Build 7601
OS Manufacturer: Microsoft Corporation
OS Build Type: Multiprocessor Free
System Type: x64-based PC
Processor(s): 8 Processor(s) Installed.

Intel64 Family 6 Model 44 Stepping 2 GenuineIntel ~3325 Mhz * 6
Total Physical Memory: 31,996 MB

Improve this question
11
  • 1
    800 million features? I can't think of any way to store that in shapefile format -- the limit is 2^31-1 short words in the .shp file or 2^31-1 bytes in the .dbf (which would limit you to ~533M 4-byte integers). Can you provide exact file sizes for the .shp and .dbf, and a description of the dBase attributes (formatted width is key)?
    – Vince
    Commented Sep 25, 2013 at 19:06
  • shapefilerepairer arcscripts.esri.com/details.asp?dbid=10806
    – Brad Nesom
    Commented Sep 25, 2013 at 19:12
  • @Vince: I forgot to remove some zeros (or omit the K). Updated question w/ actual counts/sizes. Definitely not 800 million!
    – Jay Cummins
    Commented Sep 25, 2013 at 19:14
  • Is using arcmap to do 'shape to file-geodatabase' out the question? Then running the workspace with that? - It must be hitting a limit in the Reader.
    – Mapperz ♦
    Commented Sep 25, 2013 at 19:16
  • The 'shpinfo' utility in an ArcSDE install would report if the file is actually corrupt. You're not anywhere near the shapefile limitations (unless the dBase record size is huge), so this is likely an application limit.
    – Vince
    Commented Sep 25, 2013 at 19:28

2 Answers 2

Reset to default
2

Unfortunately it looks like a fault in FME. It is related to content size, but not in a way I think you can workaround.

I've filed a problem report with the developers and it does appear a relatively simple fix. If you're able to supply a copy of the data then please report the issue via safe.com/support, referencing PR#48195 and making the data available (you can upload it to ftp.safe.com/incoming) - though hopefully we can fix the issue without it.

looks like we have the data and the issue is fixed in FME2014 (build 14167 or greater).

Improve this answer
3
  • Thanks Mark. I submitted a problem report, but forgot to reference PR#48195. I replied to support email, but not sure if that will attach it to the support ticket. C81210 - Safe Software Support Ticket Created [ ref:_00D30ePES._500a0ZorcF:ref ]
    – Jay Cummins
    Commented Sep 26, 2013 at 13:05
  • 1
    Hi Jay - looks like we have the data and the issue is fixed in FME2014 (build 14167 or greater). So it should all be OK, but give us a shout if you still have problems.
    – Mark Ireland
    Commented Oct 7, 2013 at 14:10
  • I am using FME2014 build 14168 and I think it is safe to say that I'm past the shapefile issue (36 hours and still running...).
    – Jay Cummins
    Commented Oct 10, 2013 at 19:07
2

The approach I would do in this situation is.

In ArcMap load the large shapefile and convert this into a file-geodatabase. Why? The structure of the file geodatabase has been created by ESRI not by choice but because they had to - larger and larger files impacted on the shapefile (2GB limit at the time) and also the access/personal geodatabase (.mdb) has poor stability at nearly capacity (2GB).

So moving to the shapefile to the File Geodatabase should see the (ArcObjects) reader in FME have no issues reading the geometry. Yes is more work to include the File Geodatabase into FME. But a time-saver when you think of how much time you have already used in trying to convert the shapefile directly.

Not a perfect solution more of a work-around.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.