0

My external hard drive is corrupt - according to my IT Expert possibly because of the Fat32 format. Is FAT32 really not suitable for GIS data? ie large Tifs but also thousands of 1kb txt files (e.g World files). Are there any situations when FAT32 is prefered to NTFS?

1
  • 1
    This is not really a GIS question. It is a Hardware/Software Limitation.
    – Mapperz
    Nov 7, 2014 at 14:11

3 Answers 3

4

Probably the biggest limitation you'll hit is that FAT32 can't do files larger than 4G. That may be an issue for very large imagery (TIFF, NITF), and is one of the concerns when using sqlite for storage (e.g. mbtiles or geopackage), since it is essentially a pseudo-filesystem as a single file.

I'd probably go with NTFS unless you have some particular compatibility issue where one machine can do FAT32 but not NTFS (perhaps an ancient Linux or MacOS version?).

Its difficult to say from the detail you've provided, but corruption because of FAT32 doesn't make sense to me. There is a reason why the FAT32 format is corrupted (hardware failing, software bugs, firmware bugs), but just "being FAT32" isn't a real reason.

1
  • 3
    +1 for FAT32 not causing the corruption.
    – Vince
    Nov 7, 2014 at 14:57
1

I don't think there's anything specific regarding GIS data. It's just a bunch of files.

The main reason for using FAT32 is compability for use with external drives (as yours). Otherwise NTFS is probably a better choice on Windows.

This article http://www.howtogeek.com/177529/htg-explains-why-are-removable-drives-still-using-fat32-instead-of-ntfs/ digs at bit deeper, but not too technical.

1

I disagree with @Mapperz. GIS can push both your CPU, memory, and hard drive limits because of the volume of data that is processed. Eventually, you will want to add some of these types of concerns to your GIS bag because they will affect your GIS projects.

You had either a defect in part of the surface of the drive or partial failure. Depending on the problem you may be able to recover part of the data and then reformat the drive. Pick FAT32 or NTFS. Have your IT expert check the drive to make sure it is still usable!

I used to use FAT32 all the time for my GIS work. Then one day several years ago, I downloaded an OpenStreetMap plant file that was larger than 10 gig. That is the final MsDos file/partition limit. I used the USB drive to move data between MS Windows and Linux all the time. I was more concerned about NTFS issues on Linux than MsDos issues until that fateful day. Well I switched to NTFS and have no problems with the USB drive use on either MS Windows or Linux. I've used NTFS on XP, Windows 7, and older releases of Fedora Linux back to Fedora 11.

The many small files is an issue for every operating system. That's why there are all sorts of ways to handle this issue in GIS. Think of TMS format that requires 1.0/name/z/x/y.png, if I recall correctly. Then there are just z/x/y.png strategies with the origin flipped. Behind the scene the file may be stored on the file system as z/12/12/12/12.meta. I am thinking of how Tirex in OpenStreetMap works. The MBtiles sqlite database is another way of solving the many file problem. MediaWiki uses a md5sum hash of the file name. The first parts of the name are selected to create a x/yy/xyy40charactersxxx.txt kind of structure.

The long story short is that all these schemes help applications of all types avoid file system issues. You can pick your drive partition format based on scalability, reliability, portability and what not. You will want to consider a hashing solution, if the many small files in one directory start slowing down your mapping work.

Not the answer you're looking for? Browse other questions tagged or ask your own question.