Storage hardware options for large datasets

Question

We are about to organize our data in a centralized spatial data system. The system would provide various services (mainly supporting ftp access, geodatabase editing, and several web mapping applications).

The data we already have is sized around 15TB, is expected to constantly increase, and consisted of spatial databases, aerial imagery, lidar data, and some non-spatial data. Furthermore, I understand that only a small portion of data would be actively accessed, and the storage is a sort of central repository (traffics and transfer rates are not that big deal in that manner).

I don't have any experience in managing such a big storage tier. I am under the impression that I have to purchase some NAS. However, I am not quite sure how should I plan the architecture of the storage tier (distributed or a single device), and what hardware specifications should I look for if the aim is 20TB capacity right now?

Suggestions are appreciated.

Answer 1

Concerning the hardware part: In our GIS cluster (http://gis.cri.fmach.it/cluster/) we have recently added 96TB based on low end consumer hardware, obviously selecting carefully the disks. The file system system is based on the open source storage software glusterFS which enables us to add new storage in future easily ("GlusterFS is an open source, distributed file system capable of scaling to several petabytes"). It even supports remote synchronization if not all storage devices are physically in the same place.

We use the system to store 30TB+ (growing) of postprocessed MODIS data on it, DEMs, aerial and further satellite imagery and base cartography.

Answer 2

See the answers to Managing large amounts of geospatial data?

version control is a huge issue, metadata should play a big role.
I think the stock/obvious answer would be to use a spatial database (PostGIS, Oracle, SDE, MSSQL Spatial, etc) in conjunction with a metadata server such as esri's GeoPortal or the open source GeoNetwork application, and overall I think this is generally the best solution. However, you'll likely always have a need for project-based snapshots / branches / tags. Some of the more advanced databases have ways of managing these, but they're generally not all that easy to user/manage.