5

We need to determine how much of the 6.5 TB used space on a shared network drive is geographic or spatial data. It's a squirrels nest, built up over 3 decades from hundreds of people (though by volume mostly in past 3 years) across all walks of business using a variety of naming and filing standards -- including none and the dreaded "misc".

Oh oh. <7% free space

"Spatial" in this context is anything produced by ArcView 3, ArcInfo, ArcGIS, PCI, ER Mapper, Global Mapper, QGIS, DNR Garmin, Basecamp, MapInfo ...

My plan of the moment is a simple brute force itemization of file extensions, at least to get us some ballpark estimates.

Singular types like shapefiles and ArcInfo grids & coverages are easy as the extensions .shp, .shx, .adf, ... aren't used by anything else.

Shared types like DBase .dbf and images .tif, .jpg, offer complication, though ones larger than 300 MB are very likely spatial.

Before I hack together a personal and idiosyncratic solution. Has anyone solved this already?

Improve this question
4
  • Ideally, all of the spatial data saved should be registered in some kind of metadata repository, but sadly even in large enterprises it's just a mapped drive ;) sigh. Faced this problem earlier, but instead of just estimating the size, I've gone through the piles and categorized the data, moved things around, and registered everything in a repository to keep track on what exists. Good luck with this tedious work, hope things will work out well for you.
    – Alex Tereshenkov
    Commented Jul 29, 2016 at 18:07
  • 6
    gdalinfo and ogrinfo can auto detect format from a single file, and will list other detected files identified as belonging to a data source, so can be used to audit and organize
    – mdsumner
    Commented Jul 29, 2016 at 18:30
  • Its an intractable problem, not least because of a lack of agreement on what counts as spatial. A PNG image of my cat (if I had one) and a PNG format of aerial imagery both look the same, even to GDAL/OGR. That aerial imagery PNG might or might not be spatial depending on your opinion, or depending on the existence of a world file that may be in another directory, or on some users' desktop. Or the PNG is spatial if its in a specific directory structure that makes it Z/X/Y tiles..
    – BradHards
    Commented Jul 31, 2016 at 1:35
  • Only partial support, but if on Windows, I've often started with a GUI solution to help figure out priority places to start. Check out WinDirStat - note there are linux and macosx variants.
    – SaultDon
    Commented Aug 13, 2016 at 3:38

1 Answer 1

Reset to default
1

There's always some manual effort in this type of task, but I would use something like this as a starting point. I found the basic code as vahid abdi's response to this post and just modified it a tiny bit to be compatible with both Python 2 and 3. It's rough, and I would suggest improving the script in order to sort/filter the data for specific extensions or sort by total size as well as adding easy filtering for specific extensions. I ran this and found that we have 2.6 GB of file databases (.gdf). 

import os
import collections
extensions = collections.defaultdict(int)
size = collections.defaultdict(int)

for path, dirs, files in os.walk('./'):
   for filename in files:
       extensions[os.path.splitext(filename)[1].lower()] += 1
       size[os.path.splitext(filename)[1].lower()] += os.path.getsize(path+os.sep+filename)

for key,value in extensions.items():
    print ('Extension: ', key, ' ', value, ' items')

for key,value in size.items():
    print ('Extension: ', key, ' ', value, ' size')
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.