2

OS X El Capitan, GRASS GIS 7.0.1

I have a location with about 530 read-only mapsets with each about 2600 raster maps (plus PERMANENT and one link to a read-write mapset), overall 11.255.481 raster maps and about 181 GB.

What is the best way of storing these in a space efficient way and to have them accessible read-only?

I have them at the moment in a grass database and thought of putting these into a squashfs which should work (tried with a subset and it was nice - mounting fast, space efficient), but after two days of compressing (and only having compressed 30% - slow speed presumably due to huge number of files) I gave up.

I would like to avoid having to install postgresql as I tried it before (several years ago) and given up due to it's complexity.

I will analyse the maps in R, so any storage backend which I can use From R and GRASS would be OK.

Any suggestions?


ADDED INFO

Number of Files : 11.604.844 (find . -type f | wc -l)

Size of all Files : 181G (du -sch ./*)

One timing example:

01:39:24 /Volumes/simASM$ time mksquashfs ./simASM.Outeniqua.Random.Outeniqua_2_0.Outeniqua.41719.* ~/tmp/simASM.squashfs
Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on /Users/rainerkrug/tmp/simASM.squashfs, block size 131072.
[=======================================================================================================================================================================================|] 92263/92263 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
    compressed data, compressed metadata, compressed fragments, no xattrs
    duplicates are removed
Filesystem size 365271.48 Kbytes (356.71 Mbytes)
    50.25% of uncompressed filesystem size (726846.67 Kbytes)
Inode table size 801639 bytes (782.85 Kbytes)
    20.23% of uncompressed inode table size (3962401 bytes)
Directory table size 617819 bytes (603.34 Kbytes)
    20.52% of uncompressed directory table size (3010422 bytes)
Number of duplicate files found 84524
Number of inodes 122969
Number of files 110768
Number of fragments 1775
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 12201
Number of ids (unique uids + gids) 2
Number of uids 1
    rainerkrug (501)
Number of gids 1
    staff (20)

real    2m2.877s
user    1m6.972s
sys 1m57.861s
01:41:31 /Volumes/simASM$
1

Squashfs sounds ideal for this situation.

You mentioned "I have them at the moment in a grass database and thought of putting these into a squashfs which should work (tried with a subset and it was nice - mounting fast, space efficient), but after two days of compressing (and only having compressed 30% - slow speed presumably due to huge number of files) I gave up."

This should not have happened. Mksquashfs is multi-threaded and should have compressed 1,255,481 files making up 200 GB in less than a couple of hours, especially with a modern processor.

What version of Mksquashfs are you using (mksquashfs -version), and from where? (provided by the distribution or downloaded and compiled yourself). what Linux distro are you using?

Some older versions of Mksquashfs did have some bugs where they occasionally hung if they hit the right triggers (dependant on dataset, memory and processors). It may well be you're hitting one of these bugs.

You can download the latest release of Mksquashfs from http://sourceforge.net/projects/squashfs/

The development git tree has the latest bug fixes, and is here:

https://git.kernel.org/cgit/fs/squashfs/squashfs-tools.git/

If Mksquashfs appears to have hung, you should always check this with "top" and see how much processor time it is using (%CPU). If Mksquashfs has hung the processor time will be zero or near to it. If instead Mksquashfs is processing a large difficult to compress file (which is marked as a probable duplicate and so progress can only be made once the entire file has been checked), Mksquashfs may appear to have hung, but, "top" will normally show Mksquashfs using a significant amount of CPU time.

If you're using Mksquashfs 4.3, then this version has the ability to dump the state of the internal queues to the console. This feature was added specifically to diagnose reasons for slow running or hangs.

Sending SigQUIT to Mksquashfs twice within one second (hit control-\ twice on the keyboard), will cause the queues to be dumped, e.g.

Queue and Cache status dump
===========================
file buffer queue (reader thread -> deflate thread(s))
    Max size 24128, size 0 (EMPTY)
uncompressed fragment queue (reader thread -> fragment thread(s))
    Max size 24128, size 0 (EMPTY)
processed fragment queue (fragment thread(s) -> main thread)
    Max size unlimited, size 5
compressed block queue (deflate thread(s) -> main thread)
    Max size unlimited, size 0 (EMPTY)
uncompressed packed fragment queue (main thread -> fragment deflate thread(s))
    Max size 24128, size 0 (EMPTY)
locked frag queue (compressed frags waiting while multi-block file is written)
    Max size 24128, size 0 (EMPTY)
compressed block queue (main & fragment deflate threads(s) -> writer thread)
    Max size 48256, size 0 (EMPTY)
read cache (uncompressed blocks read by reader thread)
    Max buffers 24128, Current size 8, Maximum historical size 76
block write cache (compressed blocks waiting for the writer thread)
    Max buffers 24128, Current size 281, Used 24,  Free buffers
fragment write cache (compressed fragments waiting for the writer thread)
    Max buffers 24128, Current size 14, Used 0,  Free buffers
fragment cache (frags waiting to be compressed by fragment deflate thread(s))
    Max buffers 24128, Current size 2531, Used 1,  Free buffers
fragment reserve cache (avoids pipeline stall if frag cache full in dup check)
    Max buffers 25, Current size 0, Used 0,  No free buffers
[========================================================================- 

If the above dump does show Mksquashfs has hung (the dump is probably only fully intelligible by an expert on Mksquashfs, but the key to look for is full or non-empty queues at the start of the pipeline and empty queues at the end of the pipeline, which will indicate a blockage in the middle).

Varying the amount of processors used by the -processors option, or increasing the amount of free memory by the -mem option can often work around hangs.

Altering the block size with the -b option and disabling duplicate checking with the -no-duplicates option can also often work around hangs, as these options change the scheduling and threading behaviour.

e.g.

% mksquashfs blah blah.sqsh -b 1M -mem 2G -processors 1 -no-duplicates

Will use blocks of 1Mbytes, use 2GBytes of memory, only one processor and do no duplicate checking.

There is one potential pathological cause of the slow running of Mksquashfs. You should try the -no-duplicates option to determine if this is happening. Duplicate checking is a horrendous destroyer of performance in Mksquashfs. If a file is determined to be a potential duplicate of files already in the filesystem, each potential duplicate has to be read off disk and compared byte-by-byte. This is slow. Imagine the pathological case where the 100,000th file is considered a potential duplicate of 1,000 files already in the filesystem. Each one has to be read of disk and compared. Imagine if this happens for the 100,001th file, and for the 100,002nd file, and so on, and by the time the 1,000,000 file is reached it is comparing 10,000 potential duplicates. Mksquashfs will slowly grind to a halt.

Mksquashfs has an obvious safe-guard to prevent this exponential explosion of the amount of potential duplicates to be compared, it computes a checksum of each file added, and a file is only considered a potential duplicate if both the file length and checksum match. The likelihood a same length file has the same checksum is very low (by definition a different length file is not a duplicate). But, you could be hitting the situation where you have 10,000s of files of the same length, and for some reason they generate the same checksum, but are different. In that case there is nothing to prevent the exponential explosion of duplicates that need to be checked. I have to say in the almost 15 years Mksquashfs has been available this scenario hasn't happened to my knowledge, but there is a first for everything.

  • You are right - it sounds perfect for what I want to do. I am using mksquashfs version 4.3 (2014/05/12) on OS X El Capitan installed via homebrew. The command I used was (If I am not mistaken, I chose this one because of compression speed) mksquashfs ./simASM /Volumes/ASM_Optimization/simASM.squashfs. – Rainer Dec 1 '15 at 10:28
  • MacBook Pro (Retina, 15-inch, Early 2013); 2,4 GHz Intel Core i7; 8 GB 1600 MHz DDR3 – Rainer Dec 1 '15 at 10:30
  • I added the counted number of files and the size to the question. – Rainer Dec 1 '15 at 12:22
  • OK, it took about 2 minutes to compress about 700 Mbytes. Mksquashfs does not slow down with increasing files, and it's performance is therefore largely linear. Given this, and assuming the rest of the 181 Gbytes is similar, Mksquashfs should have taken less than 9 hours to compress ((181/.7)*122)/(60*60). – Phillip Lougher Dec 1 '15 at 16:31
  • Mac OS X is not supported or tested. You probably have hit some variant behaviour between BSD and Linux which is showing as a hang. – Phillip Lougher Dec 1 '15 at 16:34
1

As I am on OS X, I created a compressed .dmg (read-only) image which contains

  • the mapsets
  • a link to ./../grassAnalysis which is a read-write mapset located at the same directory of the mount point

This reduces the several million files to one file (the .dmg) and reduces the space required from 200 GB to around 100 GB. This is not yet ideal, but the approach of putting the location into a compressed read-only image file is the way to go.

0

For Linux systems (maybe also elsewhere), there is "fuse-zip".

https://bitbucket.org/agalanin/fuse-zip/

"fuse-zip is a FUSE file system to navigate, extract, create and modify ZIP and ZIP64 archives based on libzip implemented in C++. With fuse-zip you really can work with ZIP archives as real directories. Unlike KIO or Gnome VFS, it can be used in any application without modifications."

Likely you want to use a SSD disk here :) If you are willing to try, please report your findings.

Source: https://unix.stackexchange.com/a/168812/6003

  • The problem with these archive mounting approaches is usually that the mounting takes a long time if the archive contains many files, as it is the case here. But it sounds like an option, if the compression ratio is better than 2:1 as I get out of the compressed dmg. I will possibly try it out on a subset of my data. – Rainer Nov 7 '15 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.