2

Can anyone please let me know if there is a limit to the physical number of raster TIFFS that can be placed in QGIS?

I am trying to bring in a large number of StreetView tiffs from the Ordnance Survey OpenData suite, but have reached a point where I get an error message.

I have imported 1,998 tiles without any problems. When I import the next single file, it doesn't geo-reference correctly. Then the next one brings up the following error message:

"/OS_OpenData/StreetView/no/no05nw.tif is not a supported raster data source"

I am running QGIS 2.12 Lyon on Mac OS X El Capitan (10.10.5). The GDAL version is 1.11.3-1. This also happened on the previous QGIS version (Pisa), so I upgraded today to see if that fixed it.

Improve this question
2
  • Does it always fall over on the same TIFF?
    – nhopton
    Commented Nov 12, 2015 at 15:38
  • Check the maximum number of open files: "launchctl limit maxfiles"
    – Dave X
    Commented Nov 12, 2015 at 17:10

1 Answer 1

Reset to default
1

There is a limit to the number of files that may be opened by a single process on OSX, and it can be quite low. I had a problem working with a time series of tiles and had to increase my limits.

Try this on the command line so see some of the process limits:

ulimit -a
ulimit -Ha
launchctl limit maxfiles

If these limits are close to 2000, that could be your problem and it could be resolved through something like https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1 or http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Mac-OS-X

My current results for these are:

499 [drf@v21837 ~]$ ulimit -a 
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2048
virtual memory          (kbytes, -v) unlimited
500 [drf@v21837 ~]$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65532
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2048
virtual memory          (kbytes, -v) unlimited
501 [drf@v21837 ~]$ launchctl limit maxfiles
    maxfiles    65536          65536          
502 [drf@v21837 ~]$

Note that the maxfiles == 65335

My machine's currently a Mac OSX 10.10.5 and I have a /Library/LaunchDaemons/limit.maxfiles.plist per the http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Mac-OS-X link like this:

512 [drf@v21837 ~]$ sed 's/^/    /' /Library/LaunchDaemons/limit.maxfiles.plist  
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxfiles</string>
          <string>65536</string>
          <string>65536</string>
        </array>
      <key>RunAtLoad</key>
        <true/>
      <key>ServiceIPC</key>
        <false/>
    </dict>
  </plist>
Improve this answer
4
  • Thanks for the reply. I have had a go at the two options on these websites, but I am not a big Terminal user and I think I am not doing something right, as neither seem to work. I really need an idiots guide to get these working!
    – Sean McVeigh
    Commented Nov 13, 2015 at 16:08
  • Let me look at it more to see if I can figure out exactly what I did.
    – Dave X
    Commented Nov 13, 2015 at 18:56
  • To set the limits at the system level, you have to reboot. I seem to remember that if you set them too high, they revert to defaults and ignore the settings. Are your max files limits as seen within a Terminal close to the 2000 files where you are experiencing a problem?
    – Dave X
    Commented Nov 13, 2015 at 19:06
  • 1
    I wonder if there's any mileage in the idea of 'combining' the files into a virtual raster file, a VRT (*.vrt)? Raster -> Miscellaneous -> Build Virtual Raster. You'd have to set some sensible extents before loading the file into QGIS, or QGIS would try to load the lot.
    – nhopton
    Commented Nov 14, 2015 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.