A big question is whether you are going to read the entire raster from the file into memory before processing it, or whether the file is so large that you will process it incrementally, or process some subset of the overall file.
If you will load it all into memory, then you will be doing mostly sequential access, and the fastest format will be a tossup between plain and compressed storage (depending on things like how fast your CPU is versus disk). Any of the binary file formats will probably be pretty close (ASCII will be slower).
If you need to process a subset of a very large file, then a format which groups the subset you want closer together may be faster - eg: tiles or a format which can compute offsets. Sometimes uncompressed approaches gain here because it may be trivial to compute where any given part of the image resides within the file, especially if you need only part of a very large row, but compression can be done in a granular fashion that works well for some access patterns.
Sorry, but you'll probably have to benchmark depending on your access pattern, rather than getting a one-size-fits-all. It may of course depend not just on the file format and the above factors, but on the drivers for that format and your software.