As both Paul and jvangeld have said, .rrd and .aux files are usually (but not necessarily) associated with .img files. To find out which format you have, just look at the directory contents and the images are usually the largest files.
For performance gains at the file-level, GeoTIFFs with overviews (reduced versions of the main image) and tiles is a pretty good starting point, so to achieve that you'll need to use two GDAL tools: gdal_translate and gdaladdo, that you will have to use from the command line. Assuming you're using Windows, you can download the OsGeo4W package that will install all the necessary files and provide you with a command prompt that you can work from.
To convert from whatever image format you've got to a compliant (ESRI has a habit of adding proprietary stuff to files) GeoTIFF:
gdal_translate -a_srs EPSG:4326 -co TILED=YES <source_image_file_name> <dest_image_file_name>.tif
The default tile size is 256 x 256 pixels which is reasonable for a WMS, and we're ensuring GDAL embeds the correct projection into the image by assigning EPSG:4326 (WGS84)
To create your overviews, you then need to run:
gdaladdo <dest_image_file_name>.tif -r average 2 4 8 16 32 64
This will create six "internal" or embedded overviews using a "moving average" filter. If you have a large image, you may need to adjust the number of overviews, just keep doubling the number at the end, which reduces the image size by half in each dimension. The highest value you need will be that which makes the reduced image just less than one tile square, although in practice you don't need to go that far.
You may also find you need a different filter, it depends on your data. Moving average is a reasonable tradeoff of speed for niceness, but you will need to view your image at different zoom levels to see if it is satisfactory. There isn't really a hard-and-fast rule, it's mostly just a case of trial and error.
You can view your new images in QGIS, and by zooming out, it'll automatically display the appropriate overview for that zoom level. If everything is to your satisfaction, then you can go ahead and add it to the GeoServer database.
If you want to overlay the images on Google Maps without GeoServer projecting them on the fly, you'll need to project your data from WGS84 to Web Mercator. Again this can be done with GDAL, just call gdalwarp after the gdal_translate above:
gdalwarp -t_srs EPSG:3857 -r cubic -co TILED=YES <dest_image_file_name>.tif <reproj_dest_file_name>.tif
As I've just noticed you said your source images are in ESRI Grid format, then they are going to be single-channel float32 or int32s, so you will need to assign a nodata value to your image when you run gdal_translate by adding
-a_nodata <some_value>
A usual value for nodata is -9999, but if your data has that as a valid value, you'll have to choose something else. Additionally, I've found gdalwarp doesn't use the source image's nodata value, so add:
-dstnodata <some_value>
to the gdalwarp command line parameters.
