I created a KML file from a geotiff using Geoserver, but the image looks really pixelated in Google Earth. The image is a geotiff and is only around 7MB. Any reason why it might look so pixelated?
Please do inner tile and add overviews to your tiff file, see here: http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
Found this on a mailing list and I think it is should address your situation as well:
You might want to convert it to a real GeoTIFF as suggested in the list and see if it works.
As for the pixelated images, you can try tweaking the WMS Raster Rendering options.
Try using the bilinear or bicubic interpolation. It should be able to address your pixelation issues.