I've been working on this issue for a while, and have found a workaround, but would still like to put the question out there. I have been using TileMill to create tile sets for a polygon, outside the borders of which I'd like transparency. If I export an mbtiles file and serve it as an overlay with the GeoServer MBTiles plugin, the following image is the result (the baselayer is OSM):
What I assume is happening is that TileMill doesn't create full blank tiles for the areas that have no data in them, and GeoServer interprets those missing tiles as opaque white. My question is: is there a way to get GeoServer to display those non-tile areas as transparent?
I'll describe my workaround: using information from this thread I found that I can get the desired effect by bypassing GeoServer altogether, and serving the unpacked tiles as an OpenLayers XYZ layer (the link describes how to avoid the pink error tiles). This is fine with me, but I'm still wondering if GeoServer can be used.
I also tried unpacking the tiles, adding blank "filler" tiles where necessary, and then repacking the tiles with mb-util. However, GeoServer hasn't been able to serve the repacked tiles correctly yet so I haven't gotten very far in that direction.