What finally worked for @RobotCaleb was the following:
Apply mask to alpha (band 4):
gdal_translate -mask 4 stadium3857.tif stadium3857_RGB.tif
2. Compute a new alpha channel from NoData values:
gdalwarp -dstalpha stadium3857_RGB.tif stadium3857_RGBA.tif
edit: apparently gdal2tiles
does not need an alpha channel in the input dataset but also converts NoData values into transparency, so step 2 is not necessary