The Generate XYZ Tiles (MBTiles) tool uses GDAL to generate the output (QGIS <= 3.32)
The GDAL MBTiles driver documentation states it only supports PNG and JPG output, so may only support <= version 1.2 of the MBTiles spec. WebP support was added to the MBTiles spec at v. 1.3.
There is an open GDAL enhancement ticket (#3437) requesting WebP support be added.
So in short, doesn't look like you can enable WebP output with the QGIS Generate XYZ Tiles (MBTiles) tool.
However, maptiler engine supports MBTiles and WebP output and has a free desktop version. I haven't used it so can't comment on it other than that.