The Generate XYZ Tiles (MBTiles) tool uses GDAL to generate the output.
The GDAL MBTiles driver documentation states it only supports PNG and JPG output, so may only support <= version 1.2 of the MBTiles spec. WebP support was added to the MBTiles spec at v. 1.3.
There is an open GDAL enhancement ticket (#3437) requesting WebP support be added.
So in short, doesn't look like you can enable WebP output with the QGIS Generate XYZ Tiles (MBTiles) tool.