The Generate XYZ Tiles (MBTiles) tool uses GDAL to [generate the output][1].  

The GDAL MBTiles driver documentation states it only supports [PNG and JPG output][2], so may only support <= [version 1.2 of the MBTiles spec][3]. WebP support was added to the MBTiles spec at [v. 1.3][4].  

There is an open GDAL enhancement ticket ([#3437][5]) requesting WebP support be added.

So in short, doesn't look like you can enable WebP output with the QGIS Generate XYZ Tiles (MBTiles) tool.

However, [maptiler engine][6] supports MBTiles and WebP output and has a free desktop version. I haven't used it so can't comment on it other than that.


  [1]: https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/qgis/TilesXYZ.py#L359
  [2]: https://gdal.org/drivers/raster/mbtiles.html#tile-formats
  [3]: https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md#content
  [4]: https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md#content
  [5]: https://github.com/OSGeo/gdal/issues/3437
  [6]: https://www.maptiler.com/engine/