First of all set the extent, to do this set a expression in each of this parameters (X min, Y min, X max and Y max) in the Extents menu of the map properties.

click the icon (
) in front of each of the extent parameters, select the Edit... option in the popup, set the following expressions in each parameter respectively.
X min: x_min(layer_property('layer name/id', 'extent'))
Y min: y_min(layer_property('layer name/id', 'extent'))
X max: x_max(layer_property('layer name/id', 'extent'))
Y max: y_max(layer_property('layer name/id', 'extent'))
remember to replace layer name/id with your layer name.
Now in the Position and Size menu of the map properties set the following expressions (the same way we set the expressions of the extent) to the properties Width and Height

Width:
with_variable(
'extent',
layer_property('layer name/id', 'extent'),
150 / array_max(array( -- replace here "150" with the size in mm of the longest side
x_max(@extent) - x_min(@extent),
y_max(@extent) - y_min(@extent)
)) * (x_max(@extent) - x_min(@extent))
)
Height:
with_variable(
'extent',
layer_property('layer name/id', 'extent'),
150 / array_max(array( -- replace here "150" with the size in mm of the longest side
x_max(@extent) - x_min(@extent),
y_max(@extent) - y_min(@extent)
)) * (y_max(@extent) - y_min(@extent))
)
again, set the correct layer name in layer name/id