The expression returns the error if there were circles or polygons very similar to circles among the polygons;
in fact assuming the presence of a circle:
(2*pi*r)^2 < 16*pi*r^2
pi(4*pi*r^2) < 4(4*pi*r^2)
that is pi < 4

EDIT1:
To prevent the error from appearing you can use the expression:
try(
($perimeter -sqrt($perimeter^2 - 16 * $area)) /4
,0)
which will calculate the expression in all polygons except the ones that generate error ((in these polygons, it will always return 0).
try function
: https://docs.qgis.org/3.22/en/docs/user_manual/expressions/functions_list.html#try
EDIT2:
you also need to pay attention to the number of digits after the comma (for the perimeter and area) because the difference $perimeter^2-16*$area
could be negative, so I also suggest this solution:
with_variable('perim', round($perimeter,1),
try(
(@perim-sqrt((@perim)^2 - 16*round($area,1)))/4
,0))
$perimeter^2 - 16 * $area
and see if there are any negative values$perimeter ^2 - 16 * $area
is negative !!!