That is/was a known issue with nearblack. See the discussion in https://github.com/OSGeo/gdal/issues/6264 and the image in the comment https://github.com/OSGeo/gdal/issues/6264#issuecomment-1569883495.
The new filling option -alg floodfill
https://gdal.org/programs/nearblack.html#cmdoption-nearblack-alg should help with that issue. If it does not, try gdal_footprint https://gdal.org/programs/gdal_footprint.html instead.
-alg twopasses|floodfill
Added in version 3.8.
Selects the algorithm to apply.
twopasses uses a top-to-bottom pass followed by a bottom-to-top pass.
This is the only algorithm implemented before GDAL 3.8. It may miss
with concave areas. The algorithm processes the image one scanline at
a time. A scan "in" is done from either end setting pixels to black or
white until at least "non_black_pixels" pixels that are more than
"dist" gray levels away from black, white or custom colors have been
encountered at which point the scan stops. The nearly black, white or
custom color pixels are set to black or white. The algorithm also
scans from top to bottom and from bottom to top to identify
indentations in the top or bottom.
floodfill (added in GDAL 3.8) uses the Flood Fill algorithm and will
work with concave areas. It requires creating a temporary dataset and
is slower than twopasses. When a non-zero value for -nb is used,
twopasses is actually called as an initial step of floodfill.
For troubleshooting run the command with --debub on
.
The debug info shows that with the floodfill option GDAL needs to create a new file at 50% progress.
0...10...20...30...40...50GDAL: GDALDriver::Create(GTiff,nearblackout.tif.visited.tif,12000,12000,1,Byte,000001BC97402E00)