Extract pixels with similar color (tolerance) in QGIS
There is a way to do this in QGIS using raster calculator.
Remark: the image used here for demonstration purpose is really a "difficult" one: what should be one single color in fact consists of a lot of different shades that are difficult to perfectly isolate, even using a professional image processing image. Keep this in mind - other images, as the one you posted, seem to be easier.
Manual version: the basic principle
Sample the RGB values of the raster with the Identify Features tool and for each band get the min/max values of the color you want to mask.
Use raster calculator with an expression like this and replace the pixel values for each band with those from above:
"raster@1" > 175 and "raster@1" < 210 and
"raster@2" > 175 and "raster@2" < 195 and
"raster@3" > 70 and "raster@3" < 135
Result for masking the Ottoman Empire with the values from above:

Semi-automatic version: calculate min/max range and create raster calculator expression
For a partial automatization of this solution, proceed as follows:
- Create a point layer and add a few points on spots that cover the raster color you want to mask. The points are intended to sample the RGB color values of the raster.
- Get the raster-value of all three RGB bands using QGIS expressions with
raster_value()
(similar to this solution).
- Then aggregate these values for all points to get min/max values for each band. An even more sophisticated solution could include adding a further tolerance, like subtracting/adding 10% of the total range from/to the min/max values.
- Based on that automatically create the expression used for the raster calculator.
Use this expression on the point layer to generate an output string that you can copy and introduce without any changes to the raster calculator:
regexp_replace(
array_to_string (
array_foreach (
array (1,2,3),
'"raster@' || @element || '" >' ||
eval (' array_min (array_agg (raster_value (''raster'',' || @element || ', $geometry)))' ) ||
' and "raster@' || @element || '" < ' ||
eval ('array_max (array_agg (raster_value (''raster'',' || @element || ', $geometry)))') || ' and'
)
),
'(,)|(and$)',' ')
The output of this expression looks like this for the example I used in the next screenshot: use this in the raster calculator:
"raster@1" >181 and "raster@1" < 217 and "raster@2" >178 and "raster@2" < 201 and "raster@3" >64 and "raster@3" < 114
Example used with (on the left) in yellow the points used to sample the colors to mask; on the right: the output from raster calculator used with the expression generated:

Further possibilities: PyQGIS
I suppose that this second, semi-automatic version could be further automatized using PyQGIS, but that is your domain of expertise. It would be great to create and share a skript based on the second, semi-automatic solution like: click once (or twice or more times) on the picture, then the pixel values are internally calculated (maybe even with an option to define an additional tolerance range) and an output as on the screenshots is created. Or even several outputs with different tolerance ranges to select the best one. I guess a preview or pixel selection (like in image processing) is not possible in QGIS, but I might be wrong.
Workflow with Image processing software
The above solution works more or less, but can be cumbersome and does not give you the type of intuitive work that you have with image processing software. Therefore, I use a much easier workflow:
Export the image from QGIS and create a world file.
Before further proceeding maybe save a copy of the image and worldfile.
Open the picture in an image processing software like GIMP, Affinity Photo or Photoshop and do the color selection/replacement there. Save the image.
Reopen the image in QGIS. If the corrected image and the world file are in the same folder and have the same filename (and you don't change size/resolution of the image), the georeference still works.
I suppose this kind of selecting colors with a tolerance is not a core functionality of GIS. However, it is a core functionality of image processing. Because of that, it is better and easier done there. It is much faster and gives better results.
The following result was achieved with two clicks using Sampled colour pixel selections in Affinity Photo - not worse than the one above, but much faster:
