I would also like to see an answer to this questions using either QGIS or ArcGIS GUI's. Meanwhile I am posting a Python approach to this problem because both QGIS and ArcGIS provide a way to read a raster into a NumPy array.
If you are using QGIS, you already have GDAL installed. With the following code you can read the raster as an array:
ds = gdal.Open(r'C:\path\to\file.tiff', 0)
arr = ds.ReadAsArray()
In ArcGIS you can use the RasterToNumPyArray function as follows:
arr = arcpy.RasterToNumpyArray(r'C:\path\to\file.tiff')
The resulting array will have 3 dimensions:
cols. Once you have this array, you can use NumPy to get the maximum value along the first axis (band dimension):
import numpy as np
idx = np.argmax(arr, axis=0) # 2D array -> rows * cols
You have now the band for each cell where the value is the highest (0 for red, 1 for green and 2 for blue). The next step is coverting the maximum values in each pixel to 255 and the other pixel values to 0 (so that pixel value will display the color of the maximum value). However, to accomplish this you first need to create a meshgrid so you can index your original array. Check the documentation if you are unsure what a meshgrid is.
x = np.arange(arr.shape) # 1D array from 0 to cols
y = np.arange(arr.shape) # 1D array from 0 to rows
xx, yy = np.meshgrid(x, y) # 2 2D arrays with the indices for each cell
Now you can index the actual pixels with the maximum values and convert them to 255. Then, everything that is not 255 can be converted to 0.
arr[idx, yy, xx] = 255
arr[arr != 255] = 0
If you used GDAL, then you can save your 3D array to a new GeoTIFF using the approach presented in this answer. On the other side, if you used
arcpy, you can use the NumPyArrayToRaster function. However, bear in mind that you will need to pass the raster's origin and the pixel resolution from your original raster to this function in order to get a correct georeferenced raster.
Note: in case of a tie (e.g. a pixel has the same maximum for two or three of the colors) this approach will yield the color mix between those colors. For example, in the case one pixel has R: 116, G: 166 and B: 87, both R and G values will be converted to 255 and B will be 0, yielding a yellow pixel.