You want to count the number of discrete pixel colours inside an area? As far as I know, that can't be done with zonal statistics.
e.g.
Polygon 1 - area is 1000 pixels
500 pixels are (122,135,21)
400 pixels are (22,132,178)
100 pixels are (2,156,99)
Doing that on an raw (unclassified) rgb image will give a HUGE number of possible colours. A polygon with an area of 1000 pixels could easily have 900 unique colours, for example. Most pixel colours will appear only once, and a few will appear twice.
A python/gdal script could be written to count unique colours, but it would be slow and memory-intensive for that very reason.
You can get around this by using a paletted image to "merge" similar colours into a single value. This means that greens would all appear as one value, greys another value, and so on.
(If you're familiar with Photoshop/GIMP, this is the same as reducing an image to a fixed palette size e.g. 16 colours, where each pixel is 'rounded' to the nearest colour from a representative palette of 16 colours)
Try "convert rgb to paletted" (raster > conversion > rgb to pct). More info here
This will classify pixels into groups of similar colour.
Now the problem is easier to address.
I took an aerial image and reduced it to 4 colours.
Then I ran gdalinfo to get a histogram of the values. In Linux, use the terminal, in Windows, use the OSGeo4W shell.
gdalinfo -hist /path/to/my.tiff
This gave me
313730340 154657 11795972 160083079 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
So there are 313,730,340 pixels with value 0. And only 154,657 with value 1.
You'll need to work out which value equates to which colour, though.
There's also a K-Means raster classifier in Orfeo Toolbox, this does a simplification of colours but uses a different algorithm.