Arcmap is giving me an error when using the masking option for dot density. I turn on masking for the dots in the Obama data frame using the lakes layer. Then, when I click apply, all the dots are gone. When I re-open the masking option dialogue, it has automatically selected surrounding counties; thus, the dots are masked from the entire area and do not draw. The McCain data frame does something similar. I select lakes for masking and check the use masking box. When I click apply it still draws the dots on the lakes. When I open the placement dialogue for masking again, the use masking box is no longer selected. I have tried restarting Arcmap, but still get the same result. I tried turning different layers on and off or removing them, it does not help. For example, if I remove surrounding counties from the Obama data frame, it now defaults to Franklin county after lakes was selected. Clearly there is an error with the software. I wonder if it has something to do with converting the road labels to annotation?
I have had problems getting the masking feature of dot density symbology to work if my map is very complex.
Try removing all the layers from your map except the dot density layer and the exclusion layer. If it replicates my experience, the dot density masking feature will work fine with just the two layers in the data frame.
Then you can start adding layers back as long as the dot density masking keeps working. I've never taken the time to figure out exactly what the issue is.