The symbology representation setting works when both lines are dotted. I am not sure if it works for a dotted and solid line overlap but it is worth a try. Here is my [answer](http://gis.stackexchange.com/questions/163719/how-to-handle-a-dashed-border-line-that-becomes-solid-because-it-is-shared/163745#163745) for a similar question. If the above doesn't work, You can convert your polygon features into line feature using feature to line tool and then use the erase function to remove all the areas the newly converted lines intersect with your dotted line feature. then use that instead of the polygon feature in your maps.