The secret lies on the lock layers option in the map item properties.
First, you need to prepare the layer for your "overview" map in the map canvas. That is, a layer with the desired municipalities already filtered, and turn off all the other layer that you don't want to be seen in the overview map.
Open the print composer and add a new map, just like Sorin Călinică said, and adjust it to the size you want.
Now lock the layers of that map item.
Go back to map canvas, turn the original layers on (and probably the ones you use off). Know refreshing the print composer, you will see that your overview map will not be changed.
Note: Since you already have the overall\base map prepared, you can lock the base instead. But chances are that you probably will change that map a lot more that the overview one.