Locking layers will certainly work, but I wanted to make sure there were a few other methods represented in the answers here.
I'll work with the following lines layer. As you can see, the features are already labeled.
Method One: Data-Driven Solution
Data-driven solutions are my favorite, because if you do them right, they always work. To set this up, first go to your layout, and under the Item Properties of your main map, set the Item ID to something. I've chosen
We'll need to use Rule-based labeling. Open up the default rule that appears.
In the Filter text box, enter the expression
@map_id IN ('Map 1', 'canvas'). We put
canvas in there to ensure that we can still see the labels in our main window.
With that filter in place, only Map 1 will have labels, regardless of layer locks, styling, scale, etc.
Method Two: Scale-based Labels
Given that overview maps are smaller scale than their counterpart maps, you can also set a scale boundary on your labels. This method only requires a single layer, and no themes or locking on the Layout view.
Again, we'll use Rule-based labeling. You don't actually have to establish any rules, though. Simply apply a minimum scale which is larger (i.e., a smaller number) than the scale of your overview:
It's minor, but if you are adjusting the size and scale of your layout items, you may need to adjust the value on your label scale limit.
Method Three: Map Themes and Duplicate Layers
Establish the Main Theme
Get your map styled the way you want in your map. In the Layers pane, click on the Manage Map Themes (the eyeball) icon, then select Add Theme... and name it however you like. I'll name mine "Main Theme".
Duplicate Layer and Establish Overview Theme
Right-click your labeled layer in the Layers pane and select Duplicate Layer. Rename the new layer something like "Your Layer - Overview".
In the Layer Styling pane, turn the labels off for this new layer. With the new layer on and the original off, add a new theme. Mine will be called "Overview".
If you are doing this with multiple layers, you may want to add groups to your Layers pane.
Layout: Link Maps to Themes
In you layout view, select your main map. Under Item Properties, check the box next to Follow map theme and select the "Main Map" theme. Do the same for the overview map, selecting the corresponding theme.
Using this method does have the drawback that your layer symbols are separate. Changing the styling of one will not automatically update the other. To get around this, you can right-click one layer and copy its symbology, then right-click the other and paste it. The Copy Style menu allows you to copy the symbols only, so you can safely avoid adding labels to the other layer.