Update: The solution to that particular problem, which got revealed bit by bit, was, that the Memory Layer Saver plug-in had to be installed to make the mask layer, which hides the labels, persistent and gets saved with the project.
For those who do not need to use the Mask plug-in but still want to hide particular labels or give them different appearance, the original answer might be still useful:
Labels are always on top of everything, so there is no chance to hide
them by other features.
I think there are two possibilities to get rid of the unwanted labels:
Intersect your vector layer with your area of interest, and delete the parts laying outside -> no feature, no label.
Influence the appearance of your labels directly.
There is no solution from scratch, so for most people its not obvious
how to do that.
Intersect the contour lines with the boundary, but instead of deleting parts, make e.g. a Spatial Query (possibly have to
activate the plugin first) to select those features inside the
boundary, then switch the selection to have all features outside
the boundary selected.
Create a number field named e.g. LblFontSize in the field
calculator and assign 0 to the selected features, leaving the others
without a value.
Then go to the Layer properties -> Labels -> Text -> Size. Behind
the input field with your default font size, there is kind of a drawer
symbol. Open the drop-down menu by clicking and select in the fly-out
under Attribute field your new columns name. The symbol should be
coloured yellowish now.
Apply the changes. The features without any entry in your new column
should now be labeled with the default font size, while those with a 0
are labeled with zero points hight -> invisible!