Considering these four polygons quickly drawn out in QGIS for example:
Then you can see how the four polygons relate using the various spatial predicate operators:
> st_overlaps(lap, sparse=FALSE)
[,1] [,2] [,3] [,4]
[1,] FALSE FALSE FALSE FALSE
[2,] FALSE FALSE FALSE FALSE
[3,] FALSE FALSE FALSE TRUE
[4,] FALSE FALSE TRUE FALSE
3 and 4 are the only ones that "overlap" in this sense. 1 and 2 don't overlap because 2 is fully inside 1. Polygons do not "overlap" themselves.
However 1 intersects with 2, and 3 intersects with 4. Note also that polygons overlap"intersect" with themselves:
> st_intersects(lap, sparse=FALSE)
[,1] [,2] [,3] [,4]
[1,] TRUE TRUE FALSE FALSE
[2,] TRUE TRUE FALSE FALSE
[3,] FALSE FALSE TRUE TRUE
[4,] FALSE FALSE TRUE TRUE
You can also do stuff like "contains" - 1 contains 2 but 2 does not contain 1. Polygons do contain themselves:
> st_contains(lap, sparse=FALSE)
[,1] [,2] [,3] [,4]
[1,] TRUE TRUE FALSE FALSE
[2,] FALSE TRUE FALSE FALSE
[3,] FALSE FALSE TRUE FALSE
[4,] FALSE FALSE FALSE TRUE
Note I've not constructed polygons that only touch along edges or nodes. Mostly because I'm never sure if the predicate functions think these overlap or intersect - but there's st_touches for that. If you want the full set of possibilities of polygon relations then use st_relate and work out what you want from the DE9-IM model string.