OpenLayers tiles from reprojected mapdata mixes zoom levels

Question

We have a map tile layer on OpenLayers where original projection is EPSG:3067 and it is reprojected to EPSG:3395 projection. The map is coming there quite ok, it has not the tiles as straight squares, but still ok for us.

The problem is that in the northern part of the map the tiles that are on map become from two layers. There is a horizontal line with many curves that separates the tiles in north with another zoom level and south with another.

Preview on GeoServer shows the tiles all at same zoom level, openlayers code based map does not. We had an issue about same kind when we had custom zoom levels on map, but removing them the divider appeared in north. Previously it was much souther.

What could cause the layer to take tiles from different zoom level per tile basis, having two different levels in use on the whole map, division between north and south parts.

The issue would be kinda ok, if there weren't place names on map. Now they're cut and missing partly.

Layer is initialized with a ol.source.TileWMS taking in layer name, format (image/png), transparency and a version.

The layer is a ol.layer.Tile with that tileWMS as source, title and visibility.

All that seems so basic, there is at least no hacks on there. What could cause the issue?

Edit1:

It appears the GeoServer returns tiles zoomed differently. I took the tile request urls and in the tiles from these I found tiles looking like they were zoomed to different levels.

It requests tiles 256x256 from different bounding boxes in those queries, somehow they happen have different zoom levels.

May the conversion on fly somehow mix the thing?

Edit2:

Newest finding is that the original data is a layer group, so that may cause the switch of level, if some requests give data from another group from some reason.

Answer 1

I changed boundaries of the layer to totally new numbers, other than on map and other than configured on GeoServer, this is the relevant code I added to Layer init:

 tileGrid: new ol.tilegrid.XYZ({
                extent: [-380188, 5249943, 1347312, 8226943],
                resolutions: [2048.0, 1024.0, 361.0, 256.0, 128.0, 64.0, 32.0, 16.0, 8.0, 4.0, 2.0, 0.5]
            })

This caused the effect to disappear from the map.

There was previously this same code with extent's second number 6249943, I lowered it by million and the issue disappeared.

Sidenote:

Maybe the issue happens now out of written map somewhere, but when the service returns only a limited part of world (one country, Finland), that's not a visible issue anymore. With a world map it could still be an issue somewhere. Still, won't this even realize on whole world maps, is another story. Like pointed by @AndreJ the projection in use is an incommon one.