Vector tile points displaced on some zoom levels

Question

I have a layer on geoserver with enabled tile image format: "application/vnd.mapbox-vector-tile"

Since I'm using EPSG:3765 projection (HTRS96 / Croatia TM) in my application and my data in PostGIS is also EPSG:3765, I have defined a gridset on Geoserver with the following settings (pixel size and scale values are auto generated):

This gridset is added to available gridsets of my layer.

Now, when I'm rendering vector tile layer in my Openlayers 5 application, on zoom level 11, all points are offset a bit to the south:

If I render the same layer as WMS, the points are in place on all zoom levels.

On zoom levels 12, 13, 14, 15, 16, 17 vector tile points are rendered in right places.

On zoom levels 18 and 19, vector tile points are offset again:

Pink point is WMS and black point is vector tile of the same layer

I can't figure out why are vector tiles misplaced when rendered on some zoom levels (11, 18, 19) and fit ok on zoom levels between. I suspect it is something with gridset, but can't figure out what. I have used vector tiles before but in EPSG:3857 and had no problems. Any ideas or hints would be apreciated.

Edit 1: I have defined resolutions in openlayers like this:

// Match the server resolutions
const maxResolution = 3862.66;
let defaultResolutions = [];
defaultResolutions.length = 19;
for (let i = 0; i < defaultResolutions.length; ++i) {
  defaultResolutions[i] = maxResolution / Math.pow(2, i + 1);
}

// TileGrid for EPSG:3765
let tileGrid = new TileGrid({
  extent: [247020.5267134084, 4688936.351599621, 741441.9826338722, 5160175.80631312],
  tileSize: 256,
  resolutions: defaultResolutions,
  hidpi:false,
});

Edit 2:

I tried using createXYZ:

let tileGrid = createXYZ({extent: [247020.5267134084, 4688936.351599621, 741441.9826338722, 5160175.80631312], tileSize: 256, maxZoom: 19})

Now tileGrid.resolutions_ looks like this:

[
    1931.3338121893116,
    965.6669060946558,
    482.8334530473279,
    241.41672652366395,
    120.70836326183198,
    60.35418163091599,
    30.177090815457994,
    15.088545407728997,
    7.5442727038644986,
    3.7721363519322493,
    1.8860681759661246,
    0.9430340879830623,
    0.47151704399153116,
    0.23575852199576558,
    0.11787926099788279,
    0.058939630498941395,
    0.029469815249470697,
    0.014734907624735349,
    0.007367453812367674,
    0.003683726906183837
]

This seems to match geoserver numbers exactly (openlayers has some more decimal places), but points are still misplaced like before.