Geopandas shows points outside of buffer geometry even though inside

Question

I have some ship trajectories and I want to know which point of the trajectory is inside the port(=buffer) and which points are outside. The following picture shows my Geopandas Dataframe with the geometry.

The problem is that some points that are inside, are shown as outside. The problem can be easily seen if I use a huge buffer.

After that I am loading the port's point and buffing it, like below.

And then I am finding which points are inside and which outside, with the code below.

Now if I use a huge buffer, the are still rows in the points_outside_geometry dataframe. These rows shouldn't be tracked as outside points and this can be verified by the following plot.

How is that possible?

Code:

polys = gpd.GeoDataFrame.from_file('final_port.shp').buffer(100000)
sindex = gdf.sindex
points_within_geometry = pd.DataFrame()


for poly in port_poly:

    possible_matches_index = list(sindex.intersection(poly.bounds))
    possible_matches = gdf.iloc[possible_matches_index]
    precise_matches = possible_matches[possible_matches.intersects(poly)]
    points_within_geometry = points_within_geometry.append(precise_matches)

points_within_geometry = points_within_geometry.drop_duplicates(subset=['MMSI', 'TIMESTAMP'])
points_outside_geometry = gdf[~gdf.isin(points_within_geometry)].dropna(how='all')