It is a matplotlib problem and not a GeoPandas problem.
See the solution of Seb Bacon: geopandas-legend-font-size.ipynb
As the Jupyter notebook has problems (error opening it), I describe it here.
The legend is actually a Colorbar object. To change the legend's font size, we have to get hold of the Colorbar's Axes object, and call .tick_params() on that.
import geopandas as gpd
world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
world = world[(world.pop_est>0) & (world.name!="Antarctica")]
world['gdp_per_cap'] = world.gdp_md_est / world.pop_est
First, we call the .plot() method as usual, but we capture the return value which is the Axes of the map.
ax = world.plot(column='gdp_per_cap', cmap='OrRd', legend=True)
We can use this to get hold of the Figure that contains the Axes.
fig = ax.figure
The Figure has two Axes: one for the map, and one for the Colorbar. The one we care about is the second one.
cb_ax = fig.axes
We can now set the font size for the Colorbar. Other parameters for tick_params are documented at: matplotlib.axes.Axes.tick_params