There's a lot of useful information in the comments, which I summarize here so it doesn't get buried. Credit to @Mapperz and @Juan.
Adobe products have a limit of 32,768 vertices (16 bit limit?). The limit was reached in this situation because of a vector layer with
- high vertex density (very detailed)
- worldwide extent (much wider than the map extent)
QGIS doesn't crop the data to the map extent when exporting to pdf, so the vertices outside the map extent still count towards the total number of vertices. I made a feature request.
The current solution is to reduce the number of vertices before exporting. Starting with the layer with the most vertices, apply these methods until the output falls below the 32,768 vertex limit.
- Crop the data to the map extent, as explained in @Juan's answer. This method will not alter the data within the map extent.
Simplify algorithm to reduce the vertex density. This method will alter the data within the map extent.
This algorithm simplifies the geometries in a line or polygon layer. It creates a new layer with the same features as the ones in the input layer, but with geometries containing a lower number of vertices.
A workaround would be to export as raster instead of vector. Rasters don't have vertices, so the 32,768 vertex limit doesn't apply.
These questions remain unanswered:
- Is there a way to get the vertex count?
- Is 32,768 vertices an absolute limit or is there a way to increase it?
- Does filtering the layer reduce the number of vertices in the output? This would be a more convenient solution than cropping or simplifying.