With WMS, the server is querying for these features, creating the image and sending this image over the wire to your client. Factors affecting time of response are database query, image creation and network latency. If these images are being served from the WMS cache, you get an even faster response, as the time required to generate these images will be minimized.
With WFS, the server is also querying for the features, packaging into your output format of choice and sending this over the wire. In both cases, the spatial query will be quick. Network latency will be almost be same although it may be larger for WMS images due to the larger payload. However, the main difference to server response to a WMS and WFS request will come from processing of those features retrieved from the database. Creating an image requires more computation power so response to a WMS request should generally be slower for the same number requested features.
However, your GIS engine has to do the same thing as the WMS server in response to a WFS request; it has to style these features and create the corresponding image and render this on a map. Desktop hardware is slower and since image processing is computationally expensive expect this operation to be slower on the desktop. For a small number of WFS features most of the delay contribution will come from server operation and network bandwidth constraints. On the other hand, most of the delay will come from desktop processing, when you get a large number of features in a WFS response. The reverse is almost true for WMS requests.