It depends on your needs. If you need to display a relatively small number of relatively simple features (such as all locations for shop) then you can eliminate Geoserver and simply use GeoDjango as an ersatz web feature server by serializing data into kml or geojson and serving it to OpenLayers for display, you can also do plenty of CRUD operations easily in GeoDjango.
If you have relatively larger and complex data (such as land parcels or building outlines for a city for example), then you will want the 'proper' WFS features that you get with Geoserver. You can still use GeoDjango to create/query/update/ data in the db, but GeoServer would operate entirely separately from GeoDjango, using data from the same db to fulfill requests coming in from OpenLayers.
For me GeoDjango's biggest strength is being able to use spatial queries with basically the same great ORM syntax as Django, this makes it incredibly easy to write and test complex spatial operations that perform well. The ability to chain queries together is superb. Although you can use it as a simple feature service, it would be difficult to make it perform well under heavy use compared to a purpose-built WFS like Geoserver.