I guess you are mentioning the OSM plugin. The openlayer plugin(s) adds a raster (tile-based) image, a map based on OSM data.
The OSM plugin is easy to use, but as some limitations. I have listed some of them here:
http://hub.qgis.org/projects/quantum-gis/wiki/Using_OpenStreetMap_data
Which led me to build my own tool (described along with other ones, with a video), then add some C++ code in QGIS, then develop my own tiles (outdoormaps dot org website), all of this with QGIS.
This is significantly harder to use and configure than the plugin. However, it is meant for Quantum GIS as well and works well even for large dataset, and really keep all data (all tags, tries to keep some buggy geometries).
If you really want something easier, you can also try the daily shapefiles made by cloudmade or geofabrik (but with less data and more problems).
Hope this helps,
Mayeul
(stackexchange.com anti-spam policy prevents me from putting more links).