Is there a straightforward way to download data from OpenStreetMap (OSM)?

Using North America as an example, I realize Cloudmade (http://downloads.cloudmade.com/north_america) and geofabrik (http://download.geofabrik.de/osm/north-america/) both provide downloads, but the files are enormous (0.5gb - 4gb). I've also tried using the QGIS-OSM toolbar but I have not had much luck doing this, as similar to the OSM website the area you can select is very small. The amount of data I want to download is at the city level.

Finally, is there an easy of using OSM data? If I choose to download OpenStreetMap XML Data I can't open the osm.xml files using QGIS (even using the QGIS-OSM toolbar import function). I wonder if I'm missing a more obvious way of dealing with the data.

I just want to point out the very useful StackExchange clone for help with OSM related issues! http://help.openstreetmap.org/