I'm not sure about how best to do this with R, but if you don't mind using the osmconvert tool (http://wiki.openstreetmap.org/wiki/Osmconvert - binaries available for Linux/Windows/Mac), there are some options.
Clip using a rectangle:
osmconvert input-data.osm.pbf -b=10.5,49,11.5,50 -o=my-output-file.pbf
(south-west and north-eastern corners, lng/lat)
Clip using a polygon:
osmconvert input-data.osm.pbf -B=clip-polygon.poly -o=my-output-file.pbf
The '.poly' file is in the Osmosis polygon filter file format (http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format), that page shows you how to create one manually or there are some suggested tools for converting from other formats.