UPDATE: This issue has been addressed with the following commit to QGIS master branch. The fix may soon be backported to the 2.0.x branch as well.
Since the issue stems from how many files an application can have open at once on a Mac, try moving your data to a database instead.
You can import your shapefiles into Spatialite or PostgreSQL/PostGIS databases (db). Spatialite is file-based and built on top of SQLite, allowing for easier portability and offers many advantages over shapefiles. You could import all of your shapefiles into one Spatialite db file, thereby circumventing the Mac limit.
PostgreSQL/PostGIS is more complicated but worth the investment of time and effort, and is available via Mac installers from Kyngchaos.com.
NOTE (June, 2013): Spatialite 4 offers many of the features of PostgreSQL/PostGIS and seems to be quite a bit faster than version 3, but is not currently fully functional in the master branch of QGIS. There is an effort to have it ready for QGIS 2.0 release.