I have solved the problem!
When you go to your print composer, you add a new map as you normally would. Leave enough space for your zoomed in portion to be displayed.
Add a new, smaller map in the area where you want your zoomed in portion to be displayed. Within this window, zoom to whatever area you want.
In the "Item Properties" tab, scroll down to "Overview" and hit the green + symbol.
Set the map frame to "Map 0" (this should be the full extent base map you're working with).
Go back to your full extent base map (Map 0) and go to the "Item Properties" tab and go to "Overview." Hit the green + arrow.
Set the map frame to "Map 1" (this should be the zoomed in portion).
Set the symbology for your new overview! (Note: I was fiddling with it for a bit before I realized that the border style was set to "no pen" instead of solid line, which is what I wanted. Just pay attention to the details and you should be fine!)
This is how I've done it, but there may be an easier/better way. Don't hesitate to comment if you know a better way!