You would have to be a on a fairly low-tech machine, or be zooming/panning via automation in order to have issues rendering only two XYZ layers. I suspect your issue is related to something else. If it was an actual issue, you could employ a bounds/extent trigger to stop rendering and halt network traffic for tiles that are out of view. That being said, OpenLayers does do this in the background for you.
Try creating a new OL3 map with just your two XYZ layers and see if you have the same issue. You can then use a browser profiler or network proxy (like Fiddler) to see where the performance issue might be.
Further to your question on GeoJSON v XYZ, if you decided to render GeoJSON, you're asking the browser engine to do a whole lot more than simply display an XYZ tile. Unless your data is very simple, then you're not going to get a performance boost from rendering the data yourself.