First off, as per John Gravois statement in the comments, I need to state that it is generally best practise to not over complicate a web-map with large numbers of layers for the users to have to interact with.
The first distinction you should make is separating your operational layers from your basemap layers.
Using Google Maps as an example, think of the aerials and the traditional map in the background as your basemaps. Within that traditional map, there are countless number of layers (forests, administrative boundaries, rivers, etc). By separating these layers out, you can make use of tiled map services to 'cook' them into a nice looking basemap layer that wont allow the end-user any ability to turn the sub layers on/off, but provides the user with a reference to the real world.
Then your left with operational layers. When you ask Google Maps for directions from A to B, the resulting route can be considered an operational layer for the user to interact with. When you ask google maps for pizza shops near you, the resulting pins on the map is an operational layer for you to get more information from.
Sounds like you are following best practise to create multiple web maps/apps for focussed applications, and using the applications in the ArcGIS LG Model is definitely going to save you some time.
But you still have the requirement to have a one stop shop for users to have access to all layers within the one application. Despite not being a fan, this is a common requirement, particularly in local government. It can be done, but some considerations need to be taken into account.
We did a project for a large LG client migrating them from ArcIMS to ArcGIS Server. They had an initial requirement to be able to switch between 900 layers in one web application. After spending some time educating them about tiled map services and splitting the layers into basemap/operational, we whittled this down to 300 layers (with a userbase of 1000 users). Still a big ask.
However, with some help from the Esri Capacity Planning Tool (and in house testing) we were able to meet the requirement and still have a reasonable performing application.
We served up a number of different basemaps. No strain on the server there.
After some testing (and this will depend on user workflows and the design of the end application) we realised it was actually better to host the rest of the layers in only a few dynamic map services as opposed to lots of separate services. This put less strain on the CPUs that ArcGIS Server was using. (You mention your using ArcGIS Online, but the same principle will still apply, AGO will respond slower for more complex requests).
We also knew that the majority of users only wanted a handful of layers turned on, so the defaults for the dynamic services was to have only a few layers visible. This meant that when users hit the application, it was still relatively quick for ArcGIS Server to draw image requests. For the users that wanted to cross-compare with the other zillion layers, they would have to manually turn on these other layers. It was found that the users wanted to have the layers on offer, but only really wanted a few of them turned on at any one time. As a result, it was rare for a user to turn on all layers (which would have resulted in slower responses from the server).
We actually took this further and identified that different groups within the organisation wanted only a few different layers turned on by default. With a login system, we were able to determine which group the user was from, and then present the dynamic service with only the handful of layers initially turned on that related to their group.
So, in a nutshell, if you really have to meet the requirement of a one-stop-shop for all layers in your GIS, it can be done, but I would suggest trying to discard as many non-operational layers as you can into basemaps (which you can cache) and then serve the rest up in as little dynamic services as possible and by default only have a handful of commonly used layers turned on.
I put together some guidelines in a document that might be of some interest to you.
Best Practise Guide for Dekho - Performance.
If you have access to ArcGIS for Server, take a look at Dynamic Workspaces.
You can create an application with the commonly used layers served up as dynamic/feature service(s). You can then add functionality to your application to allow end users to add in additional layers from a geodatabase via the dynamic workspace.
