Some terminology first to set the context:
Load balancing is a computer networking methodology to distribute workload across multiple computers wikipedia.
Distributed computing instead refers to a network of computers who interact with each other in order to achieve a common goal wikipedia.
The difference would be that in load balancing the requests are spread among the servers and each server is responsible for processing the requests it has ben assigned. Each server works as a normal server as we know it.
In distributed computing each single request is handled by one or more servers, each one carrying out a part of the work (say one rendering the lines, one the labels and then one one putting all the pieces together and delivering the response to the client). This of course requires special software and coding to divide, assign, coordinate and assemble the collective work. Also not all computations can be easily or efficiently distributed.
The latter would be very complicated to achieve for WMS, so I'll simply assume you are asking about the former.
Being WMS an HTTP-based protocol it is pretty easy to use a load balancer (lb in short) in front of a server farm to distribute the load among the servers. Most lb implementations will also failover requests from unresponsive/dead servers to healthy ones.
The setup would be fairly simple:
one (or more than one in a active/passive configuration for ha) load balancer: the job of the lb is to receive a request from a client and forward it to one of the servers in the server pool. How the server is chosen depends entirely on the type of load balancer you are using. Assuming all WMS requests are equal, that is they put the same load on the server, even an algorithm as simple as round-robin would work well. Most of the lbs support dead server detection, so that unresponsive servers are removed from the server pool automatically and rejoined when they become available again. The lb should also support for dynamic pool sizing, that is a pool that be grown or shrunk at runtime to account for effective load usage.
two or more servers: these servers make the server pool, that is the computation resource that will actually handle the load from all the clients. Depending on how the pool is implemented it may be possible to hot-add/remove servers to the pool depending on load factor. Tipically new servers are quickly provisioned to the pool in presence of load-spikes and deleted afterwards.
In implementing the server pool the following two points are key: how is the server pool going to be managed and how is configuration and input data made continuously and coherently available to the pool. The latter means that ALL the backend WMS servers must have access to the same data: in case of shapefiles this can be achieved by using a shared network storage like NFS (for the Unix world), a shared SAN volume (attention: requires a cluster-aware filesystem) or simply a local replica from a central repository (can be effectively achieved with rsync).
In case of GIS data stored in databases those are already accessibile from multiple clients, but one should be careful in making sure that the database does not become the performance bottleneck.
As of configuration this depends on the WMS software used: for mapserver it a merely a file copy that can be handled with one of the methods for the input data above. Geoserver requires some experimenting but it probably requires setting up a master database to hold the configuration.
As for managing the pool that is highly dependent on how and where you will run it. Amazon AWS for instance already provides much of the management infrastructure and would be a good starting point IMHO. On the other hand if you have access to IT resources inside your organization you can tap on those to provide/manage the server pool.
The load balancer instead is mostly an issue of simply picking one among the many Open Source and commercial available. On Amazon AWS the lb can be a managed service from Amazon. Other popular http load balancers are: Apache mod_proxy_balancer , NGINX HttpUpstream. For the sake of completeness I’ll mention Linux Virtual Server which is instead a generic TCP load balancer even though I wouldn;t recommend it for this kind of setup.
As for GPU processing the only WMS server that so far can be hardware accelerated is Mapserver. Mapserver has had experimental OpenGL support since version 6.0. This post seems to suggest that rendering speed can benefit greatly by OpenGL-backed rendering, but I haven’t picked up much chatter about it in the mapserver-users list.