The main issue is coping with browser performance, so I'm already only considering performance in Chrome and Firefox. Yes, I also understand browser performance is dependent on the PC performance but still, there are better and worse libraries.
I have experience with GMaps v3 but with so many markers it is very slow. Changing to GEarth plugin on the browser solves the problem. Currently I'm using OpenLayers serving from my GeoServer WMS.
I guess I need a JS lib like Openlayers tailored for very high performance, for example using WebGL to make use of the PC's GPU. I know openlayers v3 would have webgl but I really want to hear the feedback from experienced users for these kind of big data on a map.
Also it should be opensource and free to use and be able to connect to my WMS to serve the tiles, like openlayers does.
I also don't mind if it's not JS API but some plugin I have to install on my PC / browser, like GEarth, if that improves the performance radically.