After running a GeoServer war file installation in Tomcat for some time in Windows Server 2008 x64, I've become increasingly unhappy with Tomcat's chronic memory issues and leaks. Now that the GeoServer Windows installer appears to run in a 64-bit environment as a Windows service when used in tandem with a 32-bit JVM, I'm looking into abandoning the war file in favor of the Windows installer. What are the pros and cons of the Windows installer versus the war file in a 64-bit Windows environment?
I would go with the binary (OS Independent) download rather than the installer. This will let you use a 64bit JDK (and probably JRE) while giving you more flexibility about your set up. The installer can get very confused about permissions (well it's windows that gets confused) and generally be more hassle to set up than you save.
However in either case you can just point JAVA_HOME to a 64 bit VM and it will work out fine.
We have attempted to install Geoserver using the windows installer but had issues getting it running. I never could get it configured properly. We then started using the WAR file with greater success.
Before giving up on the WAR file I would echo the previous posts. We ran Tomcat on a x64 bit windows 2008 machine and it was a problem. We had much better luck when I uninstalled 32 bit java and only installed the 64 bit flavor. Everything seems to like defaulting to 32 bit. I believe that in the past year tomcat has added support for 64 bit as well. I also noticed it is now up to version 7.
Lastly, I believe that Geoserver can run with Jetty, although I have never used it, it might be worth investigating.