To store the data, the first two alternatives to mention are PostGIS and SpatiaLite.
SpatiaLite is a SQLite database with spatial capabilities which
means it is file based, compact, and fast.
PostGIS is spatial capabilities on a PostgreSQL database. That
means it is very powerful with capacity to handle large data sets,
complex queries in a efficient way.
Before moving on to the desktop you should be aware that very many tasks can be done very efficiently directly in the database. Most things you usually do in ArcMap sitting waiting for different dialogs to pop up you will do many times faster directly with a few lines of SQL code. That means that you can also store what you did and do it again on another data set very easily by just saving your SQL code. Many of the PostGIS questions here are about how to write those queries, so you can get an idea about what can be done by browsing PostGIS tagged questions.
Then the desktop side. There are a lot of desktop solutions. I think it is time to admit that on the desktop side ESRI is good. As said before most of the tasks you do in ArcMap are better done deeper down in the database, but when it comes to the tasks where you need the desktop solution ArcMap is good. What I use is QGIS, and if I had a budget to spend I would move the ESRI license costs to support bug-fixing in QGIS. QGIS is great software that can do more or less everything that ArcMap can. But maybe one or two ArcInfo licenses (in money) are needed to polish it. You can also take a look at Open Jump, GvSIG, uDIG and more.
When it comes to web services you have MapServer, GeoServer, TinyOWS, and more.
GeoServer is probably the easiest to get started with since it
has a quite intuitive web interface. GeoServer can do most of the
things you want. It can serve WMS WFS Tiled services and so on. It is
all written in Java.
MapServer is written in C and I think it is fair to say it is the
king of WMS-services. At least it won last year's shootout on FOSS4G
conference (ESRI didn't dare to participate). Mapserver is configured
through a "map-file" and I don't know of any graphical interface. But
for serious wms-serving it is a very interesting alternative.
For editable wfs-services you have GeoServer mentioned before but
also TinyOWS. TinyOWS is a small wfs-server written in C. Lately
it has been more integrated in Mapserver since you can use the same
map-file for both.
Moving from the ESRI world I think opens another very important possibility for you. If you want, you are free to move away from the Windows platform too. You can put everything on any Linux-distro.
The software mentioned above together with Linux will reduce the need of hardware a lot.
An example of well working FOSS software is the map of Norgeskart, from the official map authority in Norway.
In bottom they use PostGIS serving Mapserver. Caching tiles in Geowebcache and using an OpenLayer based client on top. It is not flashy like a Silverlight solution but that is possible to do too.
They used ESRI software before but had to leave that when their needs increased in terms of capacity and speed.