Pixels vs Coordinates
When I think Raster maps, my first thought is satellite imagery. Almost every pixel in a detailed satellite image of a urban area could contain unique information. A single tile in a web map (typically a variant of Mercator loosely referred to as "Spherical Mercator" or "Web Mercator" and supported by Google, Bing, Yahoo, OSM and ESRI)typically has 256 x 256 = 65,536 pixels, and each zoom level has (2^zoom * 2^zoom) tiles. When I think Vector, I think polygons and lines. For example, a shape file detailing zoning boundaries of an entire city (potentially millions of Raster tiles) area might only have 65,000 Vector shapes.
Accurate Scaling
It sounds like you (and probably most readers) already know the most obvious difference between raster fixed pixels and vector (coordinate maps). Vector drawings (and maps) can scale with a higher degree of fidelity than pixels because vector data contains coordinate patterns (points, polygons, lines etc) that can rendered relative to each other at different resolutions using simple formulas, while pixel resizing typically uses a smoothing algorithm that results in image artifacts.
Image Compression vs Structure Compression
In practice, most images don't have 100% unique pixels can be compressed into smaller data packets, and many vector files contain excess detail that is not needed at many low detail zoom levels. Image compression is a well known and very pretty efficient process and almost every coding library has built in classes to do this work. Vector coordinate compression, or "geometry simplification" is a bit less common (as GIS in general is a bit less common than general image manipulation). In my experience you will spend close to 0 time thinking about image compression (simply turn it off or on) and considerably more time thinking about spatial compression. Check out the Douglas Peucker Algorithm for examples, or just play around with QGIS and some Census boundary files.
Client vs Server Side Rendering
Eventually everything viewed on a computer is rendered into pixels on the screen at a particular resolution (ie zoom level). Often (especially on the web) the challenge is getting those pixels in front of users as efficiently as possible. The US Census Tract & Block group shape files are particularly interesting because they are just over the boundary of vector datasets that are 'too big' to render in a web browser as vector data. In, contrast US Counties can just barely be rendered in modern browsers as a vector download. While a US Census Block Group vector shape file would certainly be smaller than a raster tileset rendered to cover the entire US at multiple zoom levels, the Block Group Shape file is too large (close to 1GB) for a web browser to download in demand. Even if the web browser could download the file quickly, most web browsers (even using flash) are quite slow when rendering huge numbers of shapes. So, for viewing large vector datasets, you are often better off translating them into compressed images for transmission to the web browser.
Some Practical Examples
I answered a similar question a few days ago about rendering large datasets in google maps.
You can see the question and a detailed analysis of "best practice" as used by the NY Times and others today here.
A few years ago decided to transition away from flash heavy client side vector rendering towards server side vector rendering that delivers compressed image tiles to pure html & JavaScript. We have a map gallery with several versions of Html+Raster (Server Generated Image Tiles) and Flash+Vector (client side heavy rendering).