mbtiles is a format that embeds the tile images (typically 256 pixels high by 256 pixels wide) into a SQLite database. The format is fairly simple, and is really designed to avoid the need to copy many small files individually (which is often much slower than copying the same amount of data as a single file). This can also help tiles from becoming "lost", since you'll notice if the whole database doesn't make it.
There are three versions of mbtiles (1.0, 1.1 and 1.2), with an informal but descriptive specification given for each at https://github.com/mapbox/mbtiles-spec. 1.1 seems the most popular as of early 2013.
There is an vector extension, called UTFGrid. That was part of mbtiles version 1.1, but was split out in version 1.2.