Your output image will have more pixels than the sum of your input images, but this does not explain the large difference. I suggest that you look at the characteristics of your images based on gdalinfo in order to see what compression is used and check that the extents are correct. (assuming your input images have the same size, it makes 20000 * 12000 pixels per input images, which is large for an image of 50 Mb, maybe you are crossing the extent of your coordinate system when you create the mosaic.) Then you should look at the pixel depth of your images : if your input were in Bytes, then you should keep bytes.
gdal_translate -of Gtiff -ot Byte -co COMPRESS=LZW test.vrt test.tif
Remark 1 : Converting your images to jpeg before building a vrt doesn't help (it will be uncompressed before the next step) and you might loose data.
Remark 2 : using a vrt is helpful: are you sure that you need GTiff ?
EDIT: There is no miracle with the size of your images, but you should use a tiled tif as output so that you can use the jpeg compression with your large data (-co TILED=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 ). If it remains too big, the only solution is then to use gdalwarp to resample at a lower resolution.