I downloaded GADM administrative level 0 worldwide and it seems very simple file: 255 rows of polygons representing countries. But the size of the file is very large: >500 MB. Why it's happening? Usually table with 255 rows should be roughly few MB's.
The dataset is so large, because the borders are very detailed.
If you are looking for a worldwide overview, you better take the data from Natural Earth. They provide datasets suitable for different scales. http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip is the smallest one.
These 255 rows represent 255 countries and each row is not a single polygon rather it is a multipolygon. Multiplogon for each country is due to the presence of large number of disjoint coastal areas or in other words islands. If you check the number of points involved in creating China alone, it goes beyond 7,00,000 reaching 11MB.
You can check the size of each shapefile in diva-gis site.
You can reduce the size of these shapefiles in QGIS by using vector-> geometry tools ->simplify geometries.