If it using ellipsoidal option, does it mean measured using great circle formula in ellipsoid model?
Yes. I tested this by drawing a line with the measure tool from Nome, Alaska to Longyearbyen, Svalbard, with the project projection set to EPSG:3857 (Web Mercator). When using only two points, the ellipsoidal distance measured roughly 4,200km. The cartesian distance measured roughly 20,700km. When adding many intermediate points between the two, the cartesian distance remained the same, but the ellipsoidal distance increased to 6,500km. By adding the intermediate points, I forced the ellipsoidal distance to follow the Web Mercator version of the shortest line between the two, whereas without the intermediate points, the ellipsoidal distance calculation uses the great circle line instead.
If it using projection, what projection it used?
The projection you currently have set for QGIS display is what will be used for cartesian calculations. This can be checked in the very bottom-right corner of the QGIS window.
It's important to note that no map projection preserves accurate distances between all points. Equidistant projections preserve distance between the contact point (typically the center of the map) and any other point. Some other projections preserve accurate distance along the contact line(s). Therefore, the Cartesian mode of the measure line will be inaccurate in many cases at small scales (like a map of a nation or larger area), and Ellipsoidal should be preferred in most cases.