The number 124 as a binary byte (eight digits) is 01111100. To represent an RGB triple of three bytes would require three times eight, or 24, such bits. By default,
r.composite reduces this to just 15 bits by discarding the least significant bits in each band. Thus, it trims 01111100 to 01111, which is 15. (Equivalently, it divides 124 by 8 and ignores the remainder.) Apparently these three five-bit results are concatenated in the order B, G, R to form a 15-bit number (representing values between 0 and 2^15-1 = 32767, which is small enough to keep the color table to a manageable size). These values therefore represent three image bands, each with just five bits of precision rather than the original eight.
In the example of the question, the calculations proceed like this:
Blue = 124 is converted to 124/8 = 15 (plus a neglected remainder of 4). In binary this is 01111.
Green = 124 is converted to 124/8 = 15 (plus a neglected remainder of 4). In binary this is 01111.
Red = 172 is converted to 172/8 = 21 (plus a neglected remainder of 4). In binary this is 10101.
The digits are concatenated into 01111 01111 10101. This 15-digit binary number represents the value 15861 = (15*32 + 15)*32 + 21.
r.composite may do more processing than this, and it may do it slightly differently depending on the options you supply, but these operations do show the basic way in which a byte can be converted to a five-bit value.
You can approximately reverse the procedure using successive divisions by 32:
15861/32 = 495 plus a remainder of 21. (Multiplied by 8, this remainder of 21 gives 168, which is only a little bit less than the original 172 for the red band.)
495/32 = 15 plus a remainder of 15. (Multiplied by 8, this remainder of 15 gives 120, which is only a little bit less than the original 124 for the green band.)
We are left with 15, which when multiplied by 8 gives 120, which is only a little bit less than the original 124 for the blue band.
r.composite manual page at https://grass.osgeo.org/grass72/manuals/r.composite.html.