As you correctly state, the 64 bit version allows for using more memory than the 32 bit version. One example where this is important is high resolution printing, there are real world examples of crashes with huge printings which are resolved by using the 64 bit version.
Compared to that, the 64 bit version has not really any downsides - apart from the fact that it only works on 64 bit machines. It might have a slightly increased memory usage but I have yet to find a scenario where this would even come close to outweigh its advantages. So while it's technically possible to use a 32 bit QGIS on a 64 bit operating system, there is no advantage in doing so (unless you require a 32 bit only driver like ms access, thanks @Techie_Gus). On the other hand, on a 32 bit operating system, a 64 bit QGIS cannot be started.
- If you have a 64 bit machine and operating system, go for the 64 bit version (hint: if your computer is not very old, it probably is a 64 bit system)
- If you do not have a 64 bit machine and operating system go for the 32 bit version (or get a new operating system and/or machine)