How can I setup a portable ArcGIS installation (ArcMap, Catalog) to run from a flash drive / USB stick?
I think there may be an existing solution but I didn't find it yet.
Can I make it myself, and if yes how?
Short Answer: You can't.
ArcGIS Desktop is built upon layers of Legacy Components, and it hooks deep into your Windows OS.
It is a mix of FORTRAN, COBOL, C++, COM components, ActiveX and .NET assemblies. It requires Windows Registry to save some of its configurations, and the various dlls need to be registered: Some with the GAC, others as command components in the registry.
It is not a portable App, and I don't see that changing anytime soon.
You can however run ArcGIS Explorer from a flash drive, and you can use that to connect to an entrprise (SDE) database or file geodatabase. If you use fgdb with layer files all of the symbology that you build and save for the layers in desktop will display in explorer. Also ArcMobile for windows can run from a flash drive.