I want to transform WGS to MRT (GCS for Malaysia) using QGIS. Is "save as" to a new CRS the correct method? With ArcGIS, I can create transformation method to do transformation but I'm not sure in QGIS. If it is possible, how? And what is the string for xyz rotation, xyz translation, scale factor?
"Save As" to a new CRS is the correct method, however there are "better" ways to do this (Save As) operation in my experience. If I want to be 100% certain that transformations will happen accurately, I change my project CRS to the CRS that I want my export/transformation to be before doing the transformation.
If you already have some shapefiles or whatever that are in the MRT format, I'd load one of those, then right click on that one and "set project CRS to layer CRS" or something like that. Then you right click on the layer you want to reproject and 'save as', as you said, and set the CRS to "Project CRS".
qgis is really good at this stuff you should be fine.
QGIS uses EPSG codes and Proj.4 strings to read and write to spatial data. More info about the Proj4 parameters here. It is possible to create your own custom CRS in QGIS by either doing what boulder_ruby has described. Or you may edit a Proj4 string and then save it within QGIS with a custom name so you can reference it later when setting your project CRS or doing "save as" to transform your data.
also see this post on creating a custom CRS in QGIS.
hope this helps!