I am trying to create a hillshade. I am using the DEM terrain model tool, choosing the right input file and write output, but keep getting this error message "ERROR 1: Band data type Byte not supported in GMT, giving up.". What does it mean? Is there a plug-in that I have missed to install or am I writing something wrong?
I'm guessing from the error you’re attempting to write out a GMT-compatible netCDF. According to the documentation (emphasis mine)
Newly created files (with a type of GMT) will always have units of "meters" for x, y and z but the x_range, y_range and z_range should be correct. Note that netCDF does not have an unsigned byte data type, so 8bit rasters will generally need to be converted to Int16 for export to GMT.
That suggests you need to use the -ot option with something like Int16.