If you set the project and layer coordinate system, the scalebar will make sense. Can you share a data sample? Which coordinate systems are you using?
You say that "most of the layers do not seem to have a defined CRS". Therefore, in layer properties you have to tell QGIS which projection the layer data is in. If you set WGS84 there but the data is actually in projected coordinate system, all measurements will be screwed.
Did you get shapefiles without .prj file?