I don't think you can expect the data to be perfectly aligned. It depends on how and in which projection the GADM geometries have been created. Geoprocessing and projection conversions can introduce differences in accuracy and coordinate precision. Which eventually propagates to the situation you have now. A simple operation as rounding of the coordinates can transform a geometry.
Without knowing how GADM creates the geometries and what processing is applied, there is - I think - no way to know how to easily fix this.
If you want the geometries to align, why don't you use the OSM data itself? There are a lot of plugins in QGIS that make it easy to filter the data based on administrative boundaries and download it directly into QGIS.