I would like to convert them to OSM file format (.osm).
There is NO way to export data to .osm format in QGIS.
The menu point you used is just to download OSM raw data, and export it to a spatialite database.
Alternatively, you can save to GPX format (waypoints and tracks) in EPSG:4326, and load the file in the JOSM editor. It can convert GPX layers to OSM format. But take care for existing data before uploading to the OSM server.