I realized I had omitted another good link for you. There is an interesting blog by one of the ESRI developers involved in the HPC stuff (Mansour Raad). I hadn't visited it for a while, but it turns out he actually wrote an article about invoking a Spark job from within ArcGIS for Desktop using ArcGIS Geoprocessing tools. There is code samples as well. I don't know if you have any experience with ArcGIS, and especially Python as being the scripting language of the ESRI Geoprocessing framework, but it does show some of the feasibility of the suggested work flow of my first post in relation to Spark.
See here for Mansour Raad's blog:
And another remark: be aware that loading the > 140 GB *.osm uncompressed XML file of the entire US using the Load OSM File tool, may take up to two weeks! I recently imported a 12 GB uncompressed file of Poland, and loaded that in an ESRI File Geodatabase. If I remember it well, the process took about 1-2 days... There is unfortunately quite a lot of processing needed to convert the data, especially when OSM multipolygon relations are involved. The tool does a pretty good job at it, but it takes time.
Files on Geofabrik are in zipped format, compression ratio is about 1:12.
You may consider to filter the *.osm files to cut back on the workload of importing. There are tools to get specific data from them, in your case maybe only the highways, e.g. OSMOSIS, but I have no experience with these:
Lastly, prepare yourself for the fact (but I guess you may already know this), that things like address information in OSM, isn't that well structured. There are many variations.