Alex has pretty much covered it (as mkennedy did in a comment) and I've upvoted his answer, but I want to offer slightly different wording/take on the issue.
All data you can add to a map has coordinate values. However it doesn't necessarily have what those values represent defined. Is it feet or meters or degrees? What is the origin? These kinds of things are covered with the CRS definition, and you can change the definition without changing the values. This is actually one of the most common mistakes made in GIS - defining coordinates as something they aren't rather than re/un-projecting them to something else and actually changing the values. See Layers with same coordinate system should align/overlap but do not?
If not defined, the software does one of two things - displays those coordinate values using the current map/dataframe's CRS; or displays them using a default coordinate system, such as geographic WGS84. In a way, you're right. The data was already using the 'correct' coordinate system, it just wasn't labeled as such. If the data's coordinates hadn't matched the system in use, it wouldn't have aligned correctly. The software doesn't know the correct CRS, it's just coincidence. By setting or defining the CRS you give the software the other piece of information it's missing, what those coordinate values mean, and it can then automatically relate them to coordinate values in other systems.
It's important to note that just because it appears to align doesn't mean it actually does. Some coordinate systems have very subtle differences, perhaps only a few meters or even less. Never assume things are aligned unless you know what the CRS is supposed to be, and if it simply appears to be right keep the fact you're making an assumption in mind - particularly depending on your accuracy/precision tolerances and requirements.