Linking an attribute table (such as an Excel file) to a raster is possible, but only if the raster is in integer form.
With integer rasters, the values for each cell are limited to, well, integers. This is in contrast to floating-point rasters, wherein cell values are represented with decimal numbers.
Since integer raster values are constrained to just integers, it becomes possible for such rasters to be designed with an internal attribute table (known as a raster VAT or a RAT, Raster Attribute Table). Such tables look and act just like the attribute tables found in vector data, including the presence of unique ID values. This means that external tables, such as your Excel file, can be linked to the raster. No need to polygonize your raster!
Unfortunately, QGIS does not yet support integer rasters*. However, there is a crowdfunding effort underway at Kartoza to implement integer raster support for QGIS. Send 'em a pledge!
See this link for more info on the difference between integer and floating-point rasters.
*To be specific, QGIS will load and display integers rasters, but the the raster's internal VAT will not be exposed.