The specific condition (i.e., exceptions as named it) of this tool as connecting the variables is explained in [the respective help page][1] as follows:

> The Raster Calculator tool generally follows the standard connectivity
> behavior of models in ModelBuilder, with some exceptions resulting
> from the requirements of formulating a valid Map Algebra expression.
> These include the following: 
>
> * Variables are connected to the Raster
> Calculator tool **when they are selected from the list of variables**. All
> variables are automatically listed in the Variable list inside the
> tool. 
>
>* When a dataset or variable is used in the expression, a link
> between the variable and the tool will be created. If you remove the
> variable from the expression, the associated link between the variable
> and the tool will also be removed. 
>
>* If you **delete the connection** to a
> variable, the variable is **not removed** from the expression. 
>* You **should
> not rename a variable** that is connected to the tool, since the
> variable will not be renamed within the expression. If you do so, the
> expression will be invalid.


  [1]: http://desktop.arcgis.com/en/arcmap/10.5/tools/spatial-analyst-toolbox/raster-calculator.htm