It sounds like your custom tool is expecting a dataset rather than a layer.
Consequently, when you drag and drop a dataset in ArcCatalog all is fine.
However, when you try to drag and drop a layer from the Table of Contents it rightly complains that it is not a dataset.
Try drag and drop of the dataset, which is the source of the same layer, from the Catalog window of ArcMap and I expect that will work.
To make your tool accept either a dataset or a layer take a look at the type of data element that parameter is based on in your model.
To more clearly appreciate the difference between a layer (listed in the Table of Contents of ArcMap) and a spatial dataset (listed in the Catalog window of ArcMap and in ArcCatalog) I recommend reviewing Difference between map layer and spatial dataset (e.g. feature class, shapefile)?