One of your layers (the coastline) is a polyline layer. You are probably trying to clip with it and the tool would not work.
However, it looks like your coastline forms a closed shape and probably could be converted to a polygon feature class if you really want to clip with it. Feature To Polygon tool might help to convert it.
EDIT (thanks to Paul): One could use a polyline as clipping shape, but only to clip polylines and points. This tool can clip with features that have equal or higher geometry dimension value than the features being clipped (dimension is 0 for points, 1 for lines, 2 for polygons). For more details see the link in Paul's comment.