Are there any scientific approaches for defining the smallest recognizable object? For example, a lake has a complex shape, and at a certain scale it is impossible to distinguish its characteristic features and so it doesn't makes sense to display on the map. How to define this scale?
There is a good article on ESRI Blog: On map scale and raster resolution which answers your question. I will quote the rule here:
The rule is: divide the denominator of the map scale by 1,000 to get the detectable size in meters. The resolution is one half of this amount.”
So here is the appropriate resolution for a select set of map scales, as mentioned in the article:
There is something the US developed long ago, primarily for intelligence images I think, called the National Image Interpretability Rating Scales (NIIRS). While scale is a major part of NIIRS, the 'Interpretability' was a somewhat subjective base upon whether the interpreter could see certain features in the image. For example, NIIRS 2 - can detect large aircraft hangars, NIIRS 3 - can identify wing configurations on aircraft, NIIRS 4 - can identify fighter plane by type, etc.
Maybe search engine around and see what NIIRS stuff is out there.