While an NDWI image is a single-band image, it is the product of subtracting, adding and multiplying multiple bands together--in the McFeeter's case, the values of the Green and Near Infrared (NIR) bands from a previous image are used to calculate a numerical value that is then displayed in a NDWI image (where each pixel value is between -1 and 1).
Your single-band ML classification may have been "better" (or more accurate) because the NDWI version of the image is inherently "classified" a little bit already-- pixels containing water have a drastically different value than dryer things like buildings or dirt, so it is easier for the ML tool to distinguish between the two because ML classifies based on pixel value(s).
As for why breaking waves were classified as vegetation in your RapidEye 5-band image: this may have to do with how well you drew training samples to tell ML what vegetation looks like. The rough surface of the waves may have caused the RGB, Red Edge, and NIR values of the breaking waves pixels to be closer in value to pixels you trained the classifier to recognize as vegetation. When drawing training samples, make sure you're capturing all the variation in the image (i.e., make sure some pixels within these wave areas are assigned as part of the "water" training set). This article gives some helpful tactics when creating training samples; it's written for ArcGIS Desktop but the theory is the same.