I would use a vector mask or shapefile if the raster image is geospatially defined as part of a map especially if the raster is georeferenced in world coordinates (lat,lon, WGS1984, etc so forth) simply because it's easier for me to take a polygon of let's say Denver, Colorado where I live and extract satellite data using that polygon either via clip or mask. If I was trying to take a polygon sample from let's say an image of a landscape or a water body that is not georeferenced I might use the roipoly option. I think both would work for many or most use cases but it's really a matter of what easier/more available. I can go get shapefiles or click a few times on a screen with my mouse and make one. I don't have to worry too much about the coordinates as long as we are in the correct projection. For non-georeferenced data maybe the latter option is more suitable.
Looking at MATLAB docs, you can see it focuses more on images in general and is not tailored specifically for geospatial data. If your raster is geospatially referenced, shapefiles are optimized for geospatial information. You don't need to define vertices and such because they are georeferenced.
I think this question is less of a technical problem and more the seeking of outside opinions. So, this is my opinion. Essentially, it depends on the type of data and the software you are working on the data with. For geospatial imagery I would use either 1) sage_cmd or sage python api to clip or mask by polygon the raster(s) in question. If I am working with non-geo-referenced images I would use the MATLAB option. I'm not even sure how you would use a shapefile with non-georeferenced data anyway without using GDAL, saga, and other tools to georeferece it beforehand.
If I have a client and we are working on clipping images that are not necessarily geospatial then I can avoid the overhead of the geospatial data and work with roipoly.
I guess one must ask the question, why would i or anyone for that matter use a shapefile for non-georeferenced data. If we can't answer that then we should choose our software accordingly.