There might be a mathematical solution, but I prefer to solve it using QGIS:
- Create a line from (179.9, 90) to (179.9, -90) using the lat/lon coordinate system
- Densify the line using 34 control point to get a node at every 5 degrees (or more if you like)
- Reproject the layer to Mollweide
- Do the same for -179.9 East
All points in Mollweide outside the lines have no valid lat/lon coordinates.
You could as well create a polygon from the two lines, reproject the whole to Mollweide, and crop the Mollweide layer (raster or vector) to that:
- Create a text file with the following content
1;POLYGON ((179.9 90, 179.9 -90, -179.9 -90, -179.9 90, 179.9 90))
Load it into QGIS as delimited text, semicolon as delimter, WGS84 or your custom latlon CRS
Save it as shapefile, add it to the canvas and delete the text layer
Densify the polygon with
Vector -> Geometry Tools -> densify geomteries
Change project CRS to your Mollweide projection
Save the polygon under a new file name in that projection, and delete the latlon layer
To clip a shapefile, use
Vector -> Geoprocessing Tools -> Clip. Both layers must have the same CRS.
To clip a raster, use
Raster -> Extraction -> Clipper. Both layers must have the same CRS.