This isn't an automated process and more cosmetic, but I think it will get you what you want (see my example below)
- Create a mask of the areas you wish not to show. In the example below, everything on the outside of the blue circle is the mask.
- In the layout view (ARCMAP), move the edges of the data frame so that its a little bit bigger than your area of interest (in my case, the data frame was a bit bigger than the blue circle)
- Add the required grid from the data frame properties.
- Convert the grid to a graphic.
- Cut and paste the grid graphic into the data frame by using the focus data frame button (located in the layout tool bar)
- Ungroup the grid graphics
- Select all the grid lines and convert them into features (Convert Graphics to Features located in the drawing tool bar)
- Select all the grid text and convert them into annotations (Convert Graphics to Features located in the drawing tool bar)
- Create a buffer around the mask, and clip the grid lines using the buffer shapefile.
- Re-position the annotations closer to the grid lines.
This may not be the best solution, but it will give you a rounded grid.
There is always a possibility that the grid will move while editing, so just keep checking the lines against the coordinates shown by ARCMAP.
This process took about 15mins.
Hope this helps.