If you're talking about the icons inside the application itself, you can find many of them by doing a search for *.png in the application install directory. Of course what you may want to do with those icons will probably be covered by the terms of service you accept when you install the software.
If you're talking about the more 'marketing-like' icons you ...
There are tools in ArcGIS Desktop to handle that. On Esri's ArcGIS Server documentation page Delete a Service it says:
If you delete a cached service, the cache tiles, tiling scheme, and
folder structure are retained on disk. If you want to delete the cache
completely, use the Delete Map Server Cache tool. If you want to
delete the tiles but leave the ...