I am working with ArcToolBox's tools and their python supports. I found there is some restriction on file size of shape files/feature classes etc. It makes me interested to know about is there any restriction with file size on these tools/python methods also? As an example, suppose I am using "Con" method of "Spatial Analyst Tools". Does this method has any size restriction? I read "tool help" document. But I found nothing about file size here.
This help topic discusses some of the raster format limitations for Spatial Analyst: Data formats supported by Spatial Analyst
While it is focused mainly on vector overlay operations, this blog post has many suggestions, including improvements made at 10.1 and 10.1 SP1. In particular, if you are working with very large datasets, then using 64-bit geoprocessing at 10.1 SP1 is recommended. Also notable is ESRI's stance (see comments section) on not providing specific numbers regarding internal limitations as there are (allegedly) far too many variables involved.
Beyond that, there is very little that you as a developer can do to ensure that your customers will not have problems processing very large datasets, just as they would without using your custom tool, especially if, as you say, you have to support ANY dataset in ANY format. All bets are off.