There doesn't seem to be an option to remove it via the Python (2.6 in my case) installer. It's pure python so there's no issue removing it, make sure you have set another application as the default .py/.pyw editor though or you might not be able to right click and edit scripts in ArcToolbox.
Just delete (or rename) PYTHONPATH\Lib\idlelib then delete the context menu registry entries via regedit or the reg file below:
Windows Registry Editor Version 5.00
[-HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE]
[-HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE]
[-HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.File\shell\Edit with IDLE]
[-HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.NoConFile\shell\Edit with IDLE]
[-HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\D3231384301F2FF4C94E56751A9AB0E5]
If you ever want to add it back:
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE]
[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE\command]
@="\"C:\\Python26\\pythonw.exe\" \"C:\\Python26\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""
[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE]
[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE\command]
@="\"C:\\Python26\\pythonw.exe\" \"C:\\Python26\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.File\shell\Edit with IDLE]
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.File\shell\Edit with IDLE\command]
@="\"C:\\Python26\\pythonw.exe\" \"C:\\Python26\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.NoConFile\shell\Edit with IDLE]
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.NoConFile\shell\Edit with IDLE\command]
@="\"C:\\Python26\\pythonw.exe\" \"C:\\Python26\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\D3231384301F2FF4C94E56751A9AB0E5]
"02FC15168DB032044A0AA668CDEF855E"="02:\\Software\\Classes\\Python.File\\shell\\Edit with IDLE\\command\\"