You may have a corrupt Python installation. At ArcGIS Pro 3.1, a clean install may have around 250 packages. More packages than that, particularly if you've done multiple changes between ArcGIS Pro versions, may support the notion of a corrupt Python installation.
As covered in their 2017 Esri User Conference presentation Troubleshooting Python Issues in ArcGIS Desktop (Slide/Page 11), Thomas Jensen and Andrew Johnson indicate this can be fixed by renaming the Python folder and then running a repair on the ArcGIS Pro installation.
Assuming your ArcGIS Pro installation has your Python folder at C:\Program Files\ArcGIS\Pro\bin\Python, rename the Python folder to something like xxPython. You may need to provide admin credentials to do the rename.
Next, from the Windows Start menu, search for and open Apps & features. Locate and click on ArcGIS Pro. Then click Modify.
Confirm you want to allow the app to make changes to your device.
On the ArcGIS Pro Setup window, select Repair, click Next, and then click Install.
Once the repair installation is complete, you should have a clean Python install. Do note that you will need to do a fresh clone of your default arcgispro-py3 environment in order to install the packages again. However, with any luck, the packages should now successfully install in your cloned environment. Otherwise, there are larger issues at play with your system.
Once you have confirmed the fresh Python install works properly and allows you to both clone environments and install packages in the cloned environments, you should be able to safely delete the old Python folder (the one we renamed to xxPython above).