I had to run a different set of commands to try and solve my problem:
sudo rm -r /usr/local/bin/ogr*
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt purge libgdal20
sudo apt-get install gdal-bin postgis
Eventually, with apt list --installed | grep gdal
I realized that this was installing GDAL 2.4.0 while I was trying to get 3.5.2.
gdal-bin/oldoldstable,now 2.4.0+dfsg-1+deb10u1 amd64 [installed]
gdal-data/oldoldstable,now 2.4.0+dfsg-1+deb10u1 all [installed,automatic]
libgdal20/oldoldstable,now 2.4.0+dfsg-1+deb10u1 amd64 [installed,automatic]
This is because I was using Debian buster and it is oldold at this point, there's no more recent version of GDAL available for it (gdal-bin versions). So, the solution was to create an environment and manually install GDAL. Luckily, this can easily be done with conda:
conda create -n GDAL
conda activate GDAL
conda install -c conda-forge gdal