The quickfix on how to uninstall current Cuda installed in Ubuntu not via software packages is using the uninstaller. For instance, I use cuda 11.8 and I need to downgrade it into 11.6.
So, I need to find the path and trigger this command
sudo /usr/local/cuda-11.8/bin/cuda-uninstaller
Last, we can clean-up entire cuda folder
sudo rm -rf /usr/local/cuda
If you have issue with GCC for the installed CUDA and need to downgrade or upgrade it, you can follow this
MAX_GCC_VERSION=11
sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo ln -s /usr/bin/gcc-$MAX_GCC_VERSION /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-$MAX_GCC_VERSION /usr/local/cuda/bin/g++