After training using transformer, calling model.save_pretrained(path) trigger this error in Vertex AI Deep Learning VM. I’m using NVIDIA L4 instances and Jupyter Notebook.
This problem is not because transformers version after I tried several version.
This error happen because the model still in cuda GPU memory. To fix it, move the model to CPU first.
model.to('cpu')
model.save_pretrained('path')To solve this problem of running Docker with gpu “docker: Error response from daemon: could not select device driver “” with capabilities: [[gpu]]”, you need to install the nvidia container toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.listsudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart containerd
Reference:
Punica is very interesting project that showing running multiple LORAs model in single GPU. There are few things need to be done to make this project works in your local and avoiding issue like
Here are the steps
MAX_GCC_VERSION=10
sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-$MAX_GCC_VERSION $MAX_GCC_VERSION
sudo apt install g++3. Install the right torch version based on your CUDA version
pip install torch==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu1214. Build from source!
pip install ninja numpy torch
# Clone punica
git clone https://github.com/punica-ai/punica.git
cd punica
git submodule sync
git submodule update --init
# If you encouter problem while compilation, set TORCH_CUDA_ARCH_LIST to your CUDA architecture.
# I'm using RTX4090, so ADA is 8.9. Check your version
export TORCH_CUDA_ARCH_LIST="8.9"
# Build and install punica
pip install -v --no-build-isolation .Why build from source works? Because its required to compile a new CUDA kernel design SGMV
Installation for latest VLLM or SGLANG pip install vllm -U trigger error like below
ImportError: python3.11/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12To solve this problem is uninstall the torch packages and downgrade to 2.60 with cu121. I’m using nvcc version 12.2 and NVIDIA driver 12.7
pip uninstall torch torchvision torchaudio -y
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
A good news, we can enable P2P for Dual RTX 4090 or more. When running simpleP2P script, we will got this results. Don’t worry, next is how to enable it easily!
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2
Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : No
> Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA GeForce RTX 4090 (GPU0) : No
Two or more GPUs with Peer-to-Peer access capability are required for ./simpleP2P.
Peer to Peer access is not available amongst GPUs in the system, waiving test.
First thing, make sure to enable Resize BAR and disable IOMMU in BIOS (I’m using ASUS WRX80SAGE).
Next, uninstall all nvidia drivers (*yes, thats right!)
# Uninstall all nvidia
sudo apt-get --purge remove "*nvidia*"
sudo apt-get --purge remove "*cuda*" "*cudnn*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" "*libnccl*"
# disable iommu
ll /sys/class/iommu/
# install dependencies
sudo apt install git cmake
# reboot
sudo rebootI got this error when running SFT Training using Unsloth
trainer.py:3731, in Trainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
3729 loss_kwargs["num_items_in_batch"] = num_items_in_batch
3730 inputs = {**inputs, **loss_kwargs}
-> 3731 outputs = model(**inputs)
3732 # Save past state if it exists
...
198 causal_mask = xformers.attn_bias.BlockDiagonalCausalMask\
199 .from_seqlens([q_len]*bsz)\
200 .make_local_attention(window_size = sliding_window)
AttributeError: 'NoneType' object has no attribute 'attn_bias'The solution is
pip install pip3-autoremove
pip-autoremove torch torchvision torchaudio -y
pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121
pip unslothWhen you running this code, your Nvidia GPU is not detected
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
[]The solution is
pip install "tensorflow[and-cuda]==2.15.1"After applying Pipewire with AAC support in Ubuntu 24.04, I got problem where my Bose QuiteComfort is repeatingly disconnecting. The log keep saying input failed to connect
grep blue /var/log/syslogI did several things like re-install packages and configure the bluetooth /etc/bluetooth/main.conf with AutoEnable=true and ControllerMode = bredr still not working. Also edit /etc/bluetooth/input.conf with `.`UserspaceHID=true
Turns out this solution at https://knowledgebase.frame.work/ubuntu-bluetooth-S1PGxfho works well!
sudo rm -r /var/lib/bluetooth/
rm ~/.config/pulse/*
sudo apt reinstall --purge bluez gnome-bluetooth
systemctl restart bluetoothThis will remove your headphones from bluetooth (*reset). Then change your headphones mode into pairing mode.
Then do this to enter the bluetooth terminal
bluetoothctlThen lets find and pair
power on
scan on
pair Device ID
devices
trust IDIf any notification asking for authorization, then choose “yes”. This is usually causing the problem where paired bluetooth is not authorized. next time its connected, its get problem!
I’m using Bose QuietComfort Ultra and its have capabilities to support multiple codecs. Connecting into Ubuntu, currently its only support for SBC-XQ and SBC. To unlock the lossless compression on AAC in Ubuntu 24.04 we need to do several things
Lets do it
First, enable this PPA https://launchpad.net/%7Eaglasgall/+archive/ubuntu/pipewire-extra-bt-codecs
sudo add-apt-repository ppa:aglasgall/pipewire-extra-bt-codecs
sudo apt updateThen, re-install pipewire
sudo apt remove pulseaudio-module-bluetoothsudo apt install --reinstall pipewire-audio-client-libraries libspa-0.2-bluetooth libspa-0.2-jack pulseaudio-utilsNext, we configure it
sudo apt install pipewire-audio-client-libraries
sudo apt install pipewire-media-session- wireplumber
systemctl --user --now enable wireplumber.service
sudo apt install libldacbt-{abr,enc}2 libspa-0.2-bluetooth pulseaudio-module-bluetooth-sudo cp /usr/share/doc/pipewire/examples/ld.so.conf.d/pipewire-jack-*.conf /etc/ld.so.conf.d/
sudo ldconfigFollow the PPA gist : https://gist.github.com/the-spyke/2de98b22ff4f978ebf0650c90e82027e#gistcomment-4320194
Now enable and verify
systemctl --user --now enable wireplumber.service
pactl infoServer String: /run/user/1000/pulse/native
Library Protocol Version: 35
Server Protocol Version: 35
....
Server Name: PulseAudio (on PipeWire 1.0.5) <------ HERE
Server Version: 15.0.0
Default Sample Specification: float32le 2ch 48000Hz
Default Channel Map: front-left,front-right
Default Sink: bluez_output.AC_BF_71_CF_15_96.1
Default Source: alsa_input.usb-If Bluetooth re-connection has problem like its rejected or stop working after connected at fresh start, this is the solution
First un-pair the device then
sudo systemctl stop bluetooth
sudo rm -rf /var/lib/bluetooth/*
sudo systemctl start bluetooth
sudo rebootPair it again. Voila Done!
Installing TensorRT is tricky and shall be matched with the version of tensorflow. This is how you do it. First check the required version
python3 -c "import tensorflow.compiler as tf_cc; \
print(tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version())"And then the result
2025-01-05 16:30:38.552990: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-05 16:30:38.576738: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-05 16:30:38.576767: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-05 16:30:38.577423: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-05 16:30:38.581769: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-05 16:30:39.023446: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(8, 6, 1)This showing result for my tensorflow version, the tensor rt version is “(8.6.1)”. Next, go download tensorrt version eg: https://developer.nvidia.com/nvidia-tensorrt-8x-download
Extract it and go to “tensorrt/python” folder. Since i’m using mamba, I will go to my environment and install the tensor wheel packages based on my python version
(tf) ➜ python ls
tensorrt-8.6.1-cp310-none-linux_x86_64.whl tensorrt_dispatch-8.6.1-cp37-none-linux_x86_64.whl
tensorrt-8.6.1-cp311-none-linux_x86_64.whl tensorrt_dispatch-8.6.1-cp38-none-linux_x86_64.whl
tensorrt-8.6.1-cp36-none-linux_x86_64.whl tensorrt_dispatch-8.6.1-cp39-none-linux_x86_64.whl
tensorrt-8.6.1-cp37-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp310-none-linux_x86_64.whl
tensorrt-8.6.1-cp38-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp311-none-linux_x86_64.whl
tensorrt-8.6.1-cp39-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp36-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp310-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp37-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp311-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp38-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp36-none-linux_x86_64.whl tensorrt_lean-8.6.1-cp39-none-linux_x86_64.whl
And I installed
pip install tensorrt-8.6.1-cp311-none-linux_x86_64.whl Done, now your tensor rt warning is gone!
If you are looking for Ubuntu 24.04 Nvidia deb packages
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/
For CUDNN, NUMA and others, you can follow this