Author: yodi

Categories
LLM

Fix VLLM Torch libnvJitLink.so.12 issue

Installation for latest VLLM or SGLANG pip install vllm -U trigger error like below 

ImportError: python3.11/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12

To solve this problem is uninstall the torch packages and downgrade to 2.60 with cu121. I’m using nvcc version 12.2 and NVIDIA driver 12.7

pip uninstall torch torchvision torchaudio -y

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
Categories
Machine Learning

Install P2P Dual RTX 4090 Ubuntu 24.04

A good news, we can enable P2P for Dual RTX 4090 or more. When running simpleP2P script, we will got this results. Don’t worry, next is how to enable it easily!

[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : No
> Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA GeForce RTX 4090 (GPU0) : No
Two or more GPUs with Peer-to-Peer access capability are required for ./simpleP2P.
Peer to Peer access is not available amongst GPUs in the system, waiving test.

First thing, make sure to enable Resize BAR and disable IOMMU in BIOS (I’m using ASUS WRX80SAGE).

Next, uninstall all nvidia drivers (*yes, thats right!)

# Uninstall all nvidia
sudo apt-get --purge remove "*nvidia*"
sudo apt-get --purge remove "*cuda*" "*cudnn*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" "*libnccl*"

# disable iommu
ll /sys/class/iommu/

# install dependencies
sudo apt install git cmake

# reboot
sudo reboot
Continue reading “Install P2P Dual RTX 4090 Ubuntu 24.04”
Categories
LLM

Solve Unsloth AttributeError: ‘NoneType’ object has no attribute ‘attn_bias’

I got this error when running SFT Training using Unsloth

trainer.py:3731, in Trainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
   3729         loss_kwargs["num_items_in_batch"] = num_items_in_batch
   3730     inputs = {**inputs, **loss_kwargs}
-> 3731 outputs = model(**inputs)
   3732 # Save past state if it exists
...
    198     causal_mask = xformers.attn_bias.BlockDiagonalCausalMask\
    199         .from_seqlens([q_len]*bsz)\
    200         .make_local_attention(window_size = sliding_window)

AttributeError: 'NoneType' object has no attribute 'attn_bias'

The solution is 

pip install pip3-autoremove
pip-autoremove torch torchvision torchaudio -y 
pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121
pip unsloth
Categories
Tensorflow

Solve GPU not detected in Tensorflow 2

When you running this code, your Nvidia GPU is not detected 

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

[]

The solution is 

pip install "tensorflow[and-cuda]==2.15.1"
Categories
Ubuntu

Fix headphone bluetooth keep disconnecting Ubuntu

After applying Pipewire with AAC support in Ubuntu 24.04, I got problem where my Bose QuiteComfort is repeatingly disconnecting. The log keep saying input failed to connect

grep blue /var/log/syslog

I did several things like re-install packages and configure the bluetooth /etc/bluetooth/main.conf with AutoEnable=true and ControllerMode = bredr still not working. Also edit /etc/bluetooth/input.conf with `UserspaceHID=true`.

Turns out this solution at https://knowledgebase.frame.work/ubuntu-bluetooth-S1PGxfho works well!

sudo rm -r /var/lib/bluetooth/
rm ~/.config/pulse/* 
sudo apt reinstall --purge bluez gnome-bluetooth
systemctl restart bluetooth

This will remove your headphones from bluetooth (*reset). Then change your headphones mode into pairing mode.

Then do this to enter the bluetooth terminal

bluetoothctl

Then lets find and pair

power on
scan on
pair Device ID
devices
trust ID

If any notification asking for authorization, then choose “yes”. This is usually causing the problem where paired bluetooth is not authorized. next time its connected, its get problem!

Categories
Ubuntu

Enable AAC A2DP codec Ubuntu 24.04

I’m using Bose QuietComfort Ultra and its have capabilities to support multiple codecs. Connecting into Ubuntu, currently its only support for SBC-XQ and SBC. To unlock the lossless compression on AAC in Ubuntu 24.04 we need to do several things

  1. Install pipewire with AAC patch
  2. Switch from PulseAudio to Pipewire
  3. Enable it from the sound configuration

Lets do it

First, enable this PPA https://launchpad.net/%7Eaglasgall/+archive/ubuntu/pipewire-extra-bt-codecs

sudo add-apt-repository ppa:aglasgall/pipewire-extra-bt-codecs
sudo apt update

Then, re-install pipewire

sudo apt remove pulseaudio-module-bluetooth
sudo apt install --reinstall pipewire-audio-client-libraries libspa-0.2-bluetooth libspa-0.2-jack pulseaudio-utils

Next, we configure it

sudo apt install pipewire-audio-client-libraries
sudo apt install pipewire-media-session- wireplumber
systemctl --user --now enable wireplumber.service

sudo apt install libldacbt-{abr,enc}2 libspa-0.2-bluetooth pulseaudio-module-bluetooth-
sudo cp /usr/share/doc/pipewire/examples/ld.so.conf.d/pipewire-jack-*.conf /etc/ld.so.conf.d/

sudo ldconfig

Follow the PPA gist : https://gist.github.com/the-spyke/2de98b22ff4f978ebf0650c90e82027e#gistcomment-4320194

Now enable and verify

systemctl --user --now enable wireplumber.service
pactl info
Server String: /run/user/1000/pulse/native
Library Protocol Version: 35
Server Protocol Version: 35
....
Server Name: PulseAudio (on PipeWire 1.0.5) <------ HERE
Server Version: 15.0.0
Default Sample Specification: float32le 2ch 48000Hz
Default Channel Map: front-left,front-right
Default Sink: bluez_output.AC_BF_71_CF_15_96.1
Default Source: alsa_input.usb-

If Bluetooth re-connection has problem like its rejected or stop working after connected at fresh start, this is the solution

First un-pair the device then

sudo systemctl stop bluetooth
sudo rm -rf /var/lib/bluetooth/*
sudo systemctl start bluetooth
sudo reboot

Pair it again. Voila Done!

Categories
Tensorflow

Solve TF-TRT Warning: Could not find TensorRT

Installing TensorRT is tricky and shall be matched with the version of tensorflow. This is how you do it. First check the required version

python3 -c "import tensorflow.compiler as tf_cc; \
print(tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version())"

And then the result

2025-01-05 16:30:38.552990: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-05 16:30:38.576738: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-05 16:30:38.576767: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-05 16:30:38.577423: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-05 16:30:38.581769: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-05 16:30:39.023446: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(8, 6, 1)

This showing result for my tensorflow version, the tensor rt version is “(8.6.1)”. Next, go download tensorrt version eg: https://developer.nvidia.com/nvidia-tensorrt-8x-download

Extract it and go to “tensorrt/python” folder. Since i’m using mamba, I will go to my environment and install the tensor wheel packages based on my python version

(tf) ➜  python ls
tensorrt-8.6.1-cp310-none-linux_x86_64.whl           tensorrt_dispatch-8.6.1-cp37-none-linux_x86_64.whl
tensorrt-8.6.1-cp311-none-linux_x86_64.whl           tensorrt_dispatch-8.6.1-cp38-none-linux_x86_64.whl
tensorrt-8.6.1-cp36-none-linux_x86_64.whl            tensorrt_dispatch-8.6.1-cp39-none-linux_x86_64.whl
tensorrt-8.6.1-cp37-none-linux_x86_64.whl            tensorrt_lean-8.6.1-cp310-none-linux_x86_64.whl
tensorrt-8.6.1-cp38-none-linux_x86_64.whl            tensorrt_lean-8.6.1-cp311-none-linux_x86_64.whl
tensorrt-8.6.1-cp39-none-linux_x86_64.whl            tensorrt_lean-8.6.1-cp36-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp310-none-linux_x86_64.whl  tensorrt_lean-8.6.1-cp37-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp311-none-linux_x86_64.whl  tensorrt_lean-8.6.1-cp38-none-linux_x86_64.whl
tensorrt_dispatch-8.6.1-cp36-none-linux_x86_64.whl   tensorrt_lean-8.6.1-cp39-none-linux_x86_64.whl

And I installed

pip install tensorrt-8.6.1-cp311-none-linux_x86_64.whl

Done, now your tensor rt warning is gone!

If you are looking for Ubuntu 24.04 Nvidia deb packages

https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/

For CUDNN, NUMA and others, you can follow this

https://stackoverflow.com/questions/78894063/how-can-i-resolve-tensorflow-warnings-cudnn-cufft-cublas-and-numa

Categories
ML

Solve no matches found: tensorflow[and-cuda]

I found that Tensorflow 2.15 under python 3.9 not able to find my GPU. With nvcc already installed and using mamba, the solution is quite easy

python3 -m pip install 'tensorflow[and-cuda]'

Or 

pip install 'tensorflow[and-cuda]'

# with version
pip install 'tensorflow[and-cuda]==2.15.1'
Categories
Ubuntu

Fix missing screen recording in Ubuntu 24.04

First time installed Ubuntu, I can see the are two option for CTRL+Print Screen which are screenshot and screen recording. However after installing and upgrading other softwares, the screen recording is missing.

The quick solution apparently just simple as 

rm -rf ~/.cache/gstreamer-1.0/

sudo apt install --reinstall gstreamer1.0-pipewire
sudo apt install --reinstall ubuntu-desktop
sudo apt install gnome-session
Categories
Ubuntu

Install FFMPEG NVIDIA GPU in Ubuntu 24.10

Running FFMPEG with NVIDIA is blazing fast! Here is a quick way on how to install and make it run on NVIDIA and Ubuntu. First of all, if you have Ubuntu 24.10, you need to downgrade to 24.04. The reason because several libraries is not ready on the latest Ubuntu version.

Here is the steps

  1. Install required packages
sudo apt-get install build-essential yasm cmake libtool libc6 libc6-dev unzip wget libnuma1 libnuma-dev
sudo apt-get update -qq && sudo apt-get -y install \
  autoconf \
  automake \
  build-essential \
  cmake \
  git-core \
  libass-dev \
  libfreetype6-dev \
  libgnutls28-dev \
  libmp3lame-dev \
  libsdl2-dev \
  libtool \
  libva-dev \
  libvdpau-dev \
  libvorbis-dev \
  libxcb1-dev \
  libxcb-shm0-dev \
  libxcb-xfixes0-dev \
  meson \
  ninja-build \
  pkg-config \
  texinfo \
  wget \
  yasm \
  zlib1g-dev

2. Install FFMPEG needed libraries

sudo apt-get install nasm libx264-dev libx265-dev libnuma-dev libvpx-dev libfdk-aac-dev libopus-dev libdav1d-dev

3. Install NVCC version 12

https://developer.nvidia.com/cuda-downloads

4. Compile and Install

cd ~/Downloads/

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git

cd nv-codec-headers && sudo make install && cd -
git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg/

./configure --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --disable-static --enable-shared

make -j 8

sudo make install

Done!