I’m using the latest DuckDB 0.10.0 and receive memory error when exporting database with Parquet Format. I did with CSV and its work fine.
Memory configuration also set like :
SET memory_limit = '50GB';
SET max_memory = '50GB';
PRAGMA memory_limit=50GB;This still trigger OOM. The only solution that works is
SET preserve_insertion_order = false;Hope this help you in solving memory error using DuckDB.
Tensorboard and VSCode is already well-integrated. However, there is a slightly problem when running it using the latest version.
Tensorboard is installed, prompted to install repeatedly. It keep re-appearing to ask installing Tensorboard session package with the same result :
Could not install tensorboard. If pip is not available, please use the package manager of your choice to manually install this library into your Python environmentApparently, the major culprit is VSCode using different Python interpreter than the selected kernel in notebook. In this case, I’m using Anaconda with specific environment that already have tensorboard installed. To solve this, the solution is very straightfoward.
This is the most frustated problem I have encountered when using Asus Pro WS WRX80E-SAGE SE WIFI Motherboard Pro WS WRX80E-SAGE SE WIFI. The issue appeared when I changed the BIOS settings to enable the “SR-IOV” feature with the hope of solving USB devices not detected and avoiding adding “pci=nommconf” in GRUB.
Once, I rebooted, it suddenly its showing AMI Megatrend where everything was being initialized properly, and the last message was “ensure to connect the 8-pin power please enter setup to recover bios setting fatal error”. There is BIOS page at beginning to press F2 or Del, however it was not responsive and back to AMI page.
When installing TFX, I received error pip install tfx raises ResolutionTooDeep. During installation, its going over multiple different version of packages.
To solve this problem, I created requirements.txt with option to install necessary packages or all-packages that produced using pip freeze.
There are three options: TFX 1.10, 1.13 and the latest TFX 1.14.0
All the packages installation can be found here :
https://github.com/yodiaditya/datascience/tree/main/tfx
I hope this help you to solve TFX pip installation issues!
I got this error when running Tensorflow successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node,. To solve it
for a in /sys/bus/pci/devices/*; do echo 0 | sudo tee -a $a/numa_node; doneThere is a quick way to install and run Tensorflow version 1 (eg: Tensorflow 1.5) in Ubuntu, RTX 4090 and CUDA 12. Here are quick step to do it.
Depending on your needs, you can setup this using anaconda
conda create -n 36 python=3.6
conda activate 362. Install NVIDIA python index
pip install --user nvidia-pyindex3. Install Tensorflow 1.5 both GPU and CPU support
pip install --user nvidia-tensorflow[horovod]
Voila! Now you can run Tensorflow 1 using CUDA 11 or 12 on Ubuntu without problem.
Another alternative is using NVIDIA NGC.
The problem when adjusting my dual RTX 4090 in Ubuntu Server 23.10 is, when running nvidia-settings its trigger error
ERROR: The control display is undefined; please run `nvidia-settings --help`
for usage information.I’ve been search on internet and not found a better solution, then I decided try to install X and attached, while running nvidia-setting. The result its works. Here are the step by step on how to enable GPU fan speed in Ubuntu Server 23.10
sudo apt install -y xorg xinit2. Give permission to run X for user
Edit the file and change console into anybody to give permission to running X
sudo vim /etc/X11/Xwrapper.configIf you don’t change this, it will trigger error
/usr/lib/xorg/Xorg.wrap: Only console users are allowed to run the X server# Update 27 Feb 2024
The Chrome flags to disable simplified bookmarks is gone in the latest version. If you are using Ubuntu, you can downgrade chrome or chromium in Snap
Here are the steps
snap revert chromiumThis will be rolling back to version Version 121.0.6167.160 (Official Build) snap (64-bit)
Which now, when you access chrome://flags it will have bookmark-flow features
2. Disable auto-update features
snap refresh --hold=foreveror
snap refresh --hold=<duration> <snap1> <snap2>This is annoying where the bookmark or star is automatically added. To remove this feature
chrome://flags/#simplified-bookmark-save-flowThe disable it!
If you got this error when running your notebook with warning
contains a task of very large size. The maximum recommended task size is 1000 KiBThis mean PySpark warning you to increase the partition or parallelism (and might memory as well).
Example code to configure it, where you can adjust based on your workstation memory. In my case, is 192GB is my max memory
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--driver-memory 192g --executor-memory 16g pyspark-shell'
# add this one in your spark configuration
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")Full implementation
import os
from pyspark import SparkContext
os.environ['PYSPARK_SUBMIT_ARGS'] = '--driver-memory 192g --executor-memory 16g --executor-cores 10 pyspark-shell'
os.environ['PYARROW_IGNORE_TIMEZONE'] = '1'
builder = SparkSession.builder
builder = builder.config("spark.driver.maxResultSize", "5G")
spark = builder.master("local[*]").appName("FMClassifier_MovieLens").getOrCreate()
spark.conf.set("spark.sql.analyzer.failAmbiguousSelfJoin", "false")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")Additional bonus description if you would like to increase the instances as well
spark = SparkSession.builder.config('spark.executor.instances', 4).getOrCreate()
spark.conf.set("spark.sql.analyzer.failAmbiguousSelfJoin", "false")To solve Driver or CUDA 11 installation error in Ubuntu 23.10, the answer is to ensure its using the compatible GCC version. By default installation, it will using GCC 13 which is not working when compiling CUDA or NVIDIA Drivers (required GCC 10). Installing CUDA 11 is important to run Tensorflow that haven’t fully adapted with CUDA 12.
Failed to verify gcc version. See log at /var/log/cuda-installer.log for details.First step to fixing this problem is to uninstall any nvidia and cuda installation made previously
sudo apt autoremove cuda* nvidia* --purgeNext, install GCC 10
MAX_GCC_VERSION=10
sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-$MAX_GCC_VERSION $MAX_GCC_VERSION
Next, choose the GCC 10 as default by running this command
sudo update-alternatives --config gccNow, you all set! You can start to do installation of CUDA 11 in Ubuntu 23.10. Make sure to un-check the driver installation part (where we will install it later)
sudo ./cuda_11.8.0_520.61.05_linux.runNext, I’m using Ubuntu NVIDIA default installation. So, I revert back the GCC to version 13, using the same command
sudo update-alternatives --config gccsudo apt install nvidia-driver-525You can repeat the process like CUDNN, TensorRT and others installation following my previous article here
Finally, make sure if anything broken with NVCC, is to switch the GCC version to 10, not 13.