The moment my laptop choked on a 7‑B model, I knew I needed a better setup. I’d been tinkering with text generation for months, and the lag was killing my flow. That frustration turned into a weekend project that still feels fresh every time I fire up the rig.
Why I decided to go local
I was tired of waiting on cloud credits that vanished as fast as my patience. The latency of a remote API made interactive debugging feel like a bad joke. Running everything on my desk promised instant feedback, and that alone was worth the hardware splurge.
Choosing the right hardware
My first stop was the GPU market, and I quickly learned that “more VRAM” isn’t the only metric that matters. I compared a 24 GB RTX 3090, a 48 GB RTX 4090, and an AMD Radeon 7900 XTX, looking at price per gigabyte, power draw, and driver stability. After a few spreadsheet rows, the 48 GB RTX 4090 won me over because it gave me headroom for the larger models I wanted to experiment with.
The GPU I ended up with
The 4090 arrived in a box that smelled faintly of plastic and optimism. I installed it into a Corsair 750 W PSU‑rated case, making sure the PCIe slot was reinforced—those massive cards can flex the motherboard if you’re not careful. The first time I powered it on, the fans spun up like a jet engine, and I felt a weird mix of excitement and dread.
Setting up the OS
I went with Ubuntu 22.04 LTS because its package manager plays nicely with the NVIDIA stack. A fresh install gave me a clean slate, and I kept the default GNOME desktop to avoid extra bloat. I disabled the automatic updates during the initial setup; I didn’t want a sudden kernel change to break my drivers mid‑experiment.
Installing drivers and CUDA
The NVIDIA driver version 560.68 was the one recommended for the 4090 at the time, so I followed the “add‑official‑repo” method and let apt handle the heavy lifting. After the driver, I installed CUDA 12.2 and cuDNN 8.9, making sure the symbolic links pointed to the correct directories. A quick `nvidia-smi` confirmed the GPU was recognized with 48 GB of memory and a compute capability of 8.9.
Getting the right Python environment
I’m a fan of `pyenv` because it lets me juggle multiple Python versions without polluting the system. I installed Python 3.11.4, then created a virtual environment named `localai`. Inside that env, I pinned `torch` to the CUDA‑12.2 build, which required a specific wheel from the PyPI index. The installation took a while, but the final `torch.cuda.is_available()` returned `True`, which felt like a small victory.
Picking a model and why
I started with LLaMA‑2‑13B because it’s large enough to be interesting but still fits comfortably in 48 GB when using 4‑bit quantization. I also grabbed a smaller 7‑B variant for quick tests, storing both in a dedicated `/mnt/models` directory on an NVMe drive that reads at 3.5 GB/s. The speed of that SSD shaved seconds off the model loading time, which matters when you’re iterating fast.
Downloading and managing weights
I used `huggingface-cli` to pull the model files, but I ran into a hiccup: the repo required authentication, and I’d forgotten to generate a token with `read` scope. After a quick fix on the website, the download resumed, and I ended up with roughly 25 GB of compressed weights. I unpacked them with `git lfs` and verified the checksum against the model card to avoid any silent corruption.
Running inference: first test
My first inference script was a thin wrapper around `transformers`’ `pipeline`. I fed the prompt “Write a short poem about sunrise in the style of Emily Dickinson,” and waited. The output appeared in about 2.3 seconds, which was a stark contrast to the 12‑second lag I’d seen on the cloud service. That speed made me grin like a kid with a new bike.
Tuning performance
I discovered that setting `torch.backends.cudnn.benchmark = True` gave a modest boost because cuDNN could pick optimal kernels for the fixed input size. I also experimented with `torch.compile` in “inductor” mode, which compiled the model graph into a more efficient representation. The compilation step added a few seconds upfront, but subsequent runs shaved off roughly 15 percent of the latency.
Dealing with memory limits
Even with 48 GB, the 13‑B model in FP16 still hovered near 38 GB when I added a 4‑bit quantizer. I tried to push the batch size to 4, but the GPU ran out of memory within the first few tokens. The solution was to enable gradient checkpointing, which trades compute for memory, and that let me squeeze in a batch size of 2 without crashing. It’s a trade‑off I accept because the extra throughput outweighs the extra compute cost on my desktop.
My honest mistake with batch size
I once set the batch size to 8, assuming the 48 GB would handle it. The script crashed after loading the model, throwing a `CUDA out of memory` error that left me staring at the terminal for a solid ten minutes. The mistake taught me to always check the actual memory footprint with `torch.cuda.memory_summary()` before scaling up.
Automation and scripts
To avoid repeating the same setup steps, I wrote a Bash wrapper that activates the `localai` environment, sets the necessary environment variables, and launches the Python script with `nohup`. I also added a tiny watchdog that restarts the inference server if it exits unexpectedly. The whole thing lives in a Git repo, so I can clone it on any new machine and be up and running in under five minutes.
Monitoring and profiling
I rely on `nvidia-smi` in watch mode to keep an eye on GPU utilization, but for deeper insight I use `nsight systems`. Running a profiling session while the model generated a 200‑token response revealed that the first few layers were under‑utilized due to data loading bottlenecks. I fixed that by pre‑loading the attention masks onto the GPU, which lifted the average utilization from 68 % to 84 %.
Future tweaks I’m eyeing
One area I’m still exploring is multi‑GPU inference with model parallelism. The 13‑B model could be split across two 24 GB cards, freeing up VRAM for larger batch sizes. I’ve also been dabbling with AMD ROCm to see if the newer Radeon cards can match NVIDIA’s performance for these workloads, but the driver ecosystem still feels a bit rough around the edges.
Closing thoughts
Running AI models locally feels like having a private laboratory where I can experiment at my own pace. The hardware cost was non‑trivial, but the freedom to tinker without waiting for a remote queue is priceless. Every time I fire up the rig and watch the GPU lights flicker, I’m reminded that the best learning happens when you roll up your sleeves and get your hands dirty.