AudioCraft Fails On NVIDIA Blackwell: Kernel/Lib Mismatch

Hey guys! Today we're diving deep into a critical issue encountered while trying to run AudioCraft on the latest NVIDIA Blackwell architecture (GB10 / DGX Spark). It's a bit of a bumpy ride, but let's break it down so we can all understand what's happening and how we might fix it.

The Core Problem: Kernel and Library Mismatch

The main roadblock? A frustrating mismatch between the kernel and library versions. While we can compile and install Meta's AudioCraft library and its buddies on the NVIDIA Blackwell architecture (DGX Spark), we hit a wall at runtime. Think of it like trying to fit a square peg in a round hole – the pieces just don't align, leading to some serious resource penalties and a container that refuses to cooperate.

Environment Details

To give you the full picture, here's a quick rundown of the environment we're working with:

Outcome 1: Build Success (But at a Cost!)

Okay, so we did manage to get the Docker image to compile. But it was like climbing Mount Everest – we had to pull out all the stops!

The Build Stability Crisis

Initially, trying to run the Docker build without any limits was a disaster. We're talking complete loss of the host SSH connection and a mandatory hard reboot of the DGX Spark. Why? Total resource overload. The intensive C/C++ compilation steps for dependencies (like xformers) just swamped the Disk I/O and CPU, bringing everything to its knees.

Mitigation Strategy

To tame this beast, we had to get clever. We used tmux to isolate the Docker build and then capped the docker build command to just 2 cores. This confirmed our suspicions – the SSH crashes were due to resource saturation, not some low-priority process giving up the ghost. It was like performing surgery with a scalpel instead of a chainsaw. Think of it like carefully rationing resources in a survival game – every core counts!

Compilation Time: The Price of Victory

Even with these precautions, the compilation process – which included xformers and av source compilation for multiple architectures – took about 45 minutes. Yeah, you read that right. That's almost an entire episode of your favorite show! This lengthy compilation time throws a wrench in continuous integration, deployment, and even casual rebuilding. Nobody wants to wait 45 minutes every time they tweak something, especially when there's a risk of system instability during the I/O-heavy C/C++ linking phase.

Outcome 2: Runtime Failure – Kernel and Library Showdown

So, we conquered the build, but the war isn't over. Despite our hard-won compilation, the resulting container flat-out refuses to run during library initialization. It's like building a beautiful car only to find out the engine won't start. The execution log spills the beans, revealing two major conflicts that are holding us back:

A. XFORMERS/CUDA Version Mismatch

First up, we have an xformers warning that's pretty telling:

WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.9.0a0+50eac811a6.nv25.09 with CUDA 1300 (you have 2.9.0+cpu)

Translation: The xformers library, while successfully compiled, created a binary that's not playing nice with the runtime environment. It expected a GPU-enabled PyTorch build, but the script is defaulting to CPU mode because the core GPU kernels for xformers couldn't load correctly for the Blackwell architecture. This is a double whammy: performance features are MIA, and we're still facing the primary error lurking below.

B. PyTorch/TorchVision Kernel Registration Error (The Showstopper)

Now for the main event: the container throws a fit because a standard library (torchvision) can't register its kernel operations with the custom PyTorch installation. This is the critical blocker preventing the model from even starting up.

RuntimeError: operator torchvision::nms does not exist
...
File "/usr/local/lib/python3.12/dist-packages/torchvision/__init__.py", line 10, in <module>
...
The above exception was the direct cause of the following exception:
ModuleNotFoundError: Could not import module 'T5EncoderModel'. Are this object's requirements defined correctly?

In Plain English: The custom PyTorch binary (nvcr.io/nvidia/pytorch:25.09-py3), while tailored for Blackwell, isn't jiving with the standard PyPI version of torchvision. This kernel registration failure is like a domino effect, preventing transformers from importing essential components and rendering the entire AudioCraft stack unusable on the DGX Spark. It's like trying to conduct an orchestra with half the instruments missing – the music just won't flow.

Root Cause Analysis and the Plea for Help

So, what's the underlying issue here? It boils down to this: the dependencies we need (like torchvision and xformers) lack robust, compatible binaries (wheels) for the cutting-edge Blackwell/sm_120 PyTorch build that NVIDIA's NGC container provides. It's like trying to build a Lego masterpiece with some crucial pieces missing – frustrating, to say the least.

The Request

We're putting out a call for help, with two potential solutions on the table:

a) Blackwell-Native Wheels

We're dreaming of official pre-compiled wheels (either on PyPI or hosted by NGC) for key dependencies (like a stable xformers version) that boast native NVIDIA Blackwell architecture (DGX Spark) support. This would be a game-changer, eliminating the need for the lengthy and risky host compilation.

b) Guidance on Base Image Migration

Alternatively, we'd love some guidance on a validated, recommended NGC Base Image tag. We need one that's known to offer stable, internally consistent versions of torch, torchvision, and torchaudio – all fully compatible with current PyPI dependencies. We're even willing to sacrifice a smidge of Blackwell-specific optimization if it means a stable and working environment.

We're super grateful for any assistance in getting AudioCraft fully performant and deployable on NVIDIA's latest and greatest hardware. Thanks for listening, and let's hope we can crack this nut together!