NVIDIA introduces System Memory Fallback feature for Stable Diffusion

NVIDIA has a solution to Stable Diffusion maxing out the available memory

Drivers have a new feature called “System Memory Fallback”. 

The NVIDIA Control Panel may still look old, but it still continues to receive feature updates. In a recent driver release, NVIDIA introduced a new option in the Control Panel tailored for consumer GeForce graphics cards. This option is labeled as “CUDA – Sysmem Fallback Policy” and is specifically designed to address memory-intensive workloads, such as emerging generative AI programs.

This feature is primarily marketed for Stable Diffusion, a process that might demand up to 6GB of memory to be utilized simultaneously. However, since popular GPUs still feature maximum memory capacity between 6 and 8GB, the GPU memory might be still shared with other workloads preventing the full utilization by programs like Stable Diffusion. To address this, NVIDIA has implemented a new feature that prevents applications from exhausting GPU memory by efficiently switching to shared system memory.

NVIDIA Control Panel with Sysmem Fallback Policy, Source: VideoCardz

From a technical perspective, users have the flexibility to employ this feature for any application, including games. However, it may not be the best choice, as system memory is often notably slower than GPU memory, not to mention the necessary data transfer between the two.

The primary purpose of this feature is to avert potential crashes on graphics cards equipped with 6GB memory capacity. With the latest drivers, NVIDIA has thoughtfully incorporated an option to disable this fallback mechanism. Doing so can potentially enhance performance stability but does reintroduce the risk of depleting GPU memory.

How to enable or disable System Memory Fallback

1. Open NVIDIA Control Panel
2. Under 3D Settings, click Manage 3D Settings
3. Navigate to Program Settings tab
4. Select Stable Diffusion python executable from dropdown
5. Click on CUDA – Sysmem Fallback Policy and select Driver Default, Prefer No Sysmem Fallback or Prefer Sysmem Fallback.
6. Click Apply to confirm.
7. Restart Stable Diffusion if it’s already open.

Source: NVIDIA

Many thanks to @HardwareBuddy for the tip!