Spectra Super Intelligence

Train LLMs on half the GPU hardware via frequency math

Hardware

Developer Tools

Artificial Intelligence

GitHub

Visit WebsiteSee on Product HuntGithub ⧉Twitter ⧉

Hey Product Hunt 👋 I’m Kanishk, founder of Spectra Labs. For the past few months, I’ve been obsessed with one question: Why are we still letting AdamW eat half of our GPU memory on numbers that barely move? THE DIRTY SECRET OF LLM TRAINING: Every time an engineer fine-tunes or pre-trains a model, their GPUs don't crash because the weights are too big. They crash because optimizer states hoard 16 bytes per parameter. On an 8x A100 or H100 cluster, that’s $20,000 to $40,000 per month burned purely on holding momentum buffers. Until now, your only options were: 1. Pay thousands more to rent a bigger cluster. 2. Cut micro-batches down to a crawl. 3. Use 8-bit quantization (bitsandbytes), which adds quantization noise and destabilizes loss curves. THE BREAKTHROUGH: FREQUENCY-DOMAIN MATH Everyone told us quantization was the only way. We disagreed. We asked: What if we treat gradients like high-dimensional audio signals and compress them in the frequency domain using Fourier transforms? By isolating the true directional momentum in low frequencies and filtering high-frequency batch noise, we achieved continuous, smooth updates with half the footprint. REAL HARDWARE BENCHMARKS: - A100 80GB at 8k Context (with FlashAttention-2): Peak memory delta plunged from 14.2 GB down to 8.9 GB (5.3 GB VRAM saved / 37.3% reduction) with zero OOM spikes in native bfloat16. - 12-Layer Transformer: Cut optimizer state footprint from 2,831 MB to 1,415 MB (Exact 50.0% reduction). - Zero Quantization Jitter: Continuous gradient trajectories with identical convergence. IT'S A 1-LINE DROP-IN REPLACEMENT: pip install git+https://github.com/SpectraSI/Spe... Python: from spectra import SpectraAdamW optimizer = SpectraAdamW(model.parameters(), lr=1e-4, compression_ratio=0.5) We are an official NVIDIA Inception member, and we are releasing the community optimizer 100% open-source under Apache-2.0 today. Benchmark it on your GPUs, try to break it, and tell us how much VRAM you save! GitHub: https://github.com/SpectraSI/Spe... Website: https://spectralabs.si