NVIDIA's V100, An 8-Year Old GPU, Now Sells for $100 and Crushes Modern Consumer Cards in AI LLM Workloads

Key Points:

• The 8-year-old NVIDIA Tesla V100 GPU, originally priced over $10,000 but now available for around $100, outperforms newer GPUs like the 5-year-old RTX 3060 and 3-year-old RX 7800 XT in AI large language model (LLM) tasks, delivering higher token generation speeds and better power efficiency.
• The V100 features 5120 cores, 640 Tensor Cores, 16 or 32 GB of HBM2 memory, and a 250W TDP, making it highly efficient compared to modern GPUs that often exceed 1kW power consumption.
• Testing showed the V100 achieving 130 Tokens/s on a 20-billion parameter GPT model, surpassing the RX 7800 XT’s 90 Tokens/s and outperforming the RTX 3060 by 42% in token generation speed, with up to 41% better power efficiency under a 100W power limit.
• However, using the V100 in standard PCs requires additional hardware modifications such as an SXM to PCIe adapter and custom cooling solutions, which may limit its accessibility for general users.
• Despite these challenges, the V100 remains a cost-effective and powerful option for AI workloads, with the 32 GB variant offering further advantages for larger models; further testing by the tech outlet is planned.