Fujitsu’s Fugaku supercomputer may not hold the title of the world’s fastest according to the latest Top 500 supercomputer rankings, but it remains a powerhouse for diverse applications, especially in artificial intelligence. This week, Fujitsu unveiled the Fugaku-LLM, a robust large language model equipped with sophisticated Japanese language processing features. The Fugaku-LLM is tailored for a broad range of uses, from academic research to commercial deployment, showcasing the versatile capabilities of the A64FX processor in handling complex AI tasks.
Fujitsu’s Fugaku-LLM, a sophisticated large language model, was developed using an impressive 380 billion tokens across 13,824 nodes of the Fugaku supercomputer. The supercomputer’s A64FX processor, which supports FP64, FP32, FP16, and INT8 modes, is ideal for diverse applications spanning AI and traditional supercomputing tasks. The training process for Fugaku-LLM leveraged distributed parallel learning techniques, specifically optimized to harness the full potential of the supercomputer’s architecture and its Tofu Interconnect D system. This approach maximizes efficiency and performance in processing complex computational tasks.
Fujitsu’s Fugaku-LLM, equipped with 13 billion parameters, may seem modest when compared to GPT-4’s massive 175 billion parameters, but it stands as the largest LLM trained in Japan. According to Fujitsu, this model is designed for efficiency, requiring significantly less computational power for inference, making it highly suitable for businesses and researchers across Japan. The model was trained on a diverse dataset comprising approximately 60% Japanese content, with the remaining 40% consisting of English, mathematics, and code. This balanced dataset ensures that Fugaku-LLM is well-adapted for a variety of linguistic and technical tasks.
This extensive Japanese-centric training sets it apart from other Japanese models that were trained primarily on English datasets. As a result, Fugaku-LLM boasts superior proficiency in Japanese, achieving an average score of 5.5 on the Japanese MT-Bench, the top score among openly available models trained with original data from Japan. It particularly excels in humanities and social sciences, achieving an impressive benchmark score of 9.18, according to Fujitsu.
