GSMA Foundry, the industry association’s innovation arm, introduced an open-source initiative aimed at improving large language models (LLMs) for telecoms applications.
Dubbed the Open-Telco LLM Benchmarks, the initiative is intended to provide a new framework to assess AI models on capability, energy efficiency, and safety in real-world telecoms scenarios.
Described by the GSMA as an “open-source community”, the offering is backed by Khalifa University in Abu Dhabi, computer application developer Hugging Face, and non-profit coding hub Linux Foundation. Operators including Deutsche Telekom, LG Uplus and SK Telecom, as well as vendor Huawei are also supporting the launch.
As AI adoption in telecoms accelerates, the GSMA explained that existing LLMs struggle with troubleshooting, regulatory compliance, and handling technical telecoms knowledge. Tests revealed that the GPT-4 model scored below 75 per cent on a dataset designed to evaluate telecoms-specific industry knowledge of LLMs. In another case, GPT-4 also performed less than 40 per cent on a dataset assessing the model’s understanding of 3GPP standards.
“Today’s AI models struggle with telecom-specific queries, often producing inaccurate, misleading or impractical recommendations,”explained Louis Powell, head of AI initiatives at the GSMA.
The new benchmarks community will enable mobile operators, AI researchers and developers to contribute use cases, datasets, and AI models for evaluation. A standardised benchmarking system will allow the AI models to be tested against practical telecoms challenges, including domain expertise, mathematical reasoning, energy efficiency and safety. Benchmarks will be hosted on Hugging Face’s platform to promote transparency and engagement.
Powell added that by establishing benchmarks, the initiative aims to improve AI model performance while ensuring “AI in telecoms is safe, reliable and aligned with real-world operational needs”.
As the industry puts focus on ethical and sustainable AI in telecoms, the initiative aligns with the GSMA’s previous commitments. Last year, the trade body introduced the Responsible AI Maturity Roadmap, a framework designed to help mobile operators implement best-practice principles in AI development and deployment.
Source: Mobile World Live
