Introducing SearchPioneer.Lingua

Russ Cam

12 August 2024

Share:

A fast and accurate language detection library for .NET

We are thrilled to announce SearchPioneer.Lingua, a natural language detection library for .NET. This is a .NET port of the wonderful JVM and Rust implementations of Lingua created by Peter M. Stahl.

We really wanted to name the package Lingua to align with the other implementations, but unfortunately it's already taken by an unlisted and no longer maintained package 🙁.

We'll refer to SearchPioneer.Lingua as simply Lingua throughout this article.

The task of Lingua is simple; it tells you which language some provided textual data is written in. This is very useful as a preprocessing step in a variety of applications that deal with natural language, including

• Text Classification
• Spell Checking
• Search / Information Retrieval

Language detection is often done as part of large machine learning frameworks or natural language processing (NLP) applications. There are some cases however where you may not want to use such large systems; one such case is in the query flow of a high traffic search system where the overall latency of each request is paramount, and every additional millisecond counts.

Lingua supports 79 languages, with a focus on quality over quantity. Languages supported are:

Getting started

Lingua doesn't need much configuration to use and yields pretty accurate results on both long and short text, even on single words and phrases. It draws on both rule-based and statistical methods but does not use any dictionaries of words. It does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

Getting started with SearchPioneer.Lingua is as simple as installing the nuget package:

dotnet add package SearchPioneer.Lingua

Then building a detector with the desired configuration, and using it

using Lingua;
using static Lingua.Language;

var detector = LanguageDetectorBuilder
    .FromLanguages(English, French, German, Spanish) // detect only these languages
    .Build();

var detectedLanguage = detector.DetectLanguageOf("languages are awesome");
Assert.Equal(English, detectedLanguage);

Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language; The word prologue for instance is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed probabilities for each possible language have to satisfy. This can be configured as follows:

using Lingua;
using static Lingua.Language;

var detector = LanguageDetectorBuilder
  .FromLanguages(English, French, German, Spanish)
  .WithMinimumRelativeDistance(0.9)
  .Build();

var detectedLanguage = detector.DetectLanguageOf("languages are awesome");
Assert.Equal(Unknown, detectedLanguage);

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high otherwise Unknown language will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible.

Confidence values

Sometimes you want to know the confidence values for each of the configured languages. Referring back to our search system example, we may have search indexes and fields for each language that the search system supports, and want to query those indexes where the query text confidence value is above a given threshold. The library supports this operation too:

using Lingua;
using static Lingua.Language;

var detector = LanguageDetectorBuilder
  .FromLanguages(English, French, German, Spanish)
  .Build();

var detectedLanguages = detector.ComputeLanguageConfidenceValues("languages are awesome")
  .Select(kv => (kv.Key, Math.Round(kv.Value, 2)))
  .ToArray();

Assert.Equal(new []
{
  (English, 0.93),
  (French, 0.04),
  (German, 0.02),
  (Spanish, 0.01)
}, detectedLanguages);

The return value is an ordered dictionary of languages, order by confidence value descending. In the example above, the dictionary is projected into an array of tuples ordered by their truncated confidence value descending. Each confidence value is a probability between 0.0 and 1.0 and the probabilities of all languages sums to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language and the other languages will receive a value of 0.0.

Eager loading versus lazy loading

By default, Lingua lazy-loads only those language models which are considered relevant by the rule-based filter engine. Sometimes it is more beneficial to preload all language models into memory on application startup. This can be achieved as follows:

using Lingua;

var detector = LanguageDetectorBuilder
  .FromAllLanguages()
  .WithPreloadedLanguageModels()
  .Build();

The language models loaded into memory are shared across all LanguageDetector instances in a thread-safe manner.

Low accuracy mode versus high accuracy mode

Lingua's high detection accuracy can come with a latency and memory trade-off; The large language models can consume sizeable amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:

using Lingua;

var detector = LanguageDetectorBuilder
  .FromAllLanguages()
  .WithLowAccuracyMode()
  .Build();

The downside of low accuracy mode is that detection accuracy for short texts of less than 120 characters drops significantly. However, detection accuracy for texts longer than 120 characters remains mostly unaffected.

An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages; when you have knowledge about the texts you want to classify, you can almost always rule out certain languages as impossible or unlikely to occur.

Accuracy

The source repository provides the ability to test the accuracy of Lingua or any other language detection library in .NET, using 1000 single words, 1000 word pairs and 1000 sentences extracted from the same dataset used for training, namely the Wortschatz corpora provided by Leipzig University.

Accuracy reports 🎯 are available in the repository. The following table summarizes the mean accuracy of Lingua in high accuracy mode, along with two other popular language detection libraries, LanguageDetection and NTextCat. The mean of means (Average Accuracy) is also included in each case.

Each language detection library is configured and tested with all the languages it supports:

• Lingua supports 79 languages
• LanguageDetection supports 49 languages
• NTextCat supports 15 languages

The number of languages configured should be taken into consideration when perusing the following table; language detection with more configured languages compared to fewer has a higher likelihood to be wrong when results are ambiguous, which is also more likely with shorter input text. As an example,

NTextCat scores 100% on Russian language on 1000 single words, whilst Lingua scores 76.5%. Lingua erroneously classifies some Russian single words in high accuracy mode as

• Ukrainian: 6.30%
• Bulgarian: 5.50%
• Serbian: 3.40%
• Belarusian: 3.30%
• Macedonian: 3.00%
• Mongolian: 1.10%
• Kazakh: 0.90%

i.e. a mixture of other Slavic languages and those that use Cyrillic script. Importantly, NTextCat doesn't support any of these arguably similar languages, so has a lower likelihood of being wrong than Lingua.

Lingua consistently performs better than the other libraries, even when configured with more languages.

How does it work?

Most language detectors typically rely on a probabilistic N-gram model, such as that described in the N-Gram-Based Text Categorization (Cavnar & Trenkle, 1994) paper. The model is trained using the character distribution of a specific training corpus. Most libraries limit themselves to using 3-grams (trigrams), which work well for identifying the language of longer text fragments composed of multiple sentences. However, when it comes to short phrases or single words, trigrams fall short. The shorter the text, the fewer N-grams are available, making the probability estimates less reliable. To address this, Lingua employs N-grams ranging in size from 1 to 5, significantly improving the accuracy of language prediction. In low accuracy mode, only N-grams of size 3 are employed whereas high accuracy mode can use the full range of N-gram sizes from 1 to 5.

In addition to a probabilistic N-gram model, Lingua also employs a rule engine. The engine first looks at the alphabet of the input text and looks for characters that are unique to specific languages. It does so by employing Unicode Script information and a lookup table of unique characters. For example, The character ß, or "Eszett" or "sharp S", is unique to the German language. If a language can be unambiguously identified by the rule engine, the more computationally expensive probabilistic model doesn't need to be used.

Benchmarks

A language detection library needs to be both accurate and fast. The source repository contains a benchmarking suite to compare latency and allocations using BenchmarkDotNet, and the latest benchmarks ⚡ are available in the repository.

Running the English single word, word pairs and sentence benchmarks on the following local setup

BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3958/23H2/2023Update/SunValley3)
13th Gen Intel Core i9-13900K, 1 CPU, 32 logical and 24 physical cores
.NET SDK 8.0.303
  [Host]   : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX2
  ShortRun : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX2

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3  

yields the following results

Lingua is generally the fastest in low accuracy mode, never using Generation 2 (Gen2) of the managed heap, mainly using Gen0. The NTextCat library allocates the least amount of memory, though looks to get much slower as the length of text increases. The LanguageDetection library is much slower and allocates more.

Lingua makes use of Span<T> and ref structs to limit allocations. With the proposal to allow an alternate lookup structure in .NET 9 to search dictionaries and hashsets, it'll be possible to reduce allocations further, making it possible to compare N-grams parsed from input text as spans against the language model dictionaries held in memory, rather than allocating substrings for each N-gram size.

Summary

SearchPioneer.Lingua offers a powerful, efficient, and accurate solution for language detection in .NET applications. We'll look to add new features in the future, such as add identifying multiple languages in mixed-language texts, something the Rust implementation supports. We encourage you to explore the repository, try out the library in your own projects, and contribute to its continued development.

A special thank you to Peter for releasing the original implementations as open source 🙏.