Get in touch with us
We at accompio will be happy to help you.
21.02.2024
In Part 3 of the „Testing AI Systems“ series, things get practical. Tom puts various translators to a concrete test, defining test criteria, creating test cases, and finally executing them with ALM Octane.
The journey of AI testing continues. In this chapter, I will delve into the quality attributes that are in previous chapter Having been explored in detail, let's go a step further. In addition to the creation process of concrete test cases, I will also share my experiences on how I implemented and executed them in OpenText ALM Octane. The results of the test executions clearly show a winner – so who will win the race? Google Translate, DeepL, or Libre Translate?
Before test cases can be realised, it must first be established what is actually to be tested. If one thinks of a traditional translator like a dictionary, the translation should simply be easily understandable and precise. But if I now imagine that I have to translate an email from a language I don't know into German, then that's already a big challenge. Even with a suitable dictionary, I would have to look up the correct translation word for word. Differences in grammar and sentence structure between the languages are then another matter entirely.
A good AI translator should be able to understand the context of a sentence and translate it accordingly. The Accuracy and the Completeness Translation is paramount here. In addition to translating technical terms, it is particularly important that the translation is embedded in a grammatically sound and understandable context. The original meaning and nuances of the text should be retained as much as possible.
The Language diversity is crucial. Being able to translate each of the approximately 7,000 different languages in the world is a major challenge, even for a machine system. Nevertheless, AI translators should be capable of supporting a variety of languages as well as To automatically detect languages, to overcome language barriers.
Alongside language detection, AI translators should also Ambiguities recognise and treat appropriately. „I just understand train station“ is a prime example of when proverbs are translated word for word.
When creating the test cases, I focused on the Top 5 most spoken languages in the world., as well as German, focused. The reason for this selection is the broad coverage of the world's spoken languages. Since I myself am unfortunately only fluent in 3 of the 6 languages – German, English and French – I have come up with a different approach than quickly learning three new languages. A translator like Google Translate to use would certainly not be a good idea for creating reference texts for other systems, especially not when the translator itself is still being tested. Thus, after some research I came across a huge test set from Microsoft which was released at the end of November 2022 and includes test data for 128 different languages – including all the languages I looked at.
Although these test data have proven extremely useful, they should be used with caution. Especially with black-box systems such as DeepL It is not clear which data was actually used for the AI training. There is a possibility that precisely this publicly accessible information was used for it. In a subsequent test run, this would lead to False conclusions about the actual accuracy come. The systems would already know the data and one could not make a statement about how the system would perform in new unknown situations.
As already in first chapter mentioned, sufficiently large datasets are certainly suitable for using machine metrics for evaluation. Two scientists from Jordan have done exactly that, by, among other things, the translations from English to Arabic from Libre Translate compared to other tools Have. The paper shows that Libre Translate performs satisfactorily in a direct comparison with translations from other machine translation systems and is on a par with Google Translate. But can that be true, given how disappointing my initial impressions of Libre Translate were?
From one further study emerged that Journalistic writing presents the greatest challenge for the AI translators DeepL and Google Translate. The texts contained stylistic devices and other features that made an accurate and fluent translation difficult. For this reason, and the fact described above that Microsoft's data are publicly accessible, I have also incorporated various online newspaper articles from the New York Times and other current publications into the test cases.
I have translated the English texts from the online newspapers to the best of my ability and included them in my test cases. Further cases such as the correct Translation of technical terms (for example, “kidney failure” or “oesophagus”) or the correct use of gender-specific words such as job titles are also part of my tests. KI translators should particularly score well in context-dependent translations. I examined this characteristic more closely using German proverbs and ambiguous terms. Here, the KI systems should be able to translate the words not just to translate word for word mechanically. Nevertheless, checking the system output is not trivial. I have to analyse the context beforehand and then recognise it in the translation.
ALM Octane from OpenText is somewhat like the Swiss Army knife for quality assurance in software development. In the case of the AI translators, I primarily used the tool for, Test cases and test runs to document it, so as not to lose track and to ensure that everything runs smoothly. I was also able to realise the presentation of the test results very easily with it.
In the above example, I have created a test case to check the completeness of an AI system's translation. To do this, the tester first invokes the system to be tested, selects the language settings according to the specification, and copies the text to be translated into the system's input fields. Subsequently, he checks step by step, whether the translation returned by the system accurately reflects the actual content. If this is the case for every single step, the test is passed; otherwise, it is marked as „failed“. For this test case, it must be noted that the tester must absolutely be a person proficient in the German language, as the tester is given a certain degree of interpretative leeway in deciphering the translation.
In addition to the already extensively presented systems Libre Translate and DeepL, the systems Google Translate and Reverso examined more closely. The comparison with traditional dictionary translators was also always important, represented by LEO Dict and dict.cc.
In software testing, the terms „passed“ and „Failed“ fundamental for tracking the quality of the software being tested. Even if only a single step does not lead to the expected result during the execution of the tests, the entire test is considered to have failed. Otherwise, the test is marked as passed.
At this point, I made another distinction. Tests of systems that do not support AI-specific features were classified as „skippet“ marked. This applies particularly to test cases where systems are expected to deliver context-dependent translations. Thus, in the evaluation and Comparison of Systems in ALM Octane to better distinguish which systems fulfil AI-specific functional requirements. This includes, for example, the presence of feedback loops in the form of translation ratings, which use the output of an AI system and corresponding user actions to retrain and improve the models over time.
Looking at the dashboard figures, DeepL clearly highlights. My initial suspicion that Libre Translate wouldn't perform as well as the other systems has been confirmed by this – Libre Translate is at the bottom of the heap. However, it must be noted that I limited myself to a small test set in the draft of the test cases – predominantly, they were translations from German to English and vice versa.
For a more accurate evaluation of translation quality, machine metrics should be considered with a sufficiently large test set. Furthermore, my tests primarily consider functional features. Testing non-functional features should also be taken into account in a more detailed evaluation. These include, for example, the usability or performance of the systems. Ultimately, in my test environment None of the tested systems passed all tests.
I'm next chapter I am dedicating myself to Risk analysis Regarding AI translators. In doing so, I highlight possible risks arising from AI translators and also explain what solutions could play a role in addressing them.
Arrange an initial consultation
