The ghost in the machine – what edit distance reports can tell

A look at MT from a different angle: Given the choice, what do translators actually do when offered MT? And do their choice reflect the quality as measured via Edit Distance?

At Alpha we have conducted a large scale analysis of the actual use and quality of Machine Translation across clients from different domains (IT, Fashion, Automotive) and in more than 25 languages.
We use standard NMT engines from Google and deepL, and let the translators decide if and when they want to use the MT.
MT is presented in the grid in memoQ when there is no TM match better than 75%. But we have also set the Minimum threshold for showing TM hits to 60%, so translators have a choice between a low fuzzy match, a MT suggestion and translating from scratch (which would be the default if no MT is offered and threshold is set to 75%).

We have collected thousands of Edit distance reports (ED) from memoQ for selected clients and languages and analysed (amongst other things):
  • How big is the utilisation, i.e. the amount of words and segments where MT was actually used as a percentage of the number of words/segments available (all below 75% fuzzy)?
  • What does ED tell us about the quality for different domains and languages?
  • Given the choice between a low fuzzy and MT, what do translators chose if any?
  • What is the difference in ED between a low fuzzy (50-74%) and a selected MT suggestion?




 

Steen Kesmodel
Operations Research Manager, Alpha


I currently work as Operations Research Manager and Head of Translation at Alpha in Cambridge, where I started working as a translator 20 years ago. I hold degrees in philosophy and comparative literature, and before becoming a full-time translator I was a chef and had a catering company in New Zealand (highlight: cooking for the All Blacks’ annual gala dinner) and teaching literature part time at Auckland University.
At Alpha I helped introduce and deploy several memoQ servers and spend most of my working hours supporting translators and PMs, automating workflows, working with a programmer to make bespoke tools and plug-ins using the API and battling against Default settings in projects in general and QA and LQA in particular.