An Engine Agnostic Approach to Machine Translation

boltsPeople ask me, “What MT tool do you sell?” They’re used to hearing from vendors that X machine translation engine is the best one and they want to hear how I will weigh in on the issue. My answer is disappointing to those who want to believe that with machine translation, one size fits all.

Alas, as a solutions supplier, not a technology vendor, my company has no particular bias when it comes to the technology. We are engine agnostic. What we care about is what meets our customers’ needs best.

In any case, we don’t see machine translation as a single tool at all. Rather, machine translation is an industrial process. And part of that process is determining what MT tool to use for a particular project.

Sorry I can’t tell you that there’s one magical engine out there that does everything equally well.

Of all the misconceptions about machine translation, I’d say the biggest misconception today is that there is one best engine for every use case. Much as we’d like this to be true, it just isn’t borne out empirically. I don’t believe the industry will advance with MT until the discussion gets back to nuts and bolts on this issue: starting with needs and determining what engine meets those needs best.

To give you some background, Lexcelera and LexWorks have nearly a combined decade of experience with MT deployments. We started in the Rules-Based (RBMT) environment because we needed strict terminology control on our earliest projects, which tended to be software documentation, On-Line Help and courseware. As we moved into corporate communications, we continued to need strict control on terminology in defined domains. As for languages, we started in French and Spanish, which work well in any approach.

In later years, we moved from RBMT to the Hybrid approach and began to train not only RBMT engines, but also statistical (SMT) engines too.  A funny thing happened as we trained new engines for German and Japanese, however. Contrary to our expectations, the Hybrid did not perform well with Japanese and German. Quality improved when we removed the SMT component and processed those languages using just an RBMT approach.

Enter customer support. There again we had a new use case. The user generated content (UGC) such as Forums and FAQs wasn’t as nicely authored as our documentation and help files had been. Our rules-based engines didn’t know what to make of misspellings and inaccurate grammar. Both the SMT and Hybrid engines we trained gave us better results. We became experts at training stand-alone SMT engines as well and were part of a consortium awarded a large EU grant to prepare SMT engines for user generated content.

Fast forward to today. Today we know that many factors that influence what engine is the best choice for what kind of content. Language combination and content type are two factors. How much data you have to train an engine is another (big) consideration. Shelf life of the translation and whether it needs to be real-time is something else to take into consideration. Quality needed is another factor to be weighed and whether post-editing is needed for publication quality (for more on this, see my article from January’s Multilingual Computing).

Post-Editor Shortage and MT

 

Nowadays our MT process for major new projects involves building three engines – SMT, RBMT and Hybrid – to test our assumptions. We scrupulously compare the output of all three trained engines – not just BLEU scores but GTM and post-editing productivity – to determine where performance is highest.

Ultimately this piloting process counts because, to say it simply, good quality MT ends up saving more time and money than poor quality MT. The better the quality, the less human input is needed in rework. And human rework is still the biggest cost of machine translation.

Where I would like to see our industry go is to a more objective place where we set out the criteria of what we need – content source, turnaround needed, languages, quality, integration, resources, and so on – and objectively measure which engine is suited. That way we can change the discussion in this industry away from the eternal question:”What is the best MT engine?” to “What is the best MT engine for this project?”