# data[].attributes.attachment_link.readable_text
The Coleman-Liau Index is one of the most commonly used readability formulas today. But where did it come from? How does the test work? And where is it most useful?

What is the Coleman-Liau Index?

The index was created by Meri Coleman and Ti L Liau in 1975.

It differs from formulas such as Flesch-Kincaid and Gunning Fog. These formulas involve counting the number of syllables per word and per sentence. Interestingly, the Coleman-Liau Index does not involve any counting of syllables.

Coleman and Liau argued that despite developments, syllable-counting techniques lacked accuracy.

This they saw as a key weakness of existing readability formula. For example, it is difficult for a machine to detect the syllabic difference between the words 'going' and 'boing'. Syllable counting by humans is also prone to error.

They also argued that syllable counting was time-consuming. This criticism has lost its edge in the digital age. Tools like ReadablePro can calculate the readability of your text instantly. But, Coleman and Liau's argument about the potential interpretative issues remains valid.

According to Coleman and Liau, "word length in letters is a better predictor of readability than word length in syllables." Further, designing a "mechanical device" that could count in this way would be easy and accurate.

How does the test work?

So, as we've said, the formula involves no syllable counting.

The building blocks used in the formula are:

The Coleman-Liau formula estimates the years of education the reader needs to understand the text. So, a score of 6 should be equated to 6th grade in the US schooling system, i.e. 11-12 year olds. If you're writing for the general public, you should aim for a grade level of around 8-10.

When is the test most useful?

The original claims that educational organizations would benefit from Coleman-Liau. It would provide them with an easily calculated, economical measure of readability. This suggestion is in line with the roots of readability, which were created to enable standardization of school books.

The Coleman-Liau Index is now widely used. As well as being used in schools, it has also been used for medical documents. It can also evaluate online review and be used in the translation sector.

The Coleman-Liau Index is also often used alongside other formulas. ReadablePro calculates it alongside other highly respected and validated formulas. From this, it calculates an average for its unique readability grade.

Another area where the Coleman-Liau holds particular value is in measuring readability of texts written in Western-European languages.

For these non-English texts, the Coleman-Liau provides a marker for comparison of ease of reading. So, by running the test on two different texts, e.g. two German newspapers, the test results would allow you to see which textbook is harder or easier than the other.

There are some limitations outside the English language. Importantly, the formula is standardized for English language texts. Other Western-European languages such as Spanish and German differ in syntax. In comparison to the English language, they also use more compounded words.

As a result, these languages have longer words for a phrase where the English language may have a sentence. This proves difficult for Coleman-Liau as it is used to evaluate the length of words.

For example, a German word which is compounded to indicate tense could be assumed to be difficult. In fact, the reader may not need the indicated level of education to understand this word.

So, for non-English language texts, the grade level is meaningless, ie the score will not reflect the grade level of the text analyzed. The value of Coleman-Liau Index for non-English texts then is in the comparison between texts rather than the meaning of the individual scores.

Another interesting avenue for the Coleman-Liau measure is in the study of law and courts. Here, the measure has been used to explore the readability of judicial opinions. These are legal opinions written by judges during a court case. They provide a decision reached to resolve a legal dispute.

Researchers report that when trying to pass a decision which is inconsistent with the majority, judges will strategically write less readable opinions.

This was based on the Coleman-Liau grade of over 500 randomly selected Supreme Court majority opinions. By making them harder to read, the judges help to protect the court from criticism.

Writing less readable opinions will, according to the study, discourage backlash.

The Coleman-Liau Index remains a steadfast player in readability. With its range of applications, and combined with the other formulas we use in ReadablePro, it can help you see when you're using too many long words.

Sources: researchgate.net, semanticscholar.org, faculty.cs.byu.edu, wiley.com, sciencedirect.com, anthology.aclweb.org, mortengoodwin.net, gvpt.umd.edu

# data[].attributes.attachment_link.description
The index was created by Meri Coleman and Ti L Liau in 1975. It differs from formulas such as Flesch-Kincaid and Gunning Fog. These formulas involve counting the number of syllables per word and per sentence. Interestingly, the Coleman-Liau Index does not involve any counting of syllables.

# data[].attributes.attachment_link.readable_text
The SMOG grade is a measure of readability that estimates the years of education needed to understand a piece of writing. SMOG is an acronym for Simple Measure of Gobbledygook.

SMOG is widely used, particularly for checking health messages. The SMOG grade yields a 0.985 correlation with a standard error of 1.5159 grades with the grades of readers who had 100% comprehension of test materials.

The formula for calculating the SMOG grade was developed by G. Harry McLaughlin as a more accurate and more easily calculated substitute for the Gunning fog index and published in 1969. To make calculating a text's readability as simple as possible an approximate formula was also given - count the words of three or more syllables in three 10-sentence samples, estimate the count's square root (from the nearest perfect square), and add 3.

A 2010 study published in the Journal of the Royal College of Physicians of Edinburgh stated that "SMOG should be the preferred measure of readability when evaluating consumer-oriented healthcare material." The study found that "The Flesch-Kincaid formula significantly underestimated reading difficulty compared with the gold standard SMOG formula."

Applying SMOG to other languages lacks statistical validity.

To calculate SMOG

1. Count a number of sentences (at least 30)

2. In those sentences, count the polysyllables (words of 3 or more syllables).

3. Calculate using

This version (sometimes called the SMOG Index) is more easily used for mental math:

1. Count the number of polysyllabic words in three samples of ten sentences each.

2. Take the square root of the nearest perfect square

3. Add 3

SMOG conversion tables compiled by Harold C. McGraw are slightly inaccurate because they are based on the approximate formula. Furthermore, tables for texts of fewer than 30 sentences are statistically invalid, because the formula was normed on 30-sentence samples.

# data[].attributes.attachment_link.description
The SMOG grade is a measure of readability that estimates the years of education needed to understand a piece of writing. SMOG is an acronym for Simple Measure of Gobbledygook. SMOG is widely used, particularly for checking health messages.

# data[].attributes.headline
And that made me search for other readability indexes

# data[].attributes.subject
And that made me search for other readability indexes

# data[].attributes.attachment_link.readable_text
FOG - indeks czytelności, który ma na celu określenie stopnia przystępności tekstu. Jego wartość oznacza liczbę lat edukacji potrzebnych do zrozumienia tekstu, np. wartość FOG=9 jakiegoś tekstu oznacza, że tekst ten będzie zrozumiały dla absolwentów gimnazjum i osoby o wykształceniu ponadgimnazjalnym.

Oryginalny indeks FOG został sformułowany dla języka angielskiego, ale istnieje też wariant dostosowany do specyfiki polszczyzny. W języku angielskim słowa trudne mają najczęściej 3 sylaby i więcej. W języku polskim trudne wyrazy to najczęściej wyrazy 4-sylabowe i dłuższe (tak uważają Pisarek, Seretny, i grupa autorów: Miodek, Maziarz, Piekot, Poprawa i Zarzeczny).

Przystępność języka jest jedną z podstawowych cech standardu "plain language" (prosty język).

Wzór na wartość FOG sformułowany został następująco:

Wynik testu można interpretować zgodnie z poniższą tabelą:

Wartość FOG

Interpretacja

1-6

język bardzo prosty, zrozumiały już dla uczniów szkoły podstawowej

7-9

język prosty, zrozumiały już dla uczniów gimnazjum

10-12

język dość prosty, zrozumiały już dla uczniów liceum

13-15

język dość trudny, zrozumiały dla studentów studiów licencjackich

16-17

język trudny, zrozumiały dla studentów studiów magisterskich

18 i więcej

język bardzo trudny, zrozumiały dla magistrów i osób z wyższym wykształceniem

Dziennik Fakt ma indeks czytelności FOG na poziomie około 7, tygodnik Newsweek około 12, zaś teksty prawne mają FOG na poziomie 20.

# data[].attributes.attachment_link.description
FOG - indeks czytelności, który ma na celu określenie stopnia przystępności tekstu. Jego wartość oznacza liczbę lat edukacji potrzebnych do zrozumienia tekstu, np. wartość FOG=9 jakiegoś tekstu oznacza, że tekst ten będzie zrozumiały dla absolwentów gimnazjum i osoby o wykształceniu ponadgimnazjalnym. Oryginalny indeks FOG został sformułowany dla języka angielskiego, ale istnieje też wariant dostosowany do specyfiki polszczyzny.

# data[].attributes.headline
I compared my text in two different online readability calculators to see if they were similar or not

# data[].attributes.subject
I compared my text in two different online readability calculators to see if they were similar or not

# data[].attributes.attachment_link.readable_text
Last Updated on

BLEU, or the Bilingual Evaluation Understudy, is a score for comparing a candidate translation of text to one or more reference translations.

Although developed for translation, it can be used to evaluate text generated for a suite of natural language processing tasks.

In this tutorial, you will discover the BLEU score for evaluating and scoring candidate text using the NLTK library in Python.

After completing this tutorial, you will know:

A gentle introduction to the BLEU score and an intuition for what is being calculated.

How you can calculate BLEU scores in Python using the NLTK library for sentences and documents.

How you can use a suite of small examples to develop an intuition for how differences between a candidate and reference text impact the final BLEU score.

Discover how to develop deep learning models for text classification, translation, photo captioning and more in my new book, with 30 step-by-step tutorials and full source code.

Let's get started.

May/2019: Updated to reflect changes to the API in NLTK 3.4.1+.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

1. Bilingual Evaluation Understudy Score

2. Calculate BLEU Scores

3. Cumulative and Individual BLEU Scores

4. Worked Examples

Need help with Deep Learning for Text Data?

Take my free 7-day email crash course now (with code).

Click to sign-up and also get a free PDF Ebook version of the course.
Start Your FREE Crash-Course Now

Bilingual Evaluation Understudy Score

The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric for evaluating a generated sentence to a reference sentence.

A perfect match results in a score of 1.0, whereas a perfect mismatch results in a score of 0.0.

The score was developed for evaluating the predictions made by automatic machine translation systems. It is not perfect, but does offer 5 compelling benefits:

It is quick and inexpensive to calculate.

It is easy to understand.

It is language independent.

It correlates highly with human evaluation.

It has been widely adopted.

The BLEU score was proposed by Kishore Papineni, et al. in their 2002 paper " BLEU: a Method for Automatic Evaluation of Machine Translation ".

The approach works by counting matching n-grams in the candidate translation to n-grams in the reference text, where 1-gram or unigram would be each token and a bigram comparison would be each word pair. The comparison is made regardless of word order.

The primary programming task for a BLEU implementor is to compare n-grams of the candidate with the n-grams of the reference translation and count the number of matches. These matches are position-independent. The more the matches, the better the candidate translation is.

- BLEU: a Method for Automatic Evaluation of Machine Translation, 2002.

The counting of matching n-grams is modified to ensure that it takes the occurrence of the words in the reference text into account, not rewarding a candidate translation that generates an abundance of reasonable words. This is referred to in the paper as modified n-gram precision.

Unfortunately, MT systems can overgenerate "reasonable" words, resulting in improbable, but high-precision, translations [...] Intuitively the problem is clear: a reference word should be considered exhausted after a matching candidate word is identified. We formalize this intuition as the modified unigram precision.

- BLEU: a Method for Automatic Evaluation of Machine Translation, 2002.

The score is for comparing sentences, but a modified version that normalizes n-grams by their occurrence is also proposed for better scoring blocks of multiple sentences.

We first compute the n-gram matches sentence by sentence. Next, we add the clipped n-gram counts for all the candidate sentences and divide by the number of candidate n-grams in the test corpus to compute a modified precision score, pn, for the entire test corpus.

- BLEU: a Method for Automatic Evaluation of Machine Translation, 2002.

A perfect score is not possible in practice as a translation would have to match the reference exactly. This is not even possible by human translators. The number and quality of the references used to calculate the BLEU score means that comparing scores across datasets can be troublesome.

The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. [...] on a test corpus of about 500 sentences (40 general news stories), a human translator scored 0.3468 against four references and scored 0.2571 against two references.

- BLEU: a Method for Automatic Evaluation of Machine Translation, 2002.

In addition to translation, we can use the BLEU score for other language generation problems with deep learning methods such as:

Language generation.

Image caption generation.

Text summarization.

Speech recognition.

And much more.

Calculate BLEU Scores

The Python Natural Language Toolkit library, or NLTK, provides an implementation of the BLEU score that you can use to evaluate your generated text against a reference.

Sentence BLEU Score

NLTK provides the sentence_bleu() function for evaluating a candidate sentence against one or more reference sentences.

The reference sentences must be provided as a list of sentences where each reference is a list of tokens. The candidate sentence is provided as a list of tokens. For example:

Running this example prints a perfect score as the candidate matches one of the references exactly.

Corpus BLEU Score

NLTK also provides a function called corpus_bleu() for calculating the BLEU score for multiple sentences such as a paragraph or a document.

The references must be specified as a list of documents where each document is a list of references and each alternative reference is a list of tokens, e.g. a list of lists of lists of tokens. The candidate documents must be specified as a list where each document is a list of tokens, e.g. a list of lists of tokens.

This is a little confusing; here is an example of two references for one document.

Running the example prints a perfect score as before.

Cumulative and Individual BLEU Scores

The BLEU score calculations in NLTK allow you to specify the weighting of different n-grams in the calculation of the BLEU score.

This gives you the flexibility to calculate different types of BLEU score, such as individual and cumulative n-gram scores.

Let's take a look.

Individual N-Gram Scores

An individual N-gram score is the evaluation of just matching grams of a specific order, such as single words (1-gram) or word pairs (2-gram or bigram).

The weights are specified as a tuple where each index refers to the gram order. To calculate the BLEU score only for 1-gram matches, you can specify a weight of 1 for 1-gram and 0 for 2, 3 and 4 (1, 0, 0, 0). For example:

Running this example prints a score of 0.5.

We can repeat this example for individual n-grams from 1 to 4 as follows:

Running the example gives the following results.

Although we can calculate the individual BLEU scores, this is not how the method was intended to be used and the scores do not carry a lot of meaning, or seem that interpretable.

Cumulative N-Gram Scores

Cumulative scores refer to the calculation of individual n-gram scores at all orders from 1 to n and weighting them by calculating the weighted geometric mean.

By default, the sentence_bleu() and corpus_bleu() scores calculate the cumulative 4-gram BLEU score, also called BLEU-4.

The weights for the BLEU-4 are 1/4 (25%) or 0.25 for each of the 1-gram, 2-gram, 3-gram and 4-gram scores. For example:

Running this example prints the following score:

The cumulative and individual 1-gram BLEU use the same weights, e.g. (1, 0, 0, 0). The 2-gram weights assign a 50% to each of 1-gram and 2-gram and the 3-gram weights are 33% for each of the 1, 2 and 3-gram scores.

Let's make this concrete by calculating the cumulative scores for BLEU-1, BLEU-2, BLEU-3 and BLEU-4:

Running the example prints the following scores. They are quite different and more expressive than the

They are quite different and more expressive than the standalone individual n-gram scores.

It is common to report the cumulative BLEU-1 to BLEU-4 scores when describing the skill of a text generation system.

Worked Examples

In this section, we try to develop further intuition for the BLEU score with some examples.

We work at the sentence level with a single reference sentence of the following:

the quick brown fox jumped over the lazy dog

First, let's look at a perfect score.

Running the example prints a perfect match.

Next, let's change one word, ' quick' to ' fast '.

This result is a slight drop in score.

Try changing two words, both ' quick' to ' fast' and ' lazy' to ' sleepy '.

Running the example, we can see a linear drop in skill.

What about if all words are different in the candidate?

We get the worse possible score.

Now, let's try a candidate that has fewer words than the reference (e.g. drop the last two words), but the words are all correct.

The score is much like the score when two words were wrong above.

How about if we make the candidate two words longer than the reference?

Again, we can see that our intuition holds and the score is something like " two words wrong ".

Finally, let's compare a candidate that is way too short: only two words in length.

Running this example first prints a warning message indicating that the 3-gram and above part of the evaluation (up to 4-gram) cannot be performed. This is fair given we only have 2-grams to work with in the candidate.

Next, we can a score that is very low indeed.

I encourage you to continue to play with examples.

The math is pretty simple and I would also encourage you to read the paper and explore calculating the sentence-level score yourself in a spreadsheet.

Further Reading

This section provides more resources on the topic if you are looking go deeper.

Summary

In this tutorial, you discovered the BLEU score for evaluating and scoring candidate text to reference text in machine translation and other language generation tasks.

Specifically, you learned:

A gentle introduction to the BLEU score and an intuition for what is being calculated.

How you can calculate BLEU scores in Python using the NLTK library for sentences and documents.

How to can use a suite of small examples to develop an intuition for how differences between a candidate and reference text impact the final BLEU score.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Skip the Academics. Just Results.

See What's Inside

# data[].attributes.attachment_link.title
A Gentle Introduction to Calculating the BLEU Score for Text in Python

# data[].attributes.attachment_link.description
BLEU, or the Bilingual Evaluation Understudy, is a score for comparing a candidate translation of text to one or more reference translations. Although developed for translation, it can be used to evaluate text generated for a suite of natural language processing tasks. In this tutorial, you will discover the BLEU score for evaluating and scoring ...

# data[].attributes.body
I am beginner in Python, so I checked how to programme a script that calculates BLEU score

# data[].attributes.attachment_link.readable_text
BLEU is a metric to quantify effectiveness of an Machine Translation (MT). It stands for Bi Lingual Evaluation Understudy $^{[1]}$. In general it solves the problem of different human translation references by different annotators when comparing to machine generated translation. Let's start from using BLEU in NLTK translation package $^{[2]}$

Assume we have translation references, generated by 4 annotators:

and our MT system generate:

Then we asked: how good is this? Given all difference references by human annotators?

What does it means?

the sentence_bleu method produces a continuous number from $[0, 1]$. BLEU score $1.0$ means that the machine output has exactly matched one of referenced human translation and it quantitatively shows that it is a good translation. Let's take another translation example

"it is a ship" in English should also be categorized as a good translation right? However since there are no referenced said exactly like this, it got lower score.

The Problem of BLEU score

Let's again look at these example

these are bad translations, but would result such a high BLEU score! Why is that? Let's go back a little on how to calculate BLEU score

BLEU From Scratch

Looking at BLEU paper $^{[1]}$ and Andrew Ng's video $^{[2]}$ we know that we should split our implementations into components

Counting Word Vectors

Matching word vector count with all candidate or one of the candidate is an intuitive and simple way to match translation candidate / reference with hypothesis so we will start from here.

Count Clip

Now we know that we could have several translation candidate for 1 hypothesis. Counting word vector will not be enough as we could have different word vector for each of translation candidate / reference. We wanted to normalize this word vector that we have obtained with word vector that will represent all references that we have.

One way to construct word vector from references is to find the maximum number of ngram occurs in on each reference instead of summing or averaging. We call this $\text{Max_Ref_Count}$. Now we want to "clip" count for each gram from translation.

Modified Precision

The modified precission ($p_n$, where $n$ is $n$-gram) is defined by

or in english language:

sum of all count clipped of n-gram of (translation and references) divided by sum of count n-gram from translation

Brevity Penalty

Now we wanted to show that we will reward translation that matches one of reference candidate length, not shorted and not longer. Since references length may be difference, we could still find and use the closest/effective reference length. Here the term Brevity Penalty ($BP$) comes

and we make $BP$ function to be

where

$c$: length of candidate translation

$r$: effective reference length

We don't penalize if $c$ larger than candidate translation because it has already penalized by modified precision. To be exact, because in modified precision when duplicate but relevant word occurs more than it supposed to or when the number of irrelevant word occurs more, it will results in larger denominator thus pulling the $BP$ towards 0.

We now want to show implementation of $BP$ in code.

The Bleu Score

Calculating BLEU after defining steps above is straightforward!

where:

$N$: maximum number of $n$-gram. Usually 4.

$w_n$: weight for each modified precision. By default use $1/N$, so 0.25 each gram.

$p_n$: modified precision for each gram

We have understand how to construct BLEU score for difference human reference and evaluate a hypothesis translation and we have seen that it did good job in scoring the hypothesis translation with this kind of setting.

BLEU might produce relatively good score to bad translation. In one of Phillip Koehn et al$^{[4]}$ work, they have explained phenomenons that makes BLEU flawed. However, it still is a good metric, used to evaluate machine translation performance up until now.

[1] F. Reznik, "Bleu a Method for Automatic Evaluation of Machine Translation," Che vuoi ?, vol. 30, no. 2, p. 107, 2015.

[2] NLTK: Translation Package

[3] Andrew Ng's Lecture: BLEU

[4] C. C. Miles and O. Philipp, "Re-evaluating the Role of Bleu in Machine Translation Research" pp. 249-256, 2002

# data[].attributes.attachment_link.description
BLEU is a metric to quantify effectiveness of an Machine Translation (MT). It stands for BiLingual Evaluation Understudy $^{[1]}$. In general it solves the problem of different human translation references by different annotators when comparing to machine generated translation. Let's start from using BLEU in NLTK translation package $^{[2]}$

# data[].attributes.attachment_link.readable_text
Related Research

Post-editing effort has been assessed via product analysis (by comparing machine translation output to a reference translation or post-edited sentence) and via process analysis (by observing aspects of the post-editing process, such as duration, production units, pauses, and fixation behavior). These measures will be discussed in more detail in the following paragraphs.

Assessing PE Effort via Product Analysis

Often used automatic metrics like BLEU ( Papineni et al., 2002) or METEOR ( Banerjee and Lavie, 2005) compare MT output to reference translations to evaluate a machine translation system's performance. Whereas the values given by such metrics can be used to benchmark and improve MT systems, they are created on the basis of translations made independently of the MT output and thus do not necessarily provide post-editors with valid information about the effort that would be involved in post-editing the output. In addition, a score given by such metrics contains no information about the complexity of specific errors that need to be fixed during post-editing.

More recently, research into machine translation quality estimation (QE) has moved away from the independent reference translations, as used by automatic metrics, to reference post-edited sentences. Many of these QE systems have been trained on HTER data ( Snover et al., 2006). HTER measures the edit distance between the MT output and a post-edited version of that MT output. The benefit of using HTER is the fact that it is relatively easy to apply, as it only requires MT output and a post-edited text. The underlying assumption when using HTER for QE is that HTER is an indication of actual post-editing effort ( Specia and Farzindar, 2010), which implies that all edits made to MT output by a human are expected to require a comparable amount of effort. However, HTER has one important limitation: as HTER focuses on the final product, it does not take the actual process into account, so its relationship to post-editing effort is questionable. For example, a post-editor can return to the same phrase multiple times during the post-editing process, changing that particular phrase each time, but settling on one specific solution in the end. HTER will indicate how different this final solution is from the original MT output, but it does not take into account all edits made during the process. In addition, the number of edits required to solve an issue does not necessarily correlate with the cognitive demand.

Assessing PE Effort via Process Analysis

According to Krings (2001), there are three main types of process-based post-editing effort. Of these three, the easiest to define and measure is temporal effort: how much time does a post-editor need to turn machine translation output into a high quality translation? The second type of post-editing effort is somewhat harder to measure, namely technical effort. Technical effort includes all physical actions required to post-edit a text, such as deletions, insertions, and reordering. The final type of effort is cognitive effort. It refers to the mental processes and cognitive load in a translator's mind during post-editing, and can, presumably, be measured via fixation data. While it is important to distinguish between these three types of post-editing effort conceptually, it must be noted that they are, to some extent, related to one another. Temporal effort is determined by a combination of cognitive effort and technical effort, and while an increase in technical effort does not necessarily correspond to an increase in cognitive effort, the technical process is still guided by cognitive processes.

Since effort can be defined in many different ways, researchers have used different methods to measure it. Koponen et al. (2012), for example, inquired into temporal and technical effort. They used a cognitively motivated MT error classification created by Temnikova (2010) and found that more time was needed to post-edit sentences that contained more cognitively demanding errors. They considered the number of keystrokes as well, but found no relationship between the number of keystrokes and the error types. Keystrokes were more strongly influenced by individual differences between participants.

A more direct cognitive measure was used by Jakobsen and Jensen (2008), who looked at fixation duration and the number of fixations during tasks of increasing complexity (from reading to translation). They found longer average fixation durations and a higher number of fixations as the complexity of a task increased. Doherty and O'Brien (2009) looked at the same variables, focusing specifically on the reading of MT output. They found a higher number of fixations when reading bad MT output than when reading good MT output, but they did not find a significant difference between the average fixation durations for both outputs.

Lastly, pauses have also been used a measure of effort. O'Brien (2006), for example, assumed that a higher number of negative translatability indicators (NTIs, i.e., elements in the source texts that are known to be problematic for MT) would be cognitively more demanding for a post-editor. She suggested pause ratio (total time in pauses divided by the total editing duration) as a possible indicator of cognitive effort, but she did not find conclusive evidence for a relationship between pause ratio and the number of NTIs in the source text.

Later, Lacruz et al. (2012) took the number of editing events in a sentence to be an indication of cognitive effort. They introduced the average pause ratio (the average duration per pause in a segment divided by the average duration per word in the segment) as an answer to O'Brien's (2006) pause ratio, arguing that pause ratio is not sensitive enough as a measure for cognitive activity, as it does not take average pause length into account. While their results were promising, it must be noted that they reported on a case study with only one participant, and that they used a different measure of cognitive demand compared to O'Brien (2006).

Impact of MT Quality

Denkowski and Lavie (2012) made a clear distinction between analyzing MT as a final product and MT fit for post-editing, saying that evaluation methods for the first may not necessarily be appropriate for the latter. It is the latter that the present article will be concerned with: does MT quality have an impact on PE effort, and, if so, which kinds of MT errors have the highest impact on this effort?

The problem has been approached by using human quality ratings ranging from 'good' to 'bad' ( Doherty and O'Brien, 2009; Koponen, 2012; Popovic et al., 2014) and error typologies ( Koponen et al., 2012; Stymne et al., 2012).

Doherty and O'Brien (2009), Koponen (2012), and Popovic et al. (2014) used human-assigned sentence ratings with four or five levels, with the highest score indicating that no post-editing was needed and the lowest score indicating that it would be better to translate from scratch. Whereas Popovic et al. (2014) included MT output at all levels in their analysis, Doherty and O'Brien (2009) and Koponen (2012) limited theirs to the MT segments with highest and lowest quality. It is therefore hard to directly compare the three studies. For the lower quality sentences, Koponen (2012) and Popovic et al. (2014) found an increase in the number of word order edits and Doherty and O'Brien (2009) found a higher number of fixations.

Regarding error typologies, Koponen et al. (2012) used the classification proposed by Temnikova (2010), which contains various MT output errors ranked according to cognitive demand, and Stymne et al. (2012) used the classification proposed by Vilar et al. (2006). This difference in classification makes it hard to compare both studies, although they both found word order errors and incorrect words to impact post-editing effort the most: Koponen et al. (2012) studied their relationship with post-editing duration, and Stymne et al. (2012) studied their relationship with fixation duration.

For future work, researchers suggest using more fine-grained error typologies ( Koponen et al., 2012; Stymne et al., 2012) and different languages ( Koponen et al., 2012; Stymne et al., 2012; Popovic et al., 2014).

Impact of Translation Experience

The effort involved in post-editing can be expected to be different for professional translators and students on the basis of previous studies in the field of translation and revision research.

Inexperienced translators have been shown to treat the translation task as a mainly lexical task, whereas professional translators pay more attention to coherence and style ( Tirkkonen-Condit, 1990; Séguinot, 1991). Students have also been shown to require more time ( Tirkkonen-Condit, 1990), and a higher number of fixations and pauses than professional translators while translating ( Dragsted, 2010).

A comparable trend is found in revision research. Sommers (1980), for example, found that experts adopt a non-linear strategy, focusing more on meaning and composition, whereas student revisers work on a sentence level and rarely ever reorder or add information. Hayes et al. (1987), too, reported that expert revisers first attend to the global structure of a text, whereas novice revisers attend to the surface level of a text. Revision is seen as a complex process that puts a great demand on working memory. Broekkamp and van den Bergh (1996) found that students were heavily influenced by textual cues during the revision process. For example, if a text contained many grammatical errors, the reviser's focus switched to solving grammatical issues, and more global issues were ignored.

It remains to be seen whether these findings can be extrapolated to the post-editing process. The study by de Almeida and O'Brien (2010) is, to the best of our knowledge, the only one linking experience to the post-editing process. They found that more experienced translators were faster post-editors and made more essential changes as well as preferential changes.

Hypotheses

In this article, we focus on the following research questions: (i) are all effort indicators influenced by machine translation quality, (ii) is the product effort indicator HTER influenced by different machine translation error types than the process effort indicators, (iii) is there an overlap between the error types that influence the different process effort indicators, and (iv) is the impact of machine translation error types on effort indicators different for student translators than for professional translators?

On the basis of the above-mentioned research, we expect that a decrease in machine translation quality will lead to an increase in post-editing effort, as expressed by an increase in HTER ( Specia and Farzindar, 2010), the number of production units ( Koponen, 2012; Popovic et al., 2014), the number of fixations ( Doherty and O'Brien, 2009), post-editing time ( Koponen et al., 2012), fixation duration ( Stymne et al., 2012), pause ratio ( O'Brien, 2006), and a decrease in average pause ratio ( Lacruz et al., 2012).

As HTER is a product measure, we expect it to be influenced by different machine translation error types than the process effort indicators, which we expect to be influenced by comparable error types, as they are, to a certain extent, related ( Krings, 2001). More specifically, we expect process effort indicators to be influenced most by mistranslations and word order issues ( Koponen et al., 2012; Stymne et al., 2012) and lexical or semantic issues ( Popovic et al., 2014).

Given the notion that inexperienced revisers focus more on grammatical errors when there is an abundance of grammatical errors ( Broekkamp and van den Bergh, 1996), and the fact that student translators treat translation as a lexical task ( Tirkkonen-Condit, 1990), we expect students to focus mostly on the grammatical and lexical issues, whereas professional translators are expected to pay more attention to coherence, meaning, and structural issues ( Sommers, 1980). This means that we expect to see a greater increase in post-editing effort with students than with professional translators when there is an increase in grammatical and lexical issues in the text, and we expect a greater increase in post-editing effort with professional translators than with students when there is an increase in coherence, meaning, or structural issues.

Participants

Participants were 13 professional translators (3 male and 10 female) and 10 master's students of translation (2 male and 8 female) at Ghent University who had passed their final English translation examination, meaning that they were ready to start working as professional translators. All participants were native speakers of Dutch. With the exception of one professional translator, who had 2 years of experience, all other translators had experience working as a full-time professional translator between 5 and 18 years. Students' median age was 23 years (range 21-25), that of the professional translators was 37 years (range 25-51).

All participants had normal or corrected to normal vision. Two students wore contact lenses and one student wore glasses, yet the calibration with the eye tracker was successful for all three. Two professional translators wore lenses. Calibration was problematic for one of the professionals, their sessions with problematic calibration were removed from the data.

The project this study is a part of has been reviewed and approved by the Ethical Committee of the Faculty of Psychology and Educational Sciences at Ghent University. All participants gave written informed consent.

Students reported that they were aware of the existence of MT systems, and sometimes used them as an additional resource, but they had received no explicit post-editing training. Some professional translators had basic experience with post-editing, although none of the translators had ever post-edited an entire text.

All participants performed a LexTALE test ( Lemhöfer and Broersma, 2012), which is a word recognition test used in psycholinguistic experiments, so as to assess English proficiency. Other than being an indicator of vocabulary knowledge, it is also an indicator of general English proficiency. As such, it is an important factor to take into account when comparing the translation process of students and professionals. Both groups had very high scores on this test; their scores (professionals mean = 88.27; standard deviation = 9.5; students mean = 88; standard deviation = 7.75) did not differ significantly [ t(21) = 0.089, p = 0.47].

Text Selection

Both to avoid specialized text experience playing a role and to control for text complexity as much as possible, 15 newspaper articles with a comparable complexity level were selected from Newsela, a website that offers English newspaper articles at various levels of complexity (as represented by Lexile ® scores, which combine syntactic and semantic information). We selected 150/160-word passages from articles with high Lexile ® scores (between 1160L and 1190L). To control texts further, we manually compared them for readability, potential translation problems and MT quality. Texts with on average fewer than 15 or more than 20 words per sentence were discarded, as well as texts that contained too many or too few complex compounds, idiomatic expressions, infrequent words or polysemous words. The Dutch MT output was taken from Google Translate (output obtained January 24, 2014), and annotated with a two-step Translation Quality Assessment approach ( Daems et al., 2013, see also "MT Output Error Analysis").

We discarded the texts for which the MT output would be too problematic, or not problematic enough, for post-editors, based on the number of structural grammatical problems, lexical issues, logical problems and mistranslated polysemous words. The final corpus consisted of eight newspaper articles, each 7-10 sentences long. The topics of the texts varied, and the texts required no specialist knowledge to be translated.

MT Output Error Analysis

Annotation of the MT Output

To be able to identify the relationship between specific machine translation problems and post-editing effort, two of the authors annotated all translations for quality. Machine translation quality was analyzed from two perspectives.

The first was acceptability, or the adherence to target text and target language norms. This category contained all translation problems that can be identified by looking at the target text (in this case, the machine translation output) without consulting the source text. It consisted of the subcategories of grammar and syntax, lexicon, coherence, style and register, and spelling, which were all further subdivided into subcategories.

The second perspective was adequacy, or the adherence to source text norms. All problems that could be identified by comparing the source text with the target text belonged to this category. Subcategories included deletions and additions (of words or phrases), various types of mistranslations, and meaning shifts.

Each error type was given an error weight ranging from 0 (not a real error, but a change nonetheless, such as an explicitation) to 4 (contradictions and other problems that severely impacted the comprehensibility of the text).

The actual annotation took place in two steps, respecting both perspectives above: in step one the annotators were only given the target text and annotated the text for acceptability, and in step two annotators compared the translations with the source texts and annotated the texts for adequacy. The annotations of both annotators were then compared and discussed and only the annotations which both annotators agreed on were maintained for the final analysis. Annotations were made with the brat rapid annotation tool ( Stenetorp et al., 2012).

Overview of Errors in the MT Output

Out of 63 sentences, 60 sentences contained at least one error. There were more acceptability issues (201 instances) than adequacy issues (86 instances). The error categorization described above contained 35 types of acceptability issues and 17 types of adequacy issues, but not all issues were found in the machine translation output. To be able to perform statistical analyses, some of the error categories were grouped together, so that each error type occurred at least 10 times. The final classification used in this study can be seen in .

From the perspective of 'acceptability,' it was the grammar and syntax category, which turned out to be the most common error category for MT output, with word order issues, structural issues, and incorrect verb forms occurring more than 10 times each. The different types of agreement issues (noun-adjective, article-noun, subject-verb, and reference) were grouped into a new 'agreement' category, and the other grammatical issues were contained in a 'grammar other' category (superfluous or missing elements). For coherence issues, the category of 'logical problem' occurred more than 10 times, but the other categories together (conjunction, missing info, paragraph, and inconsistency) did not occur more than 10 times, so all other coherence categories were grouped together. As for the lexicon, the subcategory of 'wrong collocation' appeared often enough to stand alone, while the other subcategories (wrong preposition, named entity, and word non-existent) were grouped into 'lexicon other.' All subcategories for style and spelling were merged together into the two main categories, since there were very few instances of these errors.

From the perspective of 'adequacy,' other meaning shifts and word sense issues occurred frequently enough to be considered as separate categories, while the other subcategories (additions, deletions, misplaced words, function words, part of speech, and inconsistent terminology) were grouped together into 'adequacy other.'

The most common errors overall are grammatical errors (grammar and syntax), followed closely by adequacy issues.

Procedure

Sessions

The data were gathered during two separate sessions for each participant. The students' sessions took place in June and July 2014 and the professionals' sessions took place in April and May 2015.

The first session started with a survey, to gain an idea of participants' backgrounds, their experience with and attitude toward MT and post-editing, and a LexTALE test. This was followed by a copy task (participants copied a text to get used to the keyboard and screen) and a warm-up task combining post-editing and human translation, so that participants could get used to the environment and the different types of tasks. The actual experiment consisted of two texts that they translated from scratch, and two texts that they post-edited.

The second session started with a warm-up task as well, followed first by post-editing two texts and translating two texts from scratch. The final part of the session consisted of unsupervised retrospection (participants received the texts which they had just translated and were requested to highlight elements they found particularly difficult to translate) and another survey, to obtain insight into participants' attitude after the experiment.

The order of the texts and tasks was balanced across participants within each group in a Latin square design. Participants were instructed to deliver products of publishable quality, whether they were translating from scratch or post-editing. There was no time limit, and participants could take breaks between texts if so requested, to mimic a natural translation day. All participants had a morning and an afternoon session, though temperature and light conditions remained constant throughout the different sessions, as they took place in a basement room (controlled light is needed for accurate eyetracking data).

Registration Tools

To be able to study all aspects of the translation process, the process was registered with two keystroke logging tools and an EyeLink 1000 eyetracker.

The main keystroke logging tool was the CASMACAT translators' workbench ( Alabau et al., 2013), which is an actual workbench with added mouse tracking and keystroke logging software. A browser plugin directly added the EyeLink fixation data to the CASMACAT logging files. Though not relevant for this particular analysis, we also used the Inputlog keystroke logging tool ( Leijten and Van Waes, 2013) to register the usage of external resources during translation and post-editing.

The texts were presented to the participants one by one, with eye tracker calibration taking place before each new text. Only one sentence could be edited at a time, although the entire text was visible and participants could go back and forth through the text. The source text was shown on the left hand side of the screen, the right hand side contained the MT output.

Analysis

The final dataset comprised 721 post-edited sentences, concatenated from all post-editing sessions and enriched with MT quality information. For each sentence, the average error weight per word was calculated by summing up the error weight of all errors found in the sentence and dividing that value by the number of words in the sentence.

The statistical software package R ( R Core Team, 2014) was used to analyze the data. We used the lme4 package ( Bates et al., 2014) and the lmerTest package ( Kuznetsova et al., 2014-48pc) to perform linear mixed effects analyses. In these analyses, statistical models can be built that contain independent variables as well as random effects, i.e., subject or item-specific effects. It is also possible to have more than one independent variable, and to study the interaction effect of independent variables on one another. A statistical model is always tested against a null model, i.e., a model that assumes that there is no effect on the dependent variable. If the created model differs sufficiently from the null model, we can assume that there is an effect from the independent variables on the dependent variables.

In this particular study, we used the various post-editing effort indicators as dependent variables, see for an overview of dependent variables with descriptives for student and professional data; The independent variables are machine translation quality and experience. They were included in the models with interaction effect, as we are interested in seeing how machine translation quality has a different influence on effort indicators for students and professional translators. The random effects were participants and sentence codes (a unique id for each source text sentence), because we expected there to be individual differences across participants as well as sentence-inherent effects.

Table 1

For each of the models discussed below, we started by building a null model with the post-editing effort indicator as dependent variable and participant and sentence code as random effects. This model was then tested against the model containing two predictors (the continuous variable machine translation quality and the categorical variable experience, which was either 'student' or 'professional'), plus interaction effect.

As a measure of model fit, we used Akaike's Information Criterion (AIC) values ( Akaike, 1974). The model with the lower AIC value has a better fit compared to the model with the higher AIC value, especially when the difference between both models is greater than 4 ( Burnham and Anderson, 2004), but an AIC value in itself has no absolute meaning, which is why we only included difference in AIC value in the tables.

We looked at the impact of fine- and coarse-grained machine translation quality on different post-editing effort indicators in two different analyses. This was done to first establish whether effort indicators are indeed impacted by machine translation errors, and to verify whether the suggestion by Koponen et al. (2012) and Stymne et al. (2012) to use more fine-grained error typologies really leads to more obvious differences between the different post-editing effort indicators.

The first analysis was the most coarse-grained, in which we used the average total machine translation error weight and experience as predictor variables. For each of the post-editing effort indicators, we built a null model and a model with both predictor variables, which we then compared.

For the fine-grained analysis, we also built separate models for each of the post-editing effort indicators. Here, we added all our adapted machine translation error subcategories as possible predictors. We then tested which of the subcategories were retained by the step-function from the lmerTest package. For each of these significant predictors, we built a separate model, adding experience with interaction effect as a possible additional predictor.

Analysis 1: Coarse-Grained

A summary of the models with average total machine translation error weight and experience plus interaction as possible predictor variables can be found in . The table contains the dependent variable, i.e., the post-editing effort indicator under scrutiny in the first column, the difference in AIC value between the predictor model and the null model (i.e., model without predictor) in the second column, an overview of the significant or almost significant predictors (i.e., MT error weight and/or experience, with or without interaction effect, with p ≤ 0.05) in the third column, the effect size for each significant predictor in column four, and the p-value to indicate significance in column five. The effect column shows the actual effect that an independent variable has on a dependent variable. For non-significant effects, the table contains 'n/a.'

Table 2

For most post-editing effort indicators, machine translation quality has an impact, but experience does not, with the exception of both pause measures. The effect of machine translation quality on the effort indicators is in line with expectations, with a decrease in quality leading to an increase in duration needed (), an increase in the number of fixations () and the number of production units (), an increase in HTER score (), and a decrease in average pause ratio (). The post-editing effort indicators on which machine translation quality seems to have no statistically significant impact are average fixation duration and pause ratio. The latter is impacted by experience rather than machine translation quality (), whereas the average pause ratio is impacted by experience as well as machine translation quality. Students have a significantly higher pause ratio than professionals, presumably requiring more time to think before changing anything, but also a higher average pause ratio, which is somewhat surprising. It must be noted, however, that the latter effect is not quite significant ( p = 0.05) and so it must be interpreted with caution. We further found no significant effect of either machine translation error weight or experience on average fixation duration.

Analysis 2: Fine-Grained

We obtain a more nuanced picture when studying the results for the fine-grained level analysis in . We first built a model including all possible machine translation error types as possible predictors for the different post-editing effort indicators. The significant error types are listed in column two, showing that different types of machine translation errors impact different post-editing effort indicators. The more technical effort indicators (number of production units, pause ratio, average pause ratio, and HTER) are mostly impacted by grammatical errors (grammar, structure, word order, and agreement), whereas the more cognitive effort indicators (fixations and duration) are influenced most by coherence and other meaning shifts. We then built a model for each of the significant MT error types separately and added experience plus interaction effect as possible predictors. The significant ( p < 0.05) and almost significant ( p < 0.06) predictors for these models can be seen in column four, with the actual effect size in column five. For non-significant effects, the table contains 'n/a.'

Table 3

It is interesting to see how experience and/or experience with interaction effect now become relevant for the models with average duration (), average fixation duration (), and HTER () as dependent variables. In the case of average duration, students seem to be impacted less by an increase in coherence issues than professional translators, although the significance levels are not convincing (the experience effect only just reaches significance, the interaction effect almost reaches significance). In the case of average fixation duration, only students seem to be impacted by an increase in other meaning shifts, whereas the average fixation duration of professional translators remains comparable. The trend for HTER is similar, with students responding more strongly (yet not statistically significantly) to an increase in other meaning shifts than professional translators. In addition, HTER scores for students go up significantly with an increase in spelling errors. In the models for average number of production units and average pause ratio, a few different machine translation error types seem to influence the post-editing effort indicators, but once split into a separate model containing experience with interaction effect as additional predictors, the predictors are no longer significant. The model with word order as predictor variable of the average number of production units approaches significance, but that is the only one.

Hypothesis 1

From the coarse-grained analyses, we learned that post-editing effort indicators can indeed be predicted by machine translation quality, with the exception of average fixation duration and pause ratio.

Previous studies, too, have shown that average fixation duration does not differ significantly for good and bad MT quality ( Doherty and O'Brien, 2009; Doherty et al., 2010), and so perhaps average fixation duration is not a good measure of post-editing effort.

For pause ratio, it seems that students require significantly more time in pauses than professional translators, and this effect outweighs the impact of machine translation quality. Perhaps students need more time to think about the correct course of action, whereas this process has been become more automatic for professionals. This confirms previous findings on students' pause behavior ( Dragsted, 2010), and might be an indication that pause ratio measures something else than average pause ratio ( Lacruz et al., 2012), as the latter is influenced by machine translation quality.

Hypothesis 2

Most importantly, the fine-grained analysis shows that two out of three predictors (agreement and spelling issues) of the product effort indicator HTER are unique predictors, i.e., they do not appear in the models for any of the other post-editing effort indicators. This seems to offer support for our hypothesis that product effort indicators measure different things than process effort indicators. On the other hand, the HTER predictor 'other meaning shift' is shared with a few process effort indicators as well, which means that HTER presumably does manage to measure some of the effort measured by process effort indicators.

Hypothesis 3

Not all process effort indicators seem to be influenced by the same machine translation error types, refuting our third hypothesis. Duration, for example, is influenced most by coherence, whereas fixation duration is influenced by other meaning shifts. The machine translation error types that occur with more than one post-editing effort indicator are coherence issues, other meaning shifts, grammar and structural issues. As 'other meaning shifts' also occurred as a predictor of HTER, it would seem that being able to detect these three issue types (coherence issues, other meaning shifts, grammar and structural issues) would be the best way to ensure that as many different types of effort as possible are being taken into account.

The lack of word order issues as an influential category is striking, as we expected them to influence post-editing duration ( Koponen et al., 2012), the average fixation duration, and number of fixations ( Stymne et al., 2012). A possible explanation for these findings is the fact that our error classification includes error types such as coherence, which was not included in Temnikova's (2010) classification, and those error types outweigh the effect of commonly used error types such as word order issues. Further research on a larger dataset and different languages is needed to support or refute these claims.

The more detailed findings can largely be explained by existing literature. The number of production units was influenced most by grammatical issues ( Alves and Gonçalves, 2013), and fixations were influenced most by other meaning shifts (comparable to mistranslations), as suggested by Stymne et al. (2012). With respect to both pause measures, we found support for the claim made by Lacruz et al. (2012) that average pause ratio does not measure the same as pause ratio suggested by O'Brien (2006). Average pause ratio is influenced by structural issues and coherence issues, whereas pause ratio is influenced by grammatical issues, but this effect is outweighed by the influence of experience.

Hypothesis 4

Regarding experience effects, we expected students to be more heavily influenced by grammatical and lexical issues and professional translators to be more heavily influenced by coherence and structural issues ( Sommers, 1980; Hayes et al., 1987; Tirkkonen-Condit, 1990; Séguinot, 1991).

This almost holds true for the average duration per word, where an increase in coherence issues leads to an increase in the time needed and this effect is stronger for professional translators than for students. It must be noted, however, that this interaction effect is not quite significant ( p = 0.054).

For the average fixation duration, which is influenced most heavily by other meaning shifts, only the interaction effect between the increase in other meaning shifts and experience is significant. While the average fixation duration for professional translators remains more or less constant as the number of other meaning shifts increases, the average fixation duration for students goes up under the same circumstances.

A comparable trend can be seen with HTER, although the effect here is not significant. This can be an indication that other meaning shifts are cognitively more demanding for students, seeing how it is not an issue they usually focus on during translation. For professional translators, however, spotting and solving these issues is probably more routinised, causing their average fixation duration to remain constant and their HTER scores to be lower ( Tirkkonen-Condit, 1990; Séguinot, 1991). Students' HTER scores were also more heavily influenced by spelling errors. This finding is somewhat harder to explain, as spelling issues are mostly related to compounds and spaces before punctuation, which are both error types that can be expected to be corrected by both students and professional translators. We would have to look at the final translations to see whether this was actually the case.

Experience had no effect on the number of fixations, production units, and average pause ratio, which could indicate that these three predictors measure a different kind of post-editing effort than the other predictors. These findings could be further evaluated by studying the final post-editing product and comparing the quality of student and professional post-editors' texts.

Limitations

Although we included data from students as well as professional translators, the total dataset remains relatively small. While our fine-grained analysis showed some promising results, it must be noted that with the more fine-grained analysis, fewer observations are taken into account, and so these results need to be interpreted with caution. Further experiments with the same categorisation on more data and different language pairs should be carried out in order to further develop the claims made in this article.

After this paper was written, Google released its neural machine translation for the English-Dutch language pair. Future experiments will repeat the above analyses with this newer type of machine translation, to see whether the results can be replicated or to discover the ways in which neural machine translation differs from statistical machine translation with regards to its impact on post-editing effort.

¹Given that Koponen et al. (2012) found that keystrokes were more influenced by personal differences rather than differences in effort, we decided to include production units rather than keystrokes. Production units are editing events, i.e., continuous typing activities, that are often defined as units separated from one another by pauses of at least one second ( Carl et al., 2016).

² Instructions can be found online: users.ugent.be/∼jvdaems/TQA_guidelines_2.0.html

³ Issues not found in MT output: spelling: capitalization, typo; coherence: inconsistency, paragraph, coherence other; grammar: comparative/superlative, singular/plural; style: long/short sentences, text type; adequacy: contradiction, explicitation, hyperonymy, hyponymy, terminology, time.

⁴ Other meaning shifts are adequacy issues that cannot be classified as belonging to one of the other subcategories, and a word sense issue occurs when the wrong meaning of the word has been chosen for the context. For example, the phrase 'donned shades,' i.e., 'put on sunglasses,' was translated into Dutch as 'vormden schaduwen' (created shadows). The translation for 'donned' is a case of 'other meaning shift,' the translation for 'shades' is a word sense issue, 'shadow' being a possible translation for 'shade,' however, not in this context.
⁵

Alves F., Gonçalves J. L. (2013). Investigating the conceptual-procedural distinction in the translation process. 25 107-124. [Google Scholar]

Bates D., Maechler M., Bolker B., Walker S. (2014). [lme4: Linear Mixed-Effects Models using Eigen and S4. R Package Version 1.1-7 Available at:https://CRAN.R-project.org/package=lme4 Google Scholar]

Carl M., Schaeffer M., Bangalore S. (eds) (2016). "The CRITT translation process research database," in [New Directions in Empirical Translation Process Research (Berlin: Springer; ), 13-54. 10.1007/978-3-319-20358-4_2 [CrossRef] Google Scholar]

Dragsted B. (2010). "Coordination of reading and writing processes in translation: an eye on uncharted territory," in [Translation and Cognition, eds Shreve G. M., Angelone E., editors. (Amsterdam: John Benjamins; ). 10.1075/ata.xv.04dra [CrossRef] Google Scholar]

Hayes J., Flower L., Scriver K., Stratman J., Carey L. (1987). "Cognitive processing in revision," in [Advances in Applied Psycholinguistics: Reading, Writing, and Language Processes Vol. 2 ed. Rosenberg S., editor. (New York, NY: Cambridge University Press; ), 176-240.Google Scholar]

Krings H. (2001). [Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes. Trans. Koby G., Shreve G. M., Mischerikov K., Litzer S., Ohio K., editors. (London: The Kent State University Press; ).Google Scholar]

Kuznetsova A., Brockhoff P. B., Christensen R. H. B. (2014). [lmerTest: Tests in Linear Mixed Effects Models. R Package Version 2.0-20. Available at: http://CRAN.R-project.org/package=lmerTest Google Scholar]

Lemhöfer K., Broersma M. (2012). Introducing LexTALE: a quick and valid lexical test for advanced learners of english. [[Behav. Res. Methods 44 325-343. 10.3758/s13428-011-0146-0PubMed] [PMC free article] [CrossRef] Google Scholar]

R Core Team (2014). [R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.Google Scholar]

Tirkkonen-Condit S. (1990). "Professional vs. Non-professional translation: a think-aloud protocol study," in [Learning, Keeping and Using Language: Selected Papers from the Eighth World Congress of Applied Linguistics, eds Halliday M. A. K., Gibbons J., Nicholas H., editors. (Amsterdam: John Benjamins; ), 381-394. 10.1075/z.lkul2.28tir [CrossRef] Google Scholar]

https://nederland.googleblog.com/2017/04/grote-verbetering-voor-het-nederlands.html?m=1

# data[].attributes.attachment_link.title
Identifying the Machine Translation Error Types with the Greatest Impact on Post-editing Effort

# data[].attributes.attachment_link.description
Post-editing effort has been assessed via product analysis (by comparing machine translation output to a reference translation or post-edited sentence) and via process analysis (by observing aspects of the post-editing process, such as duration, production units, pauses, and fixation behavior). These measures will be discussed in more detail in the following paragraphs.

# data[].attributes.body
To be able to classify properly errors in my project.

# data[].attributes.attachment_link.readable_text
Although computers have stolen the spotlight from humans in many sectors, machine translation is not a threat to professional translators.

Today, we all have access to an automatic translator, and it is true that we are often tempted to use this tool. But only a couple of uses are enough to realize that these tools are not always effective. This is because they provide literal translations in their raw state, without taking into account cultural and contextual factors, which is essential to any professional translation. This is the main reason that automatic translations often make mistakes.

Although technology might have stolen the spotlight from humans in many sectors, these tools aren't a threat to professional translators. Here are the main reasons why.

Firstly, translation tools are very general. The reason is simple: they're supposed to be useful with any type of document. This is where the worst enemy of machine translation tools comes in: polysemy. You will find below a small example that should make you smile, where the computer obviously did not understand that the word "advocat" does not have the same meaning in a legal context as it does in a culinary context.

Another flaw of translation tools lies in the translation model. These tools compare numerous collections of texts and draw translation rules from them. Despite these rules, for each new translation, the professional translator is faced with new issues which only they alone can solve by making choices. These choices cannot be made by the software.

Here, the software has translated a common name for a piece of clothing into a trademark.

Here, polysemy has led to a computer error. The verb "sauter" in French can be translated as to jump, or to skip in English, depending on the context. Unluckily for these far-from-perfect tools, the verb "sauter" was used incorrectly. It should have been used to mean "skip" in order to conserve the original meaning of the sentence..

We continue our series of errors with examples below. How can we forget the publication of Benyamin Netanyahou who had the opportunity to meet "François Pays-Bas". Here, the automatic translation tool suggested by a famous social network gave rise to a particularly amusing result in French.

One more for the road? In January 2012, the Ministry of Defense site of a country in Southeast Asia published an article on the clothing of its officials. According to these recommendations, they were not to wear "eye-catching" clothing, but "mongoose fighting" uniforms were permitted. What does this even mean? The Department quickly removed these translations, putting the blame for this blunder on a famous machine translation company.

In addition, in a medical or legal context, a bad translation could have serious consequences.

In conclusion, always be cautious of this type of tool. They can be useful in some cases, but their limits must never be overlooked. Our advice is to always use professional translation services.

Translation into English: Chloe Findlay

Discover our translation agency.

# data[].attributes.attachment_link.description
Although computers have stolen the spotlight from humans in many sectors, machine translation is not a threat to professional translators. Today, we all have access to an automatic translator, and it is true that we are often tempted to use this tool.

# data[].attributes.body
Knowledge of Excel needed to calculate properly adequacy, time and number of eerors

# data[].attributes.attachment_link.readable_text
Ten artykuł zawiera opis składni formuły i zastosowania funkcji średnia w programie Microsoft Excel.

Opis

Zwraca średnią (średnią arytmetyczną) argumentów. Jeśli na przykład zakres a1: A20 zawiera liczby, formuła =średnia (a1: A20) zwraca wartość średnią tych liczb.

Składnia

ŚREDNIA(liczba1;[liczba2];...)

W składni funkcji ŚREDNIA występują następujące argumenty:

Liczba1 Argument wymagany. Pierwsza liczba, odwołanie do komórki lub zakres, dla którego ma zostać obliczona średnia.

Liczba2; ... Argument opcjonalny. Dodatkowe liczby, odwołania do komórek lub zakresy (maksymalnie 255), dla których ma zostać wyznaczona średnia.

Spostrzeżenia

Argumentami mogą być liczby, nazwy, zakresy lub odwołania do komórek, które zawierają liczby.

Wartości logiczne i tekstowe reprezentacje liczb wpisane bezpośrednio na liście argumentów nie są liczone.

Jeśli argument zakresu lub odwołania do komórki zawiera tekst, wartości logiczne lub puste komórki, to wartości te są ignorowane. Komórki o wartości zero są jednak uwzględniane w obliczeniach.

Argumenty, które są wartościami błędów lub tekstem, którego nie można przekształcić w cyfry, powodują błędy.

Aby uwzględnić wartości logiczne i tekstowe reprezentacje liczb w odwołaniu jako część obliczeń, należy użyć funkcji ŚREDNIA.A.

Aby obliczyć średnią tylko dla wartości spełniających określone kryteria, należy użyć funkcji ŚREDNIA.JEŻELI lub funkcji ŚREDNIA.WARUNKÓW.

Uwaga: Funkcja ŚREDNIA sprawdza tendencję centralną, którą jest lokalizacja środka zbioru liczb w rozkładzie statystycznym. Trzy najczęściej używane miary tendencji centralnej to:

Średnia - średnia arytmetyczna, obliczana jako suma zbioru wartości podzielona przez ich liczbę. Na przykład średnią liczb 2, 3, 3, 5, 7 i 10 jest wartość 30 podzielona przez 6, czyli 5.

Mediana - środkowa liczba w zbiorze wartości (połowa liczb jest większa lub równa medianie, a połowa liczb jest mniejsza lub równa medianie). Na przykład medianą zbioru 2, 3, 3, 5, 7 i 10 jest 4.

Wyst.najczęściej - najczęściej występująca wartość w zbiorze (dominanta). Na przykład dominantą zbioru 2, 3, 3, 5, 7 i 10 jest 3.

W przypadku symetrycznego rozkładu zbioru liczb wszystkie trzy miary tendencji centralnej są takie same. W przypadku skośnego rozkładu zbioru liczb miary mogą być różne.

Porada: Podczas obliczania średniej komórek należy pamiętać o różnicy między komórkami pustymi a komórkami zawierającymi wartości zerowe, szczególnie jeśli zostało wyczyszczone pole wyboru Pokaż zero w komórkach o zerowej wartości w oknie dialogowym Opcje programu Excel w aplikacji komputerowej programu Excel. Jeśli ta opcja jest zaznaczona, w obliczeniach są uwzględniane wartości zerowe, ale nie są uwzględniane puste komórki.

Aby znaleźć pole wyboru Pokaż zero w komórkach o zerowej wartości:

Uwaga: Ta strona została przetłumaczona automatycznie i może zawierać błędy gramatyczne lub nieścisłości. Chcemy, aby ta zawartość była dla Ciebie przydatna. Czy możesz dać nam znać, czy te informacje były pomocne? Oto angielskojęzyczny artykuł do wglądu.

# data[].attributes.attachment_link.description
Ten artykuł zawiera opis składni formuły i zastosowania funkcji średnia w programie Microsoft Excel. Zwraca średnią (średnią arytmetyczną) argumentów. Jeśli na przykład zakres a1: A20 zawiera liczby, formuła = średnia ( a1: A20) zwraca wartość średnią tych liczb. ŚREDNIA(liczba1;[liczba2];...) W składni funkcji ŚREDNIA występują następujące argumenty: Liczba1 Argument wymagany.

# data[].attributes.headline
Reading about machine translation to understand it better

# data[].attributes.subject
Reading about machine translation to understand it better

# data[].attributes.attachment_link.readable_text
Chances are that most of us have some experience with machine translation in some way. Google Translate and similar tools have been around the web for quite some time.

In fact, just yesterday I was using Google Translate because I sent a wedding gift to a friend in Spain, and received an email in Spanish - turns out they were having a problem with delivery!

Machine translation (MT) tools enable users to input their source-language text, and the engine will spit out a complete and mostly accurate (albeit quite literal) translation in the target language.

Neural machine translation(NMT) is designed to learn language much like the human brain does, adapting to your brand's unique voice and tone overtime.

MT has been around for a bit longer than some may realize and the technology has come a long way from its origins. Modern machine learning and neural networks have pushed MT into an entirely new realm.

With usable and modern MT available, the question shouldn't be if you decide to use MT, but rather, "when is the right time to use MT?"

Machine Translation In Smartling

We recognize that not every job will require a human translator. That's why Smartling makes it easy for customers to find the path that works best for each project.

Smartling already connects with the best Machine Translation engines:

More recent innovations in this space is the advent of Neural Machine Translation. Neural machine translation (NMT) is designed to learn language much like the human brain does, adapting to your brand's unique voice and tone overtime.

With direct integrations to leading providers, Smartling positions you to integrate with the best machine translation services possible.

But to understand how NMT has become so powerful, let's look a little bit further at exactly where MT came from, and how far it has evolved.

A Brief History of Machine Translation

The short story is this: machine translation stumbled first to become a usable solution, and was almost abandoned outright. Yet, modern tech break-throughs have awakened machine translation, specifically leveraging powerful neural networks for machine learning.

These are the four most prominent milestones of machine translations:

1949: MT started its infancy conceptually, with a physical device and public presentation finally making an appearance in 1954, by a Georgetown MT research team.

1966: The National Academy of Science formed a specific MT committee, known as ALPAC. A report published by ALPAC just a few years later almost decimated the industry, with strong suggestions to stop funding and for research to come to a halt -- the technology simply wasn't there to make it a possibility.

1997: MT makes its way into the mainstream as the internet takes off: AltaVista's well-known Babelfish was introduced in '97, and Google Translate came to be almost 10 years later in '06.

2013: Modern MT starts to officially gain some ground, beginning with Statistical Translation. More recently, Google has been researching and implementing NMT.

Today we see Nueral Machine Translation taking off, pushing the concept of MT into boundaries previously seen as impossible, at least according to ALPAC.

According to Jack Welde, Smartling Founder and CEO:

"Neural machine translation is gradually eliminating the demarcation between human and machine translation. It is creating more opportunities for a productive closed loop between machine and human, including machine enabled tools that make the human more productive, and human inputs that make the machine more accurate going forward."

Machine Translation Today

For a quick high-level view, and to provide further context of how MT will fit into your business' strategy, let's take a closer look at the two major distinctions in modern MT -- Statistical and Neural Network based engines.

Statistic Based engines learn through statistical analysis of a bilingual text, generally provided by the developer or user. These engines essentially develop an understanding of existing rules to determine the unique relationship between the source language and target language.

Neural Based engines are the most modern approach to MT. Neural networks are designed to mimic how the human mind learns, gaining more knowledge over time. These engines seek to understand the context of what is being translated to properly predict the correct word choice.

Neural based machine translation engines are much more capable of capturing, and even understanding, the intent or meaning of a sentence, and therefore have been quickly replacing older statistical models. The idea here isn't just swapping one word for another based on a rule or phrase.

Instead, the NMT engine is working to understand the intent behind your content in order to maintain and retain that particular message and tone in translation.

Out of the Box vs Adaptive Machine Translation

Part of the beauty behind modern Nueral Machine Translation engines is the ability for not only rapid deployment but almost complete customization. That's part of what makes them the best machine translation models we have so far.

When leveraged as a service, most Nueral Machine Translation engines can either be used straight out of the box -- but require in-depth training -- or can be pre-adapted for one specific brand or domain.

Training these engines alone can be a time-consuming process, but working with the right vendor can ensure not only quicker deployment, but also much more accurate results -- that is why Smartling offers integrations with the pioneers behind both neural networks and MT.

An adaptive NMT engine will constantly be growing its knowledge of your brand's content, voice, tone and overall style to deliver localized content that still captures the exact presentation and quality experience unique to your brand.

When to use Machine Translation?

Machine Translation is impressive technology and has come far from those early stages. Modern engines are now capable of competing with human translators, to a degree. In fact, we have a whole webinar dedicated explaining how content managers can incorporate MT into their translation process.

But in summary, when discussing MT we should be thinking along the lines of: "when do I hand off content to a human translator, and when do I let MT handle the task?"

The biggest factor will come down to content priority. Some complexities to consider might be:

Where is this content going?

What is the target audience?

What demand is this content fulfilling?

Can content be edited or revised after publication?

Are there any legal restrictions or strict brand guidelines surrounding this content?

High priority critical content, for example legal documentation or medical information, will require the expertise of a human translator. Due to the nature of this content, and the high priority of accuracy, rapid deployment is not always be the main goal. One mistake could lead to a cascade of potential issues.

Even launching a new landing page for your latest product is a perfect example of high priority content: this is where brand voice and style needs to shine, and companies should look to leverage the creative touch of a human translator.

From there we can drill down even further and determine whether or not MT is the right choice for each project. This will heavily depend on the complexities involved. For example:

Volume: If you are looking to translate as much content in as little time as possible, then MT can outpace human translators and bring otherwise massive projects into a more realistic time-table.

Simplicity: When translating repetitive information, with clearly defined language and terminology, MT can handle the repetitive and simple text.

Priority: Again, low priority content will be perfect for MT. For example, content that will be used internally might not require the same level of polish of a landing page for a new product.

It's as simple as asking yourself does this content make sense for machine translation? Do I need the creative input of a professional translator? Will I be able to correct my course of action, or once I hit send will that email exist, as it is, forever?

Finding the Right Process: MT Post Edit or Human One-Pass?

Machine Translation enables businesses to edge even closer to that perfect balance of moving content at a rapid pace and pushing to market as quickly as possible while reducing the time and capital spent on projects.

But, just because a project might fit into this criteria does not mean it will be ripe for MT -- and it doesn't mean that a human will never be involved in the process.

Whether or not the translation process will require internal review for every project is something that must be determined. In our experience, while content spends the most time during this process, only 4% of translations receive edits. So the question then becomes a determination of which path makes more sense:

For one of our largest customers, internal review was more than doubling project completion time, yet only 4% of translations were being changed-at a cost of almost $35 per change!

1. Should we rely on MT do the majority of the work and build in an internal check process with a human translator?

2. Should we simply let a human handle the translation with an automated quality check working to ensure consistency throughout the process?

For example, translating user reviews of a product might seem like the perfect option.

On one hand, there is a high volume of content, which is usually simple and repetitive, and might be too costly for a human translator.

On the other hand, the priority for this content might be too high to rely on just a machine translation service; prospects rely heavily on user reviews, and these may be critical to specific use-cases.

One potential strategy here could be a mix of both human and MT, with professional translators focusing on the most important reviews, and MT handling smaller or less prominent reviews.

Or, machine translation could be used to translate every single review, with a professional translated disclaimer at the top of the page highlighting this distinction.

The exact strategy to take will depend on the unique complexities and priorities of each individual business and project. That's where Smartling fits right in.

Smartling Lets You Decide

Leveraging the best machine translation is more about balancing the content to move low priority, high volume content as fast as possible.

This should, therefore, enable the human translators to focus on more important and crucial content.

With Smartling users can configure a Translation Workflow step to automatically assign content to the proper translation resource, whether that be a human translator or MT engine. Content can even be rejected, or assigned to a professional translator based on the complexity of the project.

For example, Lucidchart began leveraging Smartling to drastically improve their translation speed and agility, with the goal of rapid deployment to match their fast release cycles. Previous translation methods couldn't keep up and were slowing down the entire roadmap.

By injecting NMT into the process, Lucidchart can provide a consistent and cohesive localized product experience to their international users, without any negative impact on their product and release cycle.

With machine translation, Lucidchart managed to drop their translation time from 2 days to a mere 10 minutes, supporting their rapid release schedule. Smartling's professional translators can then replace or update any content at a later date if necessary.

The important part here is how flexible the platform can be. Smartling already provides deep integration with the numerous engines listed above, and will only continue to grow our automation functionality by helping users find the right MT service for their exact needs.

Where Will Machine Translation Go From Here?

In summary, machine translation has made huge strides since the very early days. Even 10 years back, with Google Translate pushing the concept mainstream, MT was far from perfect.

Modern solutions leveraging powerful machine learning are constantly pushing new boundaries and will only continue to improve overtime as their knowledge grows. However, there will always be a need for a personal touch at some point, and automation will not entirely replace human translators. Even the best machine translation engine will struggle with highly creative and complex content.

When it comes to localization, you're going to be faced with this question: should I use machine translation?

Human translators are the ones that will provide that extra level of creativity for the highest quality of work. Machine translation is all about getting the job done with a margin for error.

Want to chat with an expert? Give us a call, Smartling is here to help you move the world with words.

# data[].attributes.attachment_link.description
Chances are that most of us have some experience with machine translation in some way. Google Translate and similar tools have been around the web for quite some time. In fact, just yesterday I was using Google Translate because I sent a wedding gift to a friend in Spain, and received an email in Spanish - turns out they were having a problem with delivery!

# data[].attributes.attachment_link.title
7 Google Translate Mobile Features You Must Know

# data[].attributes.attachment_link.description
Google Translate's mobile app can do more than you might think. Here's a guide to all of Translate's features on Android and iOS.

# data[].attributes.attachment_link.readable_text
Google Translate's knowledge of more than 100 languages can help you in your daily workflow as much as it can help you on your next trip. The features below show how it can help you with entire documents or websites, or even your native tongue.

Translate websites to a different language

Google Translate can parse individual words and phrases, of course, but you can also translate entire websites into a chosen language.

Just type the entire URL of the website you want translated in the text box on the left side of Google Translate's home page. Make sure Detect language is selected, and click Translate. In a short time you'll see the website's translated homepage. All the site's pages will be translated as well as long as you continue to view the site within the Google Translate interface. You can toggle between the translation and original language using buttons on the Translate toolbar.

Translate documents

You can also translate Word documents, PDFs, and other file types in Google Translate.

Beneath the text box, click the Translate a document link. On the next page, select the source language or the Detect language option, and the language to which you want the document translated. Click the Browse button, select the file to upload it, and click Translate. A translated copy of the document will open in a new browser window.

Build your own phrasebook

Once you've gone to the trouble to translate a complex phrase, you want to remember it. Google Translate allows you to save any translation to your own personal phrasebook.

Click the star icon beneath any translated word or phrase to save it. You can review your saved translations at any time by clicking the silhouetted star icon on the Translate toolbar.

Improve your English

Most of us would never think to enter our own language in the Google Translate box, but the tool can actually help you learn more about your native tongue.

For any word entered into Google Translate, it provides a definition, the word class (noun, verb, etc.), synonyms, and examples of the word's use in a sentence. It works much like a combination dictionary and thesaurus, allowing you to school yourself on unfamiliar words and phrases and improve your written communications.

# data[].attributes.attachment_link.title
4 Google Translate features you'll use every day

# data[].attributes.attachment_link.description
Google Translate's knowledge of more than 100 languages can help you in your daily workflow as much as it can help you on your next trip. The features below show how it can help you with entire documents or websites, or even your native tongue.

# data[].attributes.attachment_link.title
Converting an Excel glossary (xls) to a Multiterm termbase (xdt)

# data[].attributes.headline
Converting Microsoft Excel file to SDL Multiterm

# data[].attributes.subject
Converting Microsoft Excel file to SDL Multiterm

# data[].attributes.headline
Older version of Multiterm, as a reference to observe the changes. Using termbases while translating in SDL Studio.

# data[].attributes.subject
Older version of Multiterm, as a reference to observe the changes. Using termbases while translating in SDL Studio.

# data[].attributes.attachment_link.title
Getting started with a termbase in SDL Trados Studio

# data[].attributes.body
-hitlist
- importing/exporting terms
- creating terms
- Termbase Wizard
-

# data[].attributes.attachment_link.title
How to manage terminology in SDL MultiTerm 2019

# data[].attributes.attachment_link.title
How to use the translation alignment tool in SDL Trados Studio 2019

# data[].attributes.attachment_link.title
Quick align example Trados Studio 2019 SR1

# data[].attributes.body
I've reviewed my old projects that I made in SDL Trados 2017 in order to refresh my memory.

# data[].attributes.headline
I already knew most of the features mentioned in this video, but I wanted to refresh my memory to be quicker during lessons.

# data[].attributes.subject
I already knew most of the features mentioned in this video, but I wanted to refresh my memory to be quicker during lessons.

# data[].attributes.attachment_link.title
How to translate a document in SDL Trados Studio 2019 translation software

# data[].attributes.headline
I was used to older versions of Trados, so I wanted to watch a video about new features before I move on.

# data[].attributes.subject
I was used to older versions of Trados, so I wanted to watch a video about new features before I move on.