| Pages in topic: [1 2] > | How do you assess translation quality without looking at the language? Thread poster: Phil Hand
|
|---|
Phil Hand 
China
Local time: 13:49
Chinese to English
Hello, Prozians. Merry Christmas to those who celebrate it, season's greetings to those who don't, and an early happy New Year to everyone!
I've been away because I have a new in-house job, but I'm back to seek the benefit of your collective wisdom. The title of my post is not a joke, believe it or not.
In my company we have a large team of in-house translators who are native speakers of Chinese, but who work mainly from Chinese to English. One of the things I would lik... See more Hello, Prozians. Merry Christmas to those who celebrate it, season's greetings to those who don't, and an early happy New Year to everyone!
I've been away because I have a new in-house job, but I'm back to seek the benefit of your collective wisdom. The title of my post is not a joke, believe it or not.
In my company we have a large team of in-house translators who are native speakers of Chinese, but who work mainly from Chinese to English. One of the things I would like to do is to try to improve quality across the whole team. One of the biggest problems they have had as a team is that for a long time there has been literally no-one in their organisation who is competent to assess quality. By this I mean no native English speakers, or any speakers with a good enough grasp to really read a document and grade its language. Another problem is far too much time spent thinking about the source text wording, and too little time thinking about communicating meaning in the target text.
My big idea is that I could kill these two birds with one stone by putting in place non-linguistic standards or assessment criteria. What I mean is ways of looking at the target text in use, to see whether it's fit for purpose.
Examples:
1) Customer satisfaction. This one is the most obvious, but it's problematic in our company, because many of our "customers" are company departments staffed by people whose English is not nearly as good as they seem to think it is. They just love to instruct us to use "their" English. Nevertheless, customer satisfaction is a vital measure.
2) Online comments. Some of our translated material is published for in-house consumption (it's a big company, over 100,000 people), or in customer-facing magazines. My thought is that unreadable material is unlikely to be read, so no-one will comment on it. Well-translated material should attract more readers, and more comments, so on average, the number of comments could be some kind of index of translation quality.
3) User feedback. Sometimes we translate computer system interfaces. User satisfaction might be correlated with translation quality (though of course there will be lots of other confounding factors as well).
Does anyone have any experience using this kind of non-linguistic quality measure? Any measures that you can recommend? Any comments on how to get useful feedback, or how to use quality indices once we've got them? Thanks for any comments you are able to offer. ▲ Collapse
| | | | Balasubramaniam L.
India
Local time: 11:19
Member (2006)
English to Hindi
+ ...
SITE LOCALIZER | Some thoughts | Dec 29, 2015 |
The problem with the yardsticks you have mentioned is that they will tell of problems much after the effects of a defective translation have done the damage, in other words, they are not preventive yardsticks. They won't be able to save you or your organization from translation-engineered disasters in time.
Since you are in a position to personally know the translators and become aware of their linguistic and educational backgrounds, as well the type of translation experience they h... See more The problem with the yardsticks you have mentioned is that they will tell of problems much after the effects of a defective translation have done the damage, in other words, they are not preventive yardsticks. They won't be able to save you or your organization from translation-engineered disasters in time.
Since you are in a position to personally know the translators and become aware of their linguistic and educational backgrounds, as well the type of translation experience they have, one way would be to assess the translators, instead of the translations, and distribute translation tasks (if you have any control on this) in such a way that the skills required for a task and the skills possessed by the translator implementing the task are better matched. This will yield better results than randomly picking any translator for any job. For example, a person with education in engineering, should do technical manuals, and a person with better command over English should do marketing material, just to take an example.
As you get to know your translators more, you will be able to do this best-fit strategy more efficiently.
Simultaneously, you should institute skill improvement programmes for all your translators so that they gradually upgrade their linguistic skills (in English).
[Edited at 2015-12-29 17:08 GMT] ▲ Collapse
| | | | Neptunia
Local time: 07:49
Italian to English
| interesting problem | Dec 29, 2015 |
The strategies you've described might work but I think the sample size would have to be vast to get real answers. Like you said - -too many possible confounding factors. The only suggestion I have would add more to the team's workload which is probably not desirable and I am not sure if it counts as "looking at the language." It involves some kind of a brief, informal, periodic Back-Translation Workshop in which you select a few lines or paragraphs from recently translated material and see if th... See more The strategies you've described might work but I think the sample size would have to be vast to get real answers. Like you said - -too many possible confounding factors. The only suggestion I have would add more to the team's workload which is probably not desirable and I am not sure if it counts as "looking at the language." It involves some kind of a brief, informal, periodic Back-Translation Workshop in which you select a few lines or paragraphs from recently translated material and see if the group can come up with a consensus of what it means in Chinese (or alternatively, compare individual translated versions). If you can find "good" ambiguous or nonsensical texts then at least the discussion might demonstrate the extent of the problem and drive home the necessity of preserving the sense of the text in the translation. Even though the translations wouldn't have names attached, you might find that soon no one would want their texts used as workshop samples. ha ha ha. ▲ Collapse
| | | | B D Finch
France
Local time: 07:49
French to English
+ ...
| My take on your assessment criteria | Dec 29, 2015 |
Merry Winter Solstice and a Happy New Year!
Phil Hand wrote:non-linguistic standards or assessment criteria. What I mean is ways of looking at the target text in use, to see whether it's fit for purpose.
Examples:
1) Customer satisfaction. Your customers who 'just love to instruct us to use "their" English' are trying to maintain their status, as self-proclaimed English-speakers, within their own company. Even if "customer satisfaction is a vital measure", beware; they might be pleased to see their own poor English repeated, but later become dissatisfied if their own customers raise complaints e.g. because of unclear or faulty assembly or safety instructions. What happens if customers are satisfied but don't realise that the material was badly translated?
2) Online comments. "Well-translated material should attract more readers, and more comments, so on average, the number of comments could be some kind of index of translation quality." Have you noticed the semi-literate standard of English produced by native speakers of English (especially trolls), on internet forums? This is less of a problem in fields where the readers a likely to have a high level of education and you are probably right for material relating to such fields. However, this might fail to identify major errors of translation, e.g. reversals of meaning or incorrect imputations of causality.
3) User feedback. I agree that there will be confounding factors as well as quality factors. One problem is that there is often a bias in favour of negative feedback unless feedback is positively invited. Clear and accurate instructions are often expected and not commented upon, while poor instructions are likely to give rise to accidents, product failures, complaints, ridicule or just queries.
[Edited at 2015-12-29 19:10 GMT]
[Edited at 2015-12-29 19:11 GMT]
| | |
|
|
|
The Misha
Local time: 01:49
Russian to English
+ ...
| I am afraid you are fighting the windmills here | Dec 29, 2015 |
Who is your output intended for anyway? If it's for the company's internal (or, broader, in-country) consumption to satisfy some regulatory mandate, the issue is moot since no one would know any different, nor care. If it is indeed intended for actual English-speaking audiences, then you are vastly outnumbered against the onslaught of Chinglish your colleagues must be producing, and it will probably matter little what you do or don't do.
As a disclaimer, I know no Chinese, nor do I ... See more Who is your output intended for anyway? If it's for the company's internal (or, broader, in-country) consumption to satisfy some regulatory mandate, the issue is moot since no one would know any different, nor care. If it is indeed intended for actual English-speaking audiences, then you are vastly outnumbered against the onslaught of Chinglish your colleagues must be producing, and it will probably matter little what you do or don't do.
As a disclaimer, I know no Chinese, nor do I have any experience with any Chinese translations into English, aside from the anecdotal (such as fook yee restaurants or some unfortunate user manuals for cheap Chinese-made goods). However, if my extensive experience with translations into English produced by nonnative residents of Moscow and other such fine places is any indication, the predominant majority of them are not suitable for consumption by unsuspecting English-speaking audiences without major rewriting. And I would assume that Russian is much less "alien" to English than Chinese.
Long story short, I am afraid there's precious little you could do except for fixing one bad translation at a time, drop by drop into that bottomless bucket.
Regardless, good luck with your valiant effort. ▲ Collapse
| | | | | Look at it backwards | Dec 29, 2015 |
Phil,
Let's see you found an assessment method to look at the translations produced by your company.
What is the next step?
How are you going to improve it?
If there are no native English speakers on that in-house team, what good it is to find out exactly what the current quality is? It seems to me you are already suspecting it is not good, so what difference does it make whether it is 2 stars out of 10 or 5 out of 10? (I just made this up.)
I am not sure how... See more Phil,
Let's see you found an assessment method to look at the translations produced by your company.
What is the next step?
How are you going to improve it?
If there are no native English speakers on that in-house team, what good it is to find out exactly what the current quality is? It seems to me you are already suspecting it is not good, so what difference does it make whether it is 2 stars out of 10 or 5 out of 10? (I just made this up.)
I am not sure how many translators are involved, but I don't think there is a way around assessing their individual abilities. After all, they are the ones producing the translations, no matter what processes are around them. You could check random samples, or subject them to the same test, whatever, but I have a feeling that you need to take it down to the level of individuals.
And, if you know you would need to bring in native or native-level editors anyway, why don't you push for bringing them in now? They could participate in the assessments and building a better quality control framework.
So, what I am saying is that perhaps you should attack the problem from the "other end", take what you would do at the end, and do it first.
I hope I am making sense... ▲ Collapse
| | | | | Mission (almost) impossible | Dec 30, 2015 |
I, too, am trying to improve the quality of non-native translations by providing detailed feedback as an editor and by slowly writing a white paper on typical translation mistakes in my field and language pairs. However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team o... See more I, too, am trying to improve the quality of non-native translations by providing detailed feedback as an editor and by slowly writing a white paper on typical translation mistakes in my field and language pairs. However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence. ▲ Collapse
| | | | jyuan_us
United States
Local time: 01:49
Member (2005)
English to Chinese
+ ...
| No employers in China has that knowledge | Dec 30, 2015 |
Anton Konashenok wrote:
However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence.
No one out of the translation circle knows that translation should be done by those native in the target language.
| | |
|
|
|
| Assess customer satisfaction | Dec 30, 2015 |
Merry Christmas! (Not merry winter solstice)
Phil Hand wrote:
Does anyone have any experience using this kind of non-linguistic quality measure? Any measures that you can recommend?
I am a native Indonesian and live in Indonesia. I use below criteria to assess my own translation, beside linguistic measure.
1) Wait for user feedback and/ or critics and provide post-sale service, such as, free consultation and "adjustment" for 1 month after delivery. This step is not only to ensure customer satisfaction, but it is also useful to examine translation quality whether your translation is satisfying.
2) The same person uses your service for more than 3 individual occasions in the same year. Your translation is more useful when it retains your customers or it converts a client to be a customer with long relationship.
3) He or she becomes your regular customer (at least 5 years).
4) Key speaker of an International event, such as an International seminar or workshop, use your translation AND he or she is happy with his or her audience reaction. When your client happily tell you about their reaction, it is a good news for you.
5) Your translation is published by at least an International scientific journal head-quartered in English-speaking countries without many "adjustment". This is a sign that your translation meets International criteria of good translation.
I composed this criteria for my self a long time ago from my experience of rise and fall cycle in this field.
[Edited at 2015-12-30 03:16 GMT]
| | | | ..... (X)
Local time: 14:49
| TAUS / Microsoft | Dec 30, 2015 |
Hi Phil,
I would recommend looking into some of the research TAUS and Microsoft are doing. I think TAUS does a lot of work and research in the area of translation quality assessment (i.e. search 'taus translation quality assessment' or 'taus dynamic quality framework').
A... See more Hi Phil,
I would recommend looking into some of the research TAUS and Microsoft are doing. I think TAUS does a lot of work and research in the area of translation quality assessment (i.e. search 'taus translation quality assessment' or 'taus dynamic quality framework').
Also, your examples reminded me of a presentation I saw by Microsoft on P3 (Post-Publish Post-Edit). While not exactly the same as what you are doing (as you are not using MT), I think some of the principles are similar (i.e. looking at user feedback / use statistics) and might apply to what you are trying to do.
Kevin
[Edited at 2015-12-30 03:03 GMT]
[Edited at 2015-12-30 13:21 GMT] ▲ Collapse
| | | | Kwok Wa LAM
Hong Kong
Local time: 13:49
English to Chinese
+ ...
| If you pay peanuts, you will get monkeys | Dec 30, 2015 |
jyuan_us wrote: Anton Konashenok wrote:
However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence. No one out of the translation circle knows that translation should be done by those native in the target language.
I think the major problem lies in extremely low pay for translators in Mainland China. There are plenty of talented Chinese on the mainland who have professional English proficiency that is more than good enough to handle the bulk of translation jobs. Unfortunately, those talents will not be appropriately compensated as being a translator. The industry in China fails to attract and retain the good ones. Hence forth, they end up with a house full of mediocre or even sub-standard producers. I once searched translation agency websites and found tones of mistakes there. I took the time to write one of the big ones to give some feedback but I think no one there even bothers.
Phil, I can definitely appreciate your situation. I bet the staff turnover rate is also very high. You have a tall order. I personally got a client from Mainland because he was not satisfied at all with the translation quality produced by mainland translation companies. He used his personal network and found me at the end. So, the cut-throat pricing strategy and less than minimum wage pay to translators is not a sustainable business model in the long run.
| | | | Phil Hand
China
Local time: 13:49
Chinese to English
TOPIC STARTER
Thanks, everyone. As ever, I am very impressed by the experience and knowledge that you all have to contribute. Particularly thanks to Kevin, for those links, and to Katalin for that very well put inversion.
As Katalin suggested, in the end I'm going to have to just get down to the hard slog of building up the skills of the team. I'm trying to look for some ways to ease and simplify that process because: (1) I'm a translator, not an educator. I'll do education if that's what it tak... See more Thanks, everyone. As ever, I am very impressed by the experience and knowledge that you all have to contribute. Particularly thanks to Kevin, for those links, and to Katalin for that very well put inversion.
As Katalin suggested, in the end I'm going to have to just get down to the hard slog of building up the skills of the team. I'm trying to look for some ways to ease and simplify that process because: (1) I'm a translator, not an educator. I'll do education if that's what it takes, but training is neither my top skill nor my first love. (2) My colleagues are already very well trained (most have an MA in translation), and very busy. More importantly, they are smart and hard-working, and I am pretty confident that if I can point them in the right direction, they'll get where we need to go themselves. At the moment it's a weird brew of institutional and cultural constraints that is holding them back.
So in part what I'm looking for is ways to empower my colleagues. At the moment, when a senior manager sends back their translation with a note saying, "I think this phrase should be 'talents cultivation'," they scuttle to comply. I'd like them to be in a position to say, actually, 'workforce development' is a much more standard phrase - and we know because our HR people found working with documents 20% easier when we write it like that.
That's the dream, though I'm far from sure that it's really possible. A TAUS document puts it rather well: "Quality is when the buyer or customer is satisfied, yet quality measurement in the translation industry...is managed by quality gatekeepers...who have specific evaluation models." When the translators are battling deeply flawed evaluators, it's hard to get anything done, so more than anything, I want to equip them with the tools to find and create their own models/standards, which they can take to the evaluators and use to persuade them. Because all the training in the world can't help when every piece of idiomatic English you try to deliver just gets thrown back in your face. ▲ Collapse
| | |
|
|
|
| Why not hire a trained professional to do a professional job? | Dec 30, 2015 |
Surely the person who can assess the quality of translations is someone who is trained in that profession. I would think that you need someone who can also assess the translation itself because there are two main components of a translation: that all important meaning is rendered, and that it reads smoothly and comprehensively in the target language. Someone trained in translation, hopefully with experience, who has native quality mastery of the target language, should be doing this assessing.... See more Surely the person who can assess the quality of translations is someone who is trained in that profession. I would think that you need someone who can also assess the translation itself because there are two main components of a translation: that all important meaning is rendered, and that it reads smoothly and comprehensively in the target language. Someone trained in translation, hopefully with experience, who has native quality mastery of the target language, should be doing this assessing. Otherwise you are going about it blindly, hoping that the output is ok, but without ever being able to be sure. ▲ Collapse
| | | | | style guides and glossaries could be very helpful | Dec 30, 2015 |
How about client-specific style guides and glossaries?
| | | | Heinrich Pesch
Finland
Local time: 08:49
Member (2003)
Finnish to German
+ ...
| Mission impossible? | Dec 31, 2015 |
The one thing I use when assessing a translation in a foreign language is Word spelling checker. If Word displays a lot of red lines under words of the source language this could be a sign for careless work. Then I test some of the offensive words, copy them into Google and whatch what it makes out of it.
Of course this is only part of the task you have in mind. But assessing the meaning of a translation without analysing the language is quite impossible. Back translation by MT is one appr... See more The one thing I use when assessing a translation in a foreign language is Word spelling checker. If Word displays a lot of red lines under words of the source language this could be a sign for careless work. Then I test some of the offensive words, copy them into Google and whatch what it makes out of it.
Of course this is only part of the task you have in mind. But assessing the meaning of a translation without analysing the language is quite impossible. Back translation by MT is one approach, but that too requires someone to actually read the stuff. ▲ Collapse
| | | | | Pages in topic: [1 2] > | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » How do you assess translation quality without looking at the language? | TM-Town | Manage your TMs and Terms ... and boost your translation business
Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.
More info » |
| | Trados Business Manager Lite | Create customer quotes and invoices from within Trados Studio
Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.
More info » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
Translation news
