Pages in topic:   [1 2] >
Some issues with how grading is handled
Thread poster: Łukasz Gos-Furmankiewicz
Łukasz Gos-Furmankiewicz
Łukasz Gos-Furmankiewicz  Identity Verified
Poland
Local time: 09:38
English to Polish
+ ...
Aug 26, 2014

Description:

Those entries will now be tagged and rated individually by interested parties with relevant experience to determine which entries will advance to the finals round.


What it really is:

Any number of these entries can be tagged and rated for writing quality and translation accuracy by registered users, including felllow participants, who state that they work in the relevant pair or — for quality of writing only — have listed the target language as their native language.

What's the relationship between the 1-5 star ratings and likes and dislikes, anyway?

Plus, I don't think it's good that native speakers who don't work in certain pairs can rate writing quality without even knowing the source language. Those ratings are essentially based on whether someone likes the sound of the translated text with no reference to the source. I realize that kind of approach is somewhat popular in the Anglosphere these days (along with some other aspects of target-centrism and source neglect), but it's quite unprofessional really. Writing quality cannot be judged without reference to the original.

Actually, non-speakers of source languages are allowed to vote for the best entry. For example, without knowing Dutch I can grade Dutch to Polish translations.

That's just wrong.


 
José Henrique Lamensdorf
José Henrique Lamensdorf  Identity Verified
Brazil
Local time: 05:38
English to Portuguese
+ ...
In memoriam
It's restricted Aug 26, 2014

Łukasz Gos-Furmankiewicz wrote:

Actually, non-speakers of source languages are allowed to vote for the best entry. For example, without knowing Dutch I can grade Dutch to Polish translations.


Łukasz, in your example above, you can rate the quality of Polish writing, however you can't rate the quality of NL > PL translation.

Someone might have crafted some brilliant text in Polish, albeit failing to maintain any loyalty to the source text.

Once I was translating a book crammed with quotes. Whenever a widely published translation was available online, provided it was acceptable, I preferred to use it instead of my own. This would make it easier to find for readers who searched for that entire work.

One of these quotes, a relatively mundane one, had been translated by no other than Machado de Assis, one of the greatest icons in Brazilian literature of all times. I found that translation, and had a really hard time to spot the actual paragraph, as Machado had turned that worldly passage into some outer space literary experience, totally unrelated to the subject matter at hand in that part of the book.

While I had to put my own translation there, I must admit that Machado's version belongs among masterpieces. I guess that's the intent in the Proz contest: possibly spot literary geniuses, though these won't get any translation prizes.


 
Mohamed A. Moustafa
Mohamed A. Moustafa  Identity Verified
United States
English to Arabic
+ ...
on the sidelines Aug 26, 2014

Just on the sidelines of your discussion: The only benefit of this contest so far is to see some of what is going on in the market! It is a mess in a way or another. It also tells something about the strengths and weaknesses of the "translation community" and their choices.

 
Christel Zipfel
Christel Zipfel  Identity Verified
Local time: 09:38
Member (2004)
Italian to German
+ ...
Did you ever hear of a contest where you can rate your fellow participants? Aug 26, 2014

Łukasz Gos-Furmankiewicz wrote:


Any number of these entries can be tagged and rated for writing quality and translation accuracy by registered users, including felllow participants, who state that they work in the relevant pair or — for quality of writing only — have listed the target language as their native language.



I didn't. Therefore in my opinion, any effort in submitting your own translation and/or rating others' is not worthwhile. At least a fair rating should be assured, and it isn't under these circumstances.


Thomas T. Frost
 
Elizabeth Tamblin
Elizabeth Tamblin  Identity Verified
United Kingdom
Local time: 08:38
French to English
It's all most peculiar! Aug 26, 2014

By definition, if someone enters a contest, they do so with the desire to win, surely?

In that case, why would a contestant vote for someone else's work? Well, when you look at the situation at the moment, it's obvious that most of the voting is negative.

To be honest, I wish I had never entered, as it seems to be a very flawed (and confusing) voting system.

[Edited at 2014-08-27 12:15 GMT]

[Edited at 2014-08-27 12:16 GMT]


Thomas T. Frost
 
Łukasz Gos-Furmankiewicz
Łukasz Gos-Furmankiewicz  Identity Verified
Poland
Local time: 09:38
English to Polish
+ ...
TOPIC STARTER
... Aug 27, 2014

José Henrique Lamensdorf wrote:

Łukasz, in your example above, you can rate the quality of Polish writing, however you can't rate the quality of NL > PL translation.


I can still pick whichever one of them I consider the best in that vote.

Someone might have crafted some brilliant text in Polish, albeit failing to maintain any loyalty to the source text.


Yes, and what point grading that? What point having jurors who understand only one language, so they don't even know where the writing comes from and can only keep guessing?

While I had to put my own translation there, I must admit that Machado's version belongs among masterpieces. I guess that's the intent in the Proz contest: possibly spot literary geniuses, though these won't get any translation prizes.


But that's not the point of any sort of translation contest.

... As convenient as it may be for publishers and subservient translation agencies to have the work done by a nameless guy in the shadows with no proper pay or credit.

[Edited at 2014-08-27 02:33 GMT]


 
Josephine Cassar
Josephine Cassar  Identity Verified
Malta
Local time: 09:38
Member (2012)
English to Maltese
+ ...
wrong marking too Aug 27, 2014

I wonder what happens when a person marks a word as being wrong when it is isn't and others agree it isn't-e.g. one is told a word is in the singular so verb has to be in the singular and you answer that it can be both singular + plural and provide examples, I do not know if it is then adjusted. Another thing is where the particular translation happens to be-at the top of a long, long list or towards the end, as people get tired of checking and are inconsistent-marking Like for a certain express... See more
I wonder what happens when a person marks a word as being wrong when it is isn't and others agree it isn't-e.g. one is told a word is in the singular so verb has to be in the singular and you answer that it can be both singular + plural and provide examples, I do not know if it is then adjusted. Another thing is where the particular translation happens to be-at the top of a long, long list or towards the end, as people get tired of checking and are inconsistent-marking Like for a certain expression in one translation, but not in another translation which has exactly the same word/sCollapse


 
Sheila Wilson
Sheila Wilson  Identity Verified
Spain
Local time: 08:38
Member (2007)
English
+ ...
Some things I think I know about marking Aug 27, 2014

Everyone is free to agree or disagree with likes or dislikes - in other words you can see the lot - nothing is hidden. Anyway, they're only to be used as a guide, and for feedback to entrants. It's the vote that counts.

You see a random order. I'm not sure if it changes every time you look at it but it certainly isn't the same order for everyone.

The red stars you see are just your own personal marks. If they're grey, you haven't voted. You have no idea what others have
... See more
Everyone is free to agree or disagree with likes or dislikes - in other words you can see the lot - nothing is hidden. Anyway, they're only to be used as a guide, and for feedback to entrants. It's the vote that counts.

You see a random order. I'm not sure if it changes every time you look at it but it certainly isn't the same order for everyone.

The red stars you see are just your own personal marks. If they're grey, you haven't voted. You have no idea what others have voted.

The two categories for writing quality and translation quality are separate. So why not let target language speakers comment on the target? Obviously, if it isn't a faithful translation then those who can vote in both categories will eliminate it with their poor votes. I don't know how the votes are analysed but I'm sure that translation quality figures high in the equation.


Hopefully if there's anything wrong with the above then staff will correct me.
Collapse


 
Łukasz Gos-Furmankiewicz
Łukasz Gos-Furmankiewicz  Identity Verified
Poland
Local time: 09:38
English to Polish
+ ...
TOPIC STARTER
... Aug 27, 2014

Sheila Wilson wrote:

The two categories for writing quality and translation quality are separate.


I'm sorry, but I disagree, Sheila. In short, I believe that good writing is a bad translation of bad writing.

So why not let target language speakers comment on the target?


Because they have no way of assessing the nuance. They will never know as simple things as whether your archaisms or non-standard syntax or any idiolect features are your own or the author's. Modern translation paradigm, unfortunately, has sunk to such low levels intellectually that this distinction is not even grasped, let alone observed.

Neither will they know if their favourite metaphor or hyperbole or specific melody when reading out loud is something you really deserve credit for or just something the author created and you only managed not to botch, so it works both ways.

Obviously, if it isn't a faithful translation then those who can vote in both categories will eliminate it with their poor votes. I don't know how the votes are analysed but I'm sure that translation quality figures high in the equation.


The whole thing is part of a larger problem: monolingual assessment and review of translation and the notion that the writing is separate from the translating (as though translation concerned itself with the substance and sense only and never the form).

Josephine Cassar wrote:

I wonder what happens when a person marks a word as being wrong when it is isn't and others agree it isn't-e.g. one is told a word is in the singular so verb has to be in the singular and you answer that it can be both singular + plural and provide examples, I do not know if it is then adjusted. Another thing is where the particular translation happens to be-at the top of a long, long list or towards the end, as people get tired of checking and are inconsistent-marking Like for a certain expression in one translation, but not in another translation which has exactly the same word/s


Inconsistency of numbers or genders is a perfect example of decisions inherited from the original author, which readers who have no way of comparison with the original can't even begin to grasp.

One could go on forever: registers, jargon words, archaism (already mentioned before), idiosyncratic features, suboptimal logical structure, pedantic or sloppy syntax and more.

If you have a letter written by someone from a generation or two before you, using language you don't normally use but are sufficiently familiar with, you will want to convey that same feeling in the translation. On the other hand, an ignorant layperson or even a modern professional linguist will look at it, pull a face and say: 'This is so not modern Ixian! It sounds like something from last century.' This is the sort of intellectual apathy of our discipline these days — the same which worships feedback from clueless clients who don't know what they're talking about (which also applies to quite some QA departments).

[Edited at 2014-08-27 11:01 GMT]


 
Maria Teresa Borges de Almeida
Maria Teresa Borges de Almeida  Identity Verified
Portugal
Local time: 08:38
Member (2007)
English to Portuguese
+ ...
Well, well Aug 27, 2014

I have been taking part in Proz.com Translation Contests since I became a Prozian in 2007. They are quite fun, occasionally challenging and a nice break from our normal routine. I don't take this kind of contests as seriously as you (apparently) do, but just for what they are.

Nonetheless, I do agree with you all that contestants should not rate someone else's work. I consider myself relatively fair and impartial, but not to the point where I would hinder myself…


Thomas T. Frost
 
José Henrique Lamensdorf
José Henrique Lamensdorf  Identity Verified
Brazil
Local time: 05:38
English to Portuguese
+ ...
In memoriam
Sportsmanship Aug 27, 2014

Teresa Borges wrote:

I have been taking part in Proz.com Translation Contests since I became a Prozian in 2007. They are quite fun, occasionally challenging and a nice break from our normal routine. I don't take this kind of contests as seriously as you (apparently) do, but just for what they are.


Definitely!

This time I also submitted translations from one language that I speak but don't translate into my working languages. I would have done it from another language having the same status, however even people who work in that language found the source text too difficult, so I gave up.

The paradigm here was a soccer game I saw once, where all the players were Formula One racing drivers.

My intent was to get some candid feedback on my performance while translating from these languages. When I speak them, people say I do it very well, however I always wonder if they are not merely being nice, in exchange for my effort to speak their language.

No need to worry, folks. I am an old dog; I won't start translating professionally from these languages now, since I know exactly how much more I would have to study them before I try doing it. Furthermore, I won't be scoring/evaluating any contest entries in these either.


 
Phil Hand
Phil Hand  Identity Verified
China
Local time: 16:38
Chinese to English
Go easy, bro Aug 27, 2014

Łukasz Gos-Furmankiewicz wrote:

But that's not the point of any sort of translation contest.

I'm with Teresa. The point is not to see the Proz competition as a "A Translation Competition Which Must Be Run For The Discovery Of Best Translation," but rather as something which translators might do for fun, which is connected to their own work, but not necessarily a direct measure of it.

After all the arguments on here, surely one of the things we know for certain is that translation is a plural, varied activity, with a lot of different approaches. The competition allows for translators to try out and discuss these texts in at least three ways. First, there is monolingual commentary on the source text (or at least there was last time). That allows us to talk about the reading & comprehension side. Next, there's translation and grading of translation. That's an overall discussion, which can bring in all elements. And there's comment on the quality of the writing, the other big element in translation. (My model goes: reading+writing=translation).

I think that's pretty good! Comprehensive and inclusive. It doesn't have to be a perfect measure of translation quality - indeed, there is no such thing.


 
Orrin Cummins
Orrin Cummins  Identity Verified
Japan
Local time: 17:38
Japanese to English
+ ...
... Aug 27, 2014

Perhaps the problem is the word "contest"? Because I have to agree with Łukasz here: if we are going to call this a contest, it shouldn't have entries graded by other "contestants." That just doesn't make sense in any common usage of the word "contest."

If a contest is really "an event in which people try to win by doing something better than others" as Merriam Webster would have us believe, then it seems there must be a system in place to assure that each entrant has a fair
... See more
Perhaps the problem is the word "contest"? Because I have to agree with Łukasz here: if we are going to call this a contest, it shouldn't have entries graded by other "contestants." That just doesn't make sense in any common usage of the word "contest."

If a contest is really "an event in which people try to win by doing something better than others" as Merriam Webster would have us believe, then it seems there must be a system in place to assure that each entrant has a fair and equal chance of winning, based on his or her own ability. Peer-grading is inherently contrary to that ideal.

Maybe a better phrasing would be "translation event" or something? By calling it a "translation contest," we are implying certain things about its structure that do not appear to be true in practice.
Collapse


Thomas T. Frost
 
Łukasz Gos-Furmankiewicz
Łukasz Gos-Furmankiewicz  Identity Verified
Poland
Local time: 09:38
English to Polish
+ ...
TOPIC STARTER
... Aug 27, 2014

My problem is more with the 'interested parties with relevant experience', as if some serious peer review process were taking place, while in reality anybody can vote.

Regarding the 'writing quality', I did assign different grades for that, sometimes higher and sometimes lower than for translation accuracy and very rarely on the exact same level in a scale from 1 to 5 (and yeah, I did grade other contestants). I'm not saying that translation quality is a homogenous blob without any
... See more
My problem is more with the 'interested parties with relevant experience', as if some serious peer review process were taking place, while in reality anybody can vote.

Regarding the 'writing quality', I did assign different grades for that, sometimes higher and sometimes lower than for translation accuracy and very rarely on the exact same level in a scale from 1 to 5 (and yeah, I did grade other contestants). I'm not saying that translation quality is a homogenous blob without any variety of aspects. However, monolingual review, regardless of the current fad in the agency world, just doesn't work.
Collapse


 
Orrin Cummins
Orrin Cummins  Identity Verified
Japan
Local time: 17:38
Japanese to English
+ ...
... Aug 27, 2014

Łukasz Gos-Furmankiewicz wrote:

My problem is more with the 'interested parties with relevant experience', as if some serious peer review process were taking place, while in reality anybody can vote.

Regarding the 'writing quality', I did assign different grades for that, sometimes higher and sometimes lower than for translation accuracy and very rarely on the exact same level in a scale from 1 to 5 (and yeah, I did grade other contestants). I'm not saying that translation quality is a homogenous blob without any variety of aspects. However, monolingual review, regardless of the current fad in the agency world, just doesn't work.


One problem begets the other, though.

ProZ has no way of really verifying anyone's ability to do anything. All it goes by is what people have entered on their profile, which means that:

Any number of these entries can be tagged and rated for writing quality and translation accuracy by registered users, including felllow participants, who state that they work in the relevant pair or — for quality of writing only — have listed the target language as their native language.


So what you have then is a contest operating on the honor system. I'd like to think that the members of this site hold themselves to a somewhat higher standard than the average user population on the Internet, but that doesn't mean that everyone operates honestly (see KudoZ for plenty of proof of that).

Just because someone declares a language pair on their profile does not mean that they are qualified to translate from or into those languages. I could go add the Swahili to Russian pair right now on my own profile but it doesn't make it so. In fact there are plenty of people who claim to translate into English who I wouldn't want to pay to write my grocery list, much less a complex technical text.

Unfortunately, the alternative is probably not to have the translation contests at all, since ProZ doesn't have the staff resources to provide impartial judges in every language pair with a source text in the contest. I'm not saying that I have a better solution--I am saying that to call it a "contest" is stretching the meaning of the word just a bit.

Ideally, maybe ProZ should offer some KudoZ points or a free month of membership or something to site members who are not participating in the current contest but are willing to objectively judge the entries in their primary language pairs. And I agree that it makes little sense to monolingually judge one side of a translation...what is to be gained from that? A highly polished turd is still a turd.


 
Pages in topic:   [1 2] >


To report site rules violations or get help, contact a site moderator:

Moderator(s) of this forum
Lucia Leszinsky[Call to this topic]

You can also contact site staff by submitting a support request »

Some issues with how grading is handled






Wordfast Pro
Translation Memory Software for Any Platform

Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

Buy now! »
TM-Town
Manage your TMs and Terms ... and boost your translation business

Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.

More info »