Amaterasu Blog

12Nov/1023

Translation Quality Experiment

Those of us who know Japanese, or work with someone who does, tend to care pretty deeply about "translation quality." But it's hard to convince a monolingual English speaker that translation quality is even an issue, much less a serious one. Thus we continue to see people trying to "translate" (lol) visual novels despite not knowing the language well enough, or worse yet, by using a machine. The only real way to prove that this does in fact matter would be to have them read an entire VN multiple times with different translation qualities, but of course this is unfeasible. Instead, I used some machines and asked a number of humans with varying skill levels to translate a pair of brief excerpts from MuvLuv Alternative. And the results do seem to convincingly show everything I wanted to show.

The translations below are completely unaltered from what the individual humans and machines in question gave me (except Amaterasu's of course, since that's a whole group). Some of what you see here is going to be a result of individual quirks, but I believe there's still a lot of interesting statements we can make based on them. None of the humans (including the one who edited the machines) were allowed to ask other humans for help, though they could ask me for context and consult non-human resources like dictionaries.

Also, there is a huge contrast between the results for the two excerpts, so it's definitely worth at least skimming both sets.

================================================================================
================================================================================
Formal Excerpt (after the unit stops to discuss a VIP with motion sickness)
================================================================================
================================================================================

Original Japanese:
【ウォーケン少佐】「軍曹、訓練部隊のファストエイドキットは、国連E規定に準拠したものか?」
【まりも】「はっ。米軍と同じものを使用しております」
【ウォーケン少佐】「白銀訓練兵、スコポラミンは、もう投与したのか?」
【武】「箱根を出る前に飲んでいただきました。3錠です」
【ウォーケン少佐】「限界量のスコポラミンが効かない……か」
―あぁもうっ! 分析はどうでもいいって!
衛士は専門的な医学教育なんか受けてないだろッ!

Amaterasu (Ixrec plus editors and QCers and everything):
【Maj. Walken】「Sergeant, do the cadets' first-aid kits comply with UN regulation E?」
【Marimo】「Affirmative. They have the same equipment as yours.」
【Maj. Walken】「Cadet Shirogane, have you given her scopolamine already?」
【Takeru】「Yes, I gave her three pills before we left Hakone.」 //literally "had her drink/swallow"
【Maj. Walken】「Even the maximum scopolamine dosage isn't enough......」
―Aaah, come on! We're just going in circles! //literally "This analysis doesn't matter!" by which he means "You're thinking too hard!"
None of us Eishi get specialized medical training anyway!

I added comments here to my translation about what a few lines literally mean in order to make it clear why the more literal translations below turned out the way they did (many of them used the word "analysis," for instance). As will soon be obvious, I like to adapt my translations quite a bit.

====================
Machine Translators
====================

Google (horrible machine translation):
Major Walken - "Sergeant Fasutoeido force training kit, the United Nations what can we comply with E?"
- Marimo "hahaha. We use the same U.S. military"
Major Walken - "Silver and training soldiers, scopolamine, or another of the administration?"
[Wu] "before leaving the Hakone Itadakimashita drink. The three tablets"
Major Walken - "... limit the amount of scopolamine were ineffective or"
- Amoutsu Oh! Analysis What Whatever!
You will not get something 衛士 Tsu professional medical education?

Babel Fish (slightly less horrible machine translation):
“As for the fast aid kit of the sergeant and the training unit, those which conform to the United Nations E stipulation?”
“[tsu]. We use same ones as the US military”
“The silver training soldier, as for scopolamine, already prescribing?”
“Before coming out of Hakone, you drank. They are 3 locks”
“The scopolamine of the limited quantity is not effective ......?”
- The [a] [a] already the [tsu]! Analysis how it is, calling!
[ro] [tsu] which as for the defense loyal retainer special medical indoctrination something is not received and is!

Honyaku (respectable by machine translation standards):
【Major war Ken】"Are a sergeant, a fast aid kit of the training corps things based on United Nations E rule?" "
【まりも】." I use a thing same as the United States Armed Forces
【Major war Ken】Do you "already administered the silver training soldier, the scopolamine?" "
【武】 I "had you drink before leaving Hakone". It is three tablets
【Major war Ken】"The scopolamine of the limit quantity does not work"……か
- Oh, already!  There is the analysis any way, and say!
The military escort does not take professional medical education!

It should be quite obvious that the machines themselves suck. While you can guess what half of the lines are trying to say, it's virtually impossible to do so with any accuracy or confidence, and sometimes (as shown with the second to last line) you won't have the slightest idea what's going on.

Edited Machine Translation (working only from the three machine translations above):
【Major Walken】 "Sergeant, are the first aid kits you use in compliance with the United Nations E standard."
【Marimo】 "We use the same issued to the American Army"
【Major Walken】 - "Shirogane, have you administered some scopalmine?"
【Takeru】 "Before heading to Hakone, I had a drink and took three tablets."
【Major Walken】 - "Isn't taking such a small amount ineffective?"
Oh? It sounds like he knows what he's talking about.
Does he really understand this stuff? It sounds like he has medical knowledge.

The idea behind using machines in a translation is that a competent editor can apparently work out what they should say and turn it into legible English. To such editors' credit, they can certainly make perfectly natural and readable English, but it's also going to be incredibly inaccurate. As you can see, four of the seven lines in this edited machine translation are almost the complete opposite of what the Japanese actually meant. Even in the top three, the usage of "some" is dubious and Marimo's はっ (meaning Sir/Yes/Yessir/Affirmative) was completely lost.

====================
Student Translators
====================

Human fluent in English but still learning Japanese (who's familiar with MLA):
【Major Walken】「Sergeant, do the Training Corps' first aid kits conform to the UN Type E ones?」
【Marimo】「Yes, we use the same ones as the US Army.」
【Major Walken】「Training Corp Shirogane, have you administered Skopolamin?」
【Takeru】「I gave her some before we reached Hakone. Three pills.」
【Major Walken】「So the maximum dosage didn't work, huh...」
―For God's sake! We don't have time to analyse this shit!
Eishi aren't even taught this medical stuff!

Human fluent in English but still learning Japanese (who knows nothing about MLA):
[Uooken Lieutenant commander] Sergeant, those training corps first aid kits, did you really distribute them conforming to the united nations E regulation?
[Marimo] Yes sir! The same as the US army does.
[Uooken Lieutenant commander] Silver training soldier, have you finished administering the skopolamin?
[Soldier] Yes, we administered a box of it before going out, 3 tablets. (wtf)
[Uooken Lieutenant commander] That isn't enough for skopolamin to be effective...
-ahhh geez! The analysis said it would be fine!
I don't think soldiers learn that in technical medical science training!

Both of these students made at least one glaring mistake, but they got the majority of it right, and their output is perfectly readable while not butchering any plot-critical aspect of the scene. Most interesting are all the errors that the student unfamiliar with MLA made which the other one did not.

====================
Fluent Translators
====================

Human fluent in Japanese but not a native English speaker:
"Sergeant, do the first aid kits of the training unit conform to the E-code of the United Nations?"
"Yes, sir. We are using what the US troops use."
"Private Shirogane. Have you already administered Skopolamin?"
"I had her take 3 pills before we left Hakone."
"Limit dose of Skopolamin is not doing the job, huh?"
Man, I don't care about the analysis!
Pilots aren't medical experts!!

Human (other than Ixrec) fluent in Japanese and a native English speaker:
Wolken: Sergeant, do your brigade's first aid kits conform to UN regulations?
Marimo: Yessir. They're the same as you'll find in the US Army.
Wolken: Trainee Shirogane, have you already given her a dosage of Scopolamine?
Takeru: I administered three doses before we left Hakone, sir.
Wolken: So even up to the maximum safe dose isn't helping... hmm.
Come the fuck on! We're not doctors here, we're pilots! There's more important shit to worry about than how many meds she took!

These two translators are perfectly competent, so the point in showing their work is to demonstrate that just because they're both good doesn't mean their output will look remotely similar. While the basic ideas being expressed are the same, countless details of how the lines are worded vary between mine and these two. Also note that the first of these two is non-native English, and thus uses some questionable wording (mainly "limit dose") but nothing harmful. On the other hand, the second is native but rather personalized (most obvious in the last line, which got adapted into something notably less formal than what any of the other versions had). Whether any of this is good or bad is subjective, but the differences are certainly big enough to make style an issue worthy of debate.

================================================================================
================================================================================
Informal Excerpt (after Yuuko kisses Takeru out of joy over the latest plot development)
================================================================================
================================================================================

Original Japanese:
【夕呼】「あたしって年下に本気になれない性格だから。欲情しないでね」
【武】「あ、当たり前じゃないですか!!」
【夕呼】「…………その割には男全開なのね……」
【武】「どこ見てるんですか!?」
【夕呼】「まあ、いいわ。顔拭いておいた方がいいわよ。そのまま歩くと独り身の兵士に恨まれるから」
【武】「言われなくてもわかってます」
こんな状況で廊下歩けるかっての。

Amaterasu (Ixrec plus editors and QCers and everything):
【Yuuko】「I'm the type who can't get serious about someone younger than me. So don't get too excited.」
【Takeru】「Wh-why the hell would I!?」 //literally "O-of course [I wouldn't]!"
【Yuuko】「............your manhood has other ideas......」 //literally "considering [you just said you aren't], your manhood is in full bloom"
【Takeru】「Where are you looking!?」
【Yuuko】「It doesn't matter. You should go wash your face. All the single soldiers will glare at you if you walk around like that.」
【Takeru】「You don't have to tell me that.」 //literally "I know that even without being told"
You think I can bear to walk down the hallways like this?

This excerpt is arguably more interesting because it contains humor, innuendo, and other things which are not explicitly stated, whereas everything in the formal excerpt was very explicit. As a result, there's much more room for adaptation, or flat out screwing up.

====================
Machine Translators
====================

Google (horrible machine translation):
Evening Call - "Can not be serious in nature because my younger me. I do not lust"
[Wu] "Oh, Is not that obvious!"
Evening Call - "... ... it's a percentage that is opening completely man ..."
[Wu] "Where Are you looking?"
Evening Call - "Well, OK. I should I'll wipe my face. Hatred from the soldiers just walk singles"
[Wu] "I know without being told"
Walk the hallway of selfishness in this situation.

Babel Fish (slightly less horrible machine translation):
“The [a] it is to do, therefore the character which is not accustomed to [tsu] [te] junior seriously. Desire without doing, don't you think?”
“The [a], natural it is not or!!”
“............Don't you think? that comparatively it is man full opening, ......”
“Where you see, it is whether!?”
“Well, good [wa]. The face the [wa] where the person who wipes is good. When you walk that way, because grudges in the soldier of the single body,”
“Is recognized and increases it is not said and also the [te]”
The corridor you can walk in such a circumstance, applying.

Honyaku (respectable by machine translation standards):
【Evening call】 Because "I have a character not to be used to seriousness to a younger person." Do not make passion
【武】 "Is not あ natural?" ! "
【Evening call】 …………It is man fully opening to the percent……"
【武】 "Where do you look" at? ? "
【Evening call】 "まあ is good". You had better wipe a face. Because I am born a grudge by a soldier of the bachelorhood when I just walk
【武】 I "understand it even if not said"
Of the state of things to be able to walk the corridor in such situation.

If you didn't hurt yourself laughing at the machine translations before, it's pretty much impossible not to this time. It's also nigh impossible to tell what's even going on here, much less if any one line is close to correct.

Edited Machine Translation (working only from the three machine translations above):
【Yuuko】 "You can't be serious. You know I don't lust after young people right?"
【Takeru】 "I already know!"
【Yuuko】 "Though in comparison... you do seem to be about the proper size..."
【Takeru】 "Where are you looking? "
【Yuuko】 "Anyway, go wash yourself off. Soldiers are supposed to walk a solitary path."
【Takeru】 "You didn't have to say that, I already know."
I didn't know what to do in this situation and just walked out into the corridor.

Ironically, since the lines were simpler and used easier vocabulary, the edited machine translation got more of the individual words right than last time. However, even on one of the short Takeru lines (the top one) it missed the point, and all of the longer lines are at least half wrong if not completely unrelated to the actual meaning. Unlike with the formal excerpt, this approach fails to even make sense, much less be funny or accurate.

====================
Student Translators
====================

Human fluent in English but still learning Japanese (who's familiar with MLA):
【Yuuko】「I'm the type to not get serious about younger guys. They're just not sexy.」
【Takeru】「W... why're you looking in my direction?!」
【Yuuko】「............I like it when a man goes full throttle between my legs......」
【Takeru】「What the hell are you looking at?!」
【Yuuko】「Meh, it doesn't matter. Having you wipe my face is just fine. In this situation, I don't want the resentful, wandering body of a lonely soldier.」
【Takeru】「It's fine if you don't say it.」
In a situation like this, I might as well walk back out the corridor.

Human fluent in English but still learning Japanese (who knows nothing about MLA):
Yuuko: I said seriously don't get too familiar with the younger (cadets?) Don't give into your sexual desires okay?
Soldier(s): ah, That isn't normal (natural?) is it?!
Yuuko: .............. Considering all that the men are full throttle... (i have no idea)
Soldier(s): Where can I see them!?
Yuuko: Well. Okay. You had better wash your face(s) first! Because as things are now (they) would hate it if you walked up to them. (wtfx2)
Soldier(s): I (we) understand that without you even saying it.
Such things were being said while walking through the corridor.

Both students had no idea what was going on for the most part. In terms of accuracy, they're probably only slightly better than the edited machine translation. However, being humans, they were able to perceive which lines were meant to be funny, and as a result managed to make their output somewhat humorous despite the inaccuracy.

====================
Fluent Translators
====================

Human fluent in Japanese but not a native English speaker:
"I can't get serious about men under my age, so don't get turned on."
"O-Of course I won't!!"
"...Then what's that bulge down there...?"
"Where are you looking at!?"
"Oh well. You should wipe that off your face. You'd make enemies with loners if you walk around like that."
"You don't need to tell me."
There's no way I can walk around like this.

Human (other than Ixrec) fluent in Japanese and a native English speaker:
Yuuko: I regret to inform you that I have no interest in kids. Try to keep those hormones of yours in check.
Takeru: W-who said anything about hormones!?
Yuuko: A little friend downstairs told me all about them...
Takeru: Can we keep this conversation upstairs, please!?
Yuuko: Yeah, yeah. Just lemme give you one piece of advice: You walk around like that and you're gonna get some nasty stares from people not lucky enough to have someone to direct their pent-up... energy toward.
Takeru: I'm fine, thanks.
You really think I'm gonna parade around the halls like this?

The stylistic differences I pointed out before are even more evident this time around, since both (all three if you count me) were trying to adapt jokes. Notice the non-native adapted the jokes into something much more explicit, and once again had a questionable but harmless word choice ("loners"). In contrast, the personalized one used a much more roundabout innuendo than I did, completely rewrote Yuuko's third line, and used a more colorful vocabulary than any of the others. Once again, it's entirely up to you whether any of this is good or bad.

========================================
Conclusion
========================================

The obvious take-home-message here is two-fold:
1) Do not try to translate anything until you find it easy if not effortless to read in one langauge and write in the other.
2) No matter how good translators are, their work will always differ strongly from each other's.

Less obvious claims include:
3) If you aren't that good yet, it's better for everyone in the long run if you focus on improving your Japanese or English or whatever rather than struggling to produce substandard work at a snail's pace (exception for private translations meant only for self-education, as long they're never publicly released).
4) The difference between reading a badly translated story and a well-translated one would be huge. Just try to imagine what'd happen to your favorite story if the meaning of every other line got arbitrarily changed.
5) It is very possible for a translation to be so bad that it really is worse than no translation at all. In fact, I've seen quite a few of these myself.

Of course there's no way for a non-Japanese speaker to tell for sure if a translation is inaccurate (though if the English sucks that's often a big hint), but it is important that you're all at least aware of the possibility.

Comments (23) Trackbacks (1)
  1. >Human fluent in Japanese but not a native English speaker
    Oh, I can recognize that kind of translation. It reads a lot like TakaJun’s. Anyway, it’s an interesting comparison that you’ve got there. That last line almost made me cry, translators like you and Amaterasu are more than what we readers deserve.

  2. >Human (other than Ixrec) fluent in Japanese and a native English speaker

    Moogy?!

  3. Awesome blog post.

  4. >【Yuuko】「…………I like it when a man goes full throttle between my legs……」
    Priceless.

    I don’t like the way the fluent Japanese / native English speaker localized his translation.

    The non native English / fluent Japanese translation would be ideal with an editor

  5. Eh, you sound like a jerk saying this (‘”translate” (lol)’- those two words alone shifted how I would have otherwise seen this post, because you were fairly unbiased for the rest of your post), even though I agree with you entirely.

    I think that your third point is probably your most important one for potential translators. That said, I think it’s definitely possible to write a quality translation even without mastering Japanese first. All I need is a good story, and a translator with a good eye for dialogue and who is able to keep the whole story in perspective can capture the story. Sure they might miss the line that have a second, deeper thematic meaning, but I don’t care about that. If a conversation makes sense, and is true to both the characters and the story, then that is not a problem. Your examples of non-Japanese speakers were poor examples of that, for sure. Their conversations didn’t make sense. But it wouldn’t be difficult to look at it objectively and fix those conversations, they were just lazy.

    Though I suppose someone willing to put that amount of creative effort into a translation would be willing to put in the required technical effort.

  6. @Matt
    Some so-called translation projects truly do deserve nothing but mockery – the type begun by those who think it’s somehow acceptable, or somehow doing others a service, to dump completely unedited machine translations into a patch with results that verge on the incomprehensible. I have no idea what kind of person fails to recognize this is both totally inadequate as a translation and a waste of time (since people can ATLAS to their heart’s content), but hopefully this post illustrates once and for all why machine translations should never be relied upon for understanding.

    As for your opinion, while what you’re talking about may have its merits, it is not a quality -translation-. The word “translation” implies equivalence, and equivalence requires understanding. If someone (possibly the translator themselves) rewrites lines without understanding or considering the author’s intent and what they were trying to communicate, they are blurring the line between translation and fanfiction.

    This is ridiculously subjective, of course, and if bad translations are inevitable I suppose I’d be happier to see more of them err on the side of being too liberal with their rewriting rather than being embarrassingly stilted and literal as many are. (Quite a few such liberal translations exist in classic literature, like Urquhart’s Rabelais, which are thought poor as translations but recognized as significant literary achievements in their own right.) But in my opinion, if you want to write a fanfic, write a fanfic – don’t call it a translation.

  7. So, not knowing japanese and not being a native english speaker, how do I know which groups to trust? Can you write a list? Probably not (since the rage-hate-flame-whatever it would get). In that case, any tips?
    My english is good enough to spot a machine translation or one that obviously doesn’t make any sense, but as you pointed out, that is rarely the issue.

  8. There’s really no way to know for sure without having other translators look over every translation out there and state their opinions on them, which is part of the problem here. In the real world, with classic literature, this actually happens to almost everything. Around here, it does happen on a regular basis, but only to the extent of “Does this guy have any idea what he’s doing? Yes? Okay, moving on.”

    Any translator trusted by the community (referring more to other tlers than readers here) such as me, makoto, takajun, phar and so on are virtually guaranteed to put out quality translations. As for newer guys, all you can really do is ask someone to scrutinize their work (which is often done if there’s any suspicion the tler isn’t that good, so you can try asking in #ammy or #tlwiki if you’re concerned about a specific project). The good news is: how long they’ve been translating and/or learning the languages is usually a good indicator of quality, since you really do need to spend years (plural) learning Japanese in order to stand a chance at translating it well.

    @skycat: Thanks for responding to Matt for me, I have nothing to add to what you said.

  9. I love Takajun. He’s translated two of the best eroge’s I’ve ever played: Fate/Stay Night and Sengoku Rance.

    Flawless quality in his translation.

  10. “>Human (other than Ixrec) fluent in Japanese and a native English speaker

    Moogy?!”

    SubaHibi

    wwww.

    You could always ask Makoto or Phar etc…

  11. The thought that people are jumping into translating without actually knowing a language is troubling. What is even more troubling is that there are individuals out there doing Japanese to English translations who haven’t mastered EITHER language.

    To expand on the point made above, for those that don’t know Japanese it is difficult to know who to trust. However, this is actually one of the first serious discussions I have seen on the issue that hasn’t deteriorated into a troll-fest and a flame war. The absence of Moogy in this discussion helps in that matter. Although its only a matter of time before he jumps in and sends everything to hell.

  12. @TickTock I regret to inform you that I am the “Human (other than Ixrec) fluent in Japanese and a native English speaker” in the post ( ´_ゝ`)

  13. Also @TickTock (kinda @cody too), in case you’re wondering why I deliberately chose Moogy, it was because I felt he was the most likely to effectively showcase how different someone else’s translation style could be from mine. There certainly would not have been as much contrast had I chosen makoto or phar or whoever.

    That, and makoto is working on MLA so he’s already seen at least one of those excerpts, and I’ve never directly spoken to phar before (same for probably anyone else you can think of).

  14. Well I stand corrected, even with Moogy involved it’s been a good and informative discussion.

  15. Very good examples. It reminds me why I don’t read VNs using text hookers. It seems a combination of J-native person for translation and E-native for QC would do a good job if they’re allowed to discuss vexed questions.

  16. Thanks for the answer, I will check it out. I actually wanted to ask about the Osananjimi wa Daitouryou translation over at TLWiki, since I saw some wierd lines somewhere (don’t ask where) but when I clicked on another random script, it seemed fine. So I don’t know…Also, it’s been in QC for a bit too long, and I don’t really understand why, that is usually the not-so-hard part, right? It’s not that I want them to rush with it, just strange that they got stuck on something like that. (I know that QC is important, so don’t get me wrong.)

  17. First of all: most people are unreliable no matter how easy their job. For most projects, there’s really nothing strange about a translator, editor, or QCer temporarily dying and delaying a project for weeks.

    As for OsaDai specifically, I simply don’t know what it is. However, some kind of translation check had to be performed (I do not know whether the original tler did a second pass or some other guy showed up or the editor nagged external sources or what), so that *might* be what “QC” actually refers to here. Only a wild guess though.

    If you wanted any comment on the quality of the translator, I was told he was a native Japanese speaker who couldn’t read kanji and thus made a number of translation mistakes. Why he couldn’t use an electronic dictionary to check the kanji was never explained, so I’m quite uninformed here.

  18. Damn, missed your comment.

    A J-native with an E-native *editor* (not QCer, editor, big difference) is definitely a good combination. Personally, I would not say that is necessarily ideal, because more of the original is liable to be lost in the initial translation pass than anywhere else (editing pass, tlc pass, whatever) and that means that the translator’s English writing skill is just as important as their Japanese reading skill. But, since being a native speaker and being a good reader/writer have a very weak correlation at best, there probably is no definitive answer here.

  19. This is great. I was just asked not an hour before reading this ‘why doesn’t your project get released nightly like open source stuff does?! Like, nightly alphas or something.’

    He didn’t seem to understand when I replied we were aiming for something of quality and a first pass translation wouldn’t be satisfying at all, and he would only end up complaining. I wish I had had this link to give him then. I have also tweeted this, for good measure.

  20. It would be interesting to see how someone fluent in Chinese but not Japanese would do. The differences in written text are pretty small, after all.

  21. I took a Chinese class in high school a couple years back. I wasn’t nearly as good at Japanese then as I am now, but good enough that I can safely say knowing Japanese makes it a lot easier to learn Chinese, but is nowhere near enough to actually read anything in Chinese. This probably works the other way around to, and I’d say it’s strong enough that even being fluent in one of them would leave you inferior to a machine translator in the other.

    The “small differences” you refer to are only small in comparison to other pairs of languages. For instance, English and Spanish also have relatively small differences, but I seriously doubt you can read any Spanish just from knowing English.

  22. What if I’m also an editor? I could’ve made my own interpretation or version of those lines.

    The whole point here is that, it doesn’t really matter how you translate something, as long as you’re not damaging any critical details and that your grammar is readable.

    Take G-senjou’s TL team for example. there was a lot of “shit hits the fan” and “shit happens” and even like “hell yeah!” in their script. yet no one really complained.

  23. The editor will never be able to do more than a fraction of the interpretation that the translator does, so no point in discussing them separately from the translator.

    But yes, once you’re at the level where you can understand what’s going on in Japanese and can write comprehensibly and unawkwardly in English, it really doesn’t matter if you know the meaning of each and every one of the 20,000+ lines or have a habit of making all your characters curse a little more than the average writer.

    I’m actually really lenient on quality past the point of reliable comprehension. It’s just that the vast majority of fan translators are nowhere near that point.


Leave a comment

Before asking a question, please check our FAQ, my list of VN reviews and my VN wishlist. Most of the questions we get are already answered on those pages.