AI Workflows Outscored Human Translators In 4 Of 6 Content Types – China Benchmark Study via @sejournal, @MPentzek
All Paid Media PPC News Social MediaAdvertising Video Advertising Columns Ask A PPC ExpertNEW PPC Pulse Guide AI Max Invalid Traffic Doubled, Weeks Before Black Friday Invalid traffic on AI Max search doubled to 5.28%. Get the channel-by-channel retail benchmarks before Black Friday. Digital Marketing All Digital Marketing Analytics & Data Ecommerce Lead Generation Content Marketing Social Platforms Google YouTube Reddit LinkedIn TikTokNEW WordPress Other CMS Platforms Guide Your ChatGPT Calls Are Hiding in the 'Direct/Other' Bucket Calls from ChatGPT, Claude, and Perplexity land in 'direct/other.' This checklist shows six steps to track them as their own source. SEJ Live Webinars Rundowns PodcastNEW Ebooks All Resources Webinar Defend Your 2027 Link Budget With Tactics That Still Work Join this webinar for a keep-or-cut list of off-page tactics and a step-by-step plan for running the ones that still work on your brand. Human translators lost four of six content types to post-edited AI in a 774-output English-to-Chinese benchmark and model choice beat post-editing Marcus Pentzek 13 minutes ago ⋅ 13 min read Marcus Pentzek Bio Follow We conducted a benchmark study of English-to-Chinese localization, evaluating content types, task types and workflow models to compare human performance against machines. Professional human translators scored outside the top five on four of six content types. On marketing copy, they finished 10th of 15, 22.2 points behind the best AI workflow. Where humans did win, the margin over the best machine workflow was 2.8 points. We measured 774 localized outputs across six content types, seven workflow models (human, Chinese LLM*, Chinese LLMPE**, Western LLM*, Western LLMPE**, MT*, MTPE**), and three task types (translation, transcreation, creation). Each output was scored on accuracy and consistency, fluency and language quality, and style and cultural adaptation – each dimension weighted equally at one third of the total. Scoring was performed blind*** by Chinese native-speaking professional localizers.**** This measured localization quality only. No ranking, traffic, or conversion data was collected. Disclosure: The benchmark discussed here is a joint research project by EC Innovations, a localization services provider, and Jademond Digital, where I am a Partner and Director. Both firms sell services related to the practices this research evaluates. Professional human localization was tested against fourteen machine and hybrid workflows across six content types. It finished first on two of them. On the other four, it finished outside the top five – 7th, 7th, 9th, and 10th out of 15. Table 1: Human localization finished first on two content types and outside the top five on four. Ranks are competition ranks out of the 15 workflows tested. Human professionals won two of six categories and finished 10th of 15 on marketing content, 22.2 points behind post-edited Qwen. PE-Qwen won three categories outright, tied a fourth, and placed second on the remaining two. This is the actual shape of the finding, and it is more interesting than “humans win SEO.” Human expertise is decisively better at a specific thing: content where factual precision and terminological consistency dominate, and creative latitude is near zero. That describes informational and SEO content. It does not describe marketing copy, UI strings, or social content – and on those, paying professional translation rates buys a worse result. EC Innovations’ reading of the marketing result is worth repeating: Human translators appear to over-correct the language, smoothing copy toward formal correctness and stripping out the contemporary register that marketing content depends on. The same instinct that makes a linguist excellent at terminology discipline makes them a poor fit for writing that needs to sound like the internet. We measured localization quality. We did not track a single SERP position, and no one should present a 22-point – or a 2.8-point – quality gap as a ranking outcome. The published evidence connecting content quality metrics to ranking is thinner than the industry generally concedes. Portent’s crawl of 756,297 ranking pages found no correlation between readability and Google ranking position (Portent, 2021), and specifically called out the circularity of firms that sell content quality asserting that content quality drives rankings. That criticism lands on research funded by localization companies too. Treat everything in this article as an input hypothesis to test, not a demonstrated ranking effect. Humans won two categories. Both wins are narrower than the headline suggests. Here is the full per-model ranking for SEO content – meta descriptions, headlines, keyword-carrying body copy – from a dataset that has not been published at this granularity: TABLE 2: How authors, tools, and workflows compete on SEO content. “PE” stands for post-editing by a human professional; “Google MT” is Google Translate. The human lead over the best AI workflow is 2.8 points. Humans still finish first, but the gap to the best available AI workflow is only 2.8 points and given that each score is an average of three equally weighted dimension ratings, a sub-three-point difference is inside the range where I would not want to make a six-figure sourcing decision without replicating it on my own content. Post-edited Qwen or Doubao is roughly at parity with professional human localization for SEO content and raw output from any model is not. There is also a finding here for anyone running legacy infrastructure. Post-edited Google Machine Translation (MT) scored 66.7 on SEO. Ahead of raw Qwen and ahead of every raw model in the study. If you have a mature MT pipeline with translation memory and an established termbase, adding a post-editing layer gets you further than replatforming onto a raw LLM would. On SEO content, professional human translation scored 74.1 out of 100. Raw Chinese LLM output and raw Western LLM output both scored an average of 60.7. A 13.4-point gap in favor of humans. That number is as close to meaningless as a procurement input, and understanding why is the most useful thing an SEO can take from this dataset. “Raw Chinese LLM: 60.7” is a straight average of four models: Qwen, Doubao, DeepSeek, and Kimi. Here is what those four actually scored on SEO content: TABLE 3: Chinese LLMs scoring spread on SEO Content – Averaged: 60.7 A 9.2-point spread. And SEO is the tightest category in the study. On technical content, the same four models span 22.3 points, from Qwen at 70.4 to Kimi at 48.1. On marketing, 22.2 points. TABLE 4: Western LLMs scoring spread on SEO Content – Averaged: 60.7 Nobody deploys “Chinese LLM.” They deploy Qwen, or Doubao, or DeepSeek. The category average describes a model that does not exist. The two categories do not behave the same way. Across all six content types, the widest gap between the two Western models is 8.4 points, on informational content. Among the Chinese four, it reaches 22.3, on technical. If you have standardized on GPT or Gemini, “Western LLM” is a roughly honest description of what you will get. “Chinese LLM” is not a description of anything. It is the average of a field whose best and worst members sit more than twenty points apart, and the model you actually deploy could be at either end. In the SEO numbers, both categories average 60.7. Identical summary statistic, entirely different distribution behind it. Gemini leads the raw field on user-generated content at 67.6, the only content type in the study where a Western model finishes ahead of every Chinese one (without post-edit). The same problem shows up from the other direction. Kimi finished last among the Chinese four in all six content types, and removing it, which is what any enterprise does the moment it runs a two-week bake-off, moves the “Chinese LLM” figure by between 1.3 and 4.9 points depending on the category. The category average is not merely imprecise. it is being dragged by a model most buyers would eliminate in week one. This is not a criticism of how the study aggregated – equal-weight averaging across a defined model set is the correct way to characterize a category. It is a caution about how the resulting number gets used. If you read “Chinese LLMs score 60.7 on SEO content” and conclude Chinese LLMs are unfit for SEO work, you have drawn a conclusion the underlying data does not support. Post-editing is usually described as a quality layer you either buy or you don’t. The category view shows it is nothing of the sort. On user-generated content, a post-editing pass over Google Translate output is worth +30.6 points. On marketing content, the same pass over a Western LLM draft is worth -0.9. The variable is not how much editing you buy. It is whether the draft you hand the editor is close enough to right that editing improves it, or wrong enough that the editor spends the budget fighting it. That UGC figure – 33.3 raw to 63.9 post-edited – is the largest single movement in the study, and an unsurprising one. Raw machine translation of informal Chinese social copy starts from near-unusable.
Source: Search Engine Journal
This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.
📚 Visit more helpful articles on Joab Peters Blog
No comments