Can More Efficient Tasks Still Measure Academic Writing Ability?

Since its inception in 1964 – the same year the Beatles first toured America – TOEFL iBT has gone through multiple revisions. One source of perpetual experimentation is TOEFL’s Writing section, which has operated in many different forms across the decades.

Several years ago, TOEFL iBT began researching the efficacy of a new, 10-minute academic discussion task. This research aimed at answering a core question: Was this new task as adept at capturing academic English writing ability as the more traditional, half-hour essay task? 

A new study provides encouraging evidence that it does. Comparing the retired Independent Writing task with the newer Write for an Academic Discussion task, our research found strong alignment in scores, linguistic indicators, and real-time writing behaviors. 

The results suggest that a carefully designed, efficient writing task can preserve meaningful evidence of academic writing proficiency – even when it produces shorter and differently shaped responses.

Traditional high-stakes writing assessments have often relied on lengthy, open-ended essays. In the former TOEFL iBT Independent Writing task, test takers had 30 minutes to write roughly 300 words in response to a broad opinion prompt. This task helped reveal a writer’s capacity to develop an argument, organize ideas, and use varied language.

Extended, open-ended essays, however, also pose real limitations. They can be time-consuming to administer and score, and their decontextualized prompts may not closely resemble the types of writing students actually do in contemporary academic communication, which increasingly includes shorter, audience-aware contributions.

The new Write for an Academic Discussion (WAD) task was designed with that changing environment in mind. In this task, test takers read a professor’s question and two sample student responses, then contribute their own response to the discussion. The format is shorter, more contextualized, and more interactive than a conventional independent essay. 

The core question in our research was whether the score that WAD produces can still support the same interpretation as the Independent task. That is: Does a higher score on WAD still reflect stronger academic English writing ability?

Our study examined responses from 1,095 adult English learners at seven Chinese universities. All participants completed both task types through a platform that recorded their written responses, as well as every keystroke and cursor movement test takers made while composing their responses. This allowed us to compare both the product and process of writing. 

Participants received scores from trained TOEFL iBT raters using task-specific, five-point holistic rubrics focused on content and elaboration, organization and coherence, and language variety and accuracy. 

The researchers then analyzed:

• Task-score agreement across the two formats
• Textual features such as length, syntax, vocabulary, accuracy, and reuse of prompt language
• Keystroke-based indicators of planning, pausing, and revising

This combination matters. Comparable scores alone do not prove that two tasks measure the same construct. Writers might arrive at similar scores through very different linguistic resources or cognitive processes. Looking at both the finished texts and test takers’ writing behavior offered a stronger basis for evaluating score interpretation.

Our clearest finding was that scores were strongly consistent across tasks. Nearly half of participants (47%) received the exact same score on the WAD and Independent tasks. Another 50% received adjacent scores, meaning their scores differed by only one point. 

In total, 97% of participants received identical or adjacent scores, while just 3% differed by more than one score point. The correlation between task scores was also strong, at r=.72.

Importantly, our statistical model found no meaningful difference in average scores between the two task types at the study’s conservative significance threshold. In practical terms, students who demonstrated stronger writing ability in the extended essay generally performed strongly on the WAD task.

That finding was significant. If a shorter task substantially changed who received high or low scores, stakeholders would have reason to question whether the test was measuring the same skill. Instead, the findings indicate that WAD largely preserves the rank ordering of writers established by the longer Independent task.

The two tasks did not generate identical texts, although this is an inevitable product of their design. WAD responses were not as lengthy as Independent ones. But the more important question is how the linguistic features elicited by each task related to quality. 

Across both tasks, higher-scoring writers produced responses with stronger lexical resources, greater syntactic variety, and better grammatical, word-usage, and mechanical accuracy. In other words, the linguistic indicators associated with stronger academic writing remained stable across both formats.

One difference deserves attention: response length was less strongly related to score in WAD than in the extended essay. This is arguably a positive result. In a 10-minute task, quality cannot depend primarily on writing more. Instead, writers must use limited space efficiently by taking a clear position, contributing relevant support, and expressing ideas accurately and coherently. 

The analysis also found more reuse of individual prompt words and short phrases in WAD, which is unsurprising because the task includes more source material. Crucially, the study found no evidence of widespread long-string copying. This suggests that writers were drawing on the discussion context without simply reproducing it.

The keystroke data added another layer of evidence. In the study, we examined how long writers spent before beginning to type, how long they paused within and between words and sentences, and how much time they devoted to deleting or inserting text.

Most process patterns were broadly similar once we adjusted for different task time limits. In both tasks, higher-scoring writers tended to spend slightly more time planning before writing, pause for shorter periods at several linguistic levels, and spend less time on deletion and word-level insertion. These patterns are consistent with more fluent language production and more efficient monitoring while writing under time pressure.

The largest task-level difference involved planning. Writers spent a greater proportion of their available time in pre-writing during WAD. That result makes sense: before composing, test takers needed to read and interpret the professor’s question and the sample student posts. Rather than signaling a different construct, this pattern appears to reflect the task’s richer discourse context. 

Notably, score-by-task interactions for process measures were generally small. Higher and lower scorers differed in comparable ways across both formats. This reinforces the finding that the WAD and Independent tasks draw on similar underlying abilities.

The central implication is not that every short writing task is automatically equivalent to a longer essay. It is that efficiency and validity are not inherently incompatible. When a task is deliberately designed, scored with an appropriate rubric, and supported by empirical validation, a brief and contextualized response can still yield meaningful evidence of academic writing ability. 

For assessment providers, this offers a path toward reducing testing time and scoring burden without necessarily weakening the interpretability of scores. For institutions, it suggests that modernized writing formats still provide credible information about an applicant’s ability to communicate academically in English. 

And for test takers, it underscores that success on a short task is not mainly about speed or word count. It depends on planning quickly, responding directly, using language precisely, and making an effective contribution within clear constraints.

The WAD task generated scores that closely aligned with those from the traditional essay; it distinguished stronger and weaker writers using many of the same linguistic indicators; and it elicited broadly comparable planning, pausing, and revision behaviors.