Shin and Cimasko, in an article in the latest issue of Computers and Composition, analyzed multimodal web page arguments of ESL students in a first-year composition course.

Their findings include

  • All students placed a higher priority on the linguistic mode, that is, the written word.
  • Only one student used imagery in all of his drafts, although other students used them by the final draft.
  • Only one student included audio files (after the instructor recommended it).
  • Hyperlinks were used but primarily for bibliographies instead of from within the texts of their arguments.
  • Although the written word predominated, non-linguistic additions "added new meanings ... as representations of emotional dimensions that could not be conveyed easily--or appropriately--in traditional academic discourse" (p. 388).
  • Layout in terms of background design and font choice was strongly influenced by written essays with only a few students attempting some variation of color.

Explanation of findings
Shin and Cimasko state that these findings are likely due to

  • students' prior experiences,
  • the writing practices of the students' communities,
  • the context in which they wrote their texts, and
  • students' perceptions of multimodal texts.

As Shin and Cimasko wrote,

Multimodal composition was interpreted as a distraction from the primary goal of developing academic capability through written language. The students thus opted for the traditional and established centrality of linguistic design, resisted new modalities, and applied those new modalities that were used in ways that did not take full advantage of their rhetorical potential.

Comments on article
These findings make sense. People do what they're accustomed to doing and according to their expectations of how they should do something. And it's commonly understood that visuals can enhance understanding in ways that the written word alone cannot do. (However, see "Using videos" in Home Schooling and Videos). And with the authors, I accept that learning the same rhetorical concepts in different modes can enhance understanding of those concepts although the evidence for this position is not without some qualification (see, for example, Multimodal Learning Through Media). Having said that, I question the need for multimodal writing in freshman composition that the authors propose.

The authors support multimodal writing because they believe,

multimodal approaches to composition provide writers who are having difficulty in using language, including those writers for whom English is a second language (ESL), with powerful tools for sharing knowledge and for self-expression. ... ESL students need to gain knowledge of how to use non-linguistic modes at the same time that they are developing their English writing abilities. (p. 377)

However, it's not clear to me that:

  • ESL students in first-year composition need to learn these tools, or that
  • First-year composition is a course that should include self-expression.

Some of the tools noted in the article included using audio, video, and animation. Will they really need these tools in future course work or in future careers? Including instruction on new tools requires time. Should time be sacrificed for learning these modes instead of working on written genre conventions?

Whether or not first-year composition should include self-expression depends on the purpose of the course. Generally speaking—and despite one's personal position on its purpose—it's considered to be an introduction into academic writing, often academic argumentation. Without entering into the debate on voice and identity, let me just say that my ESL students, mostly Generation 1.5 students, have little problem with self-expression. What they do find difficult is writing in an academic register. In such a context, self-expression is not a priority.

In addition, it's not clear that first-year composition is the best place for students to learn how to use visual modes, especially with respect to self-expression. One reason given for this is that some researchers argue that it is necessary for "developing certain kinds of disciplinary knowledge" (p. 377). I managed to get three of the sources cited, but the support was not strong.

One researcher cited was van Leeuwen, who wrote on three principles of multimodality: information value, salience, and framing. However, he did not argue that they were "necessary" for developing disciplinary knowledge. Still, I can imagine that if these three principles are universal across modes, it would be useful to know them.

A second researcher cited, Ann Johns, wrote on how a single student was adept at using graphs and charts to understand her macroeconomics work. Undoubtedly, graphs and charts are a part of academic writing. However, these sorts of visuals are not the type used for "self-expression."

Along these lines, another scholar cited, Miller, wrote,

In short, visuals in academic articles provide data to convince the reader of the validity of the findings and allow the readers to see how the data were obtained and to interpret the data themselves. These visuals are impregnated with theory (Bazerman, 1988) to show not only that they are anchored in the literature but that they have wider implications.

In journalism however, the writer is interested in presenting news rather than in convincing the reader of the validity of the report. In news articles, findings are highlighted, but the means by which the findings were obtained are placed in the background, just the opposite of in science. The reader is not positioned as knowledgeable but as needing to be enticed into the article. The launching point, therefore, is human interest rather than scientific argumentation. (p. 31)

In other words, the visuals used in academic writing are related to data rather than to self-expression. Interestingly, Shin and Cimasko wrote of "emotional" representation, something more akin to the journalistic perspective of "entic[ing']" readers rather than the academic perspective of "convincing" and supporting an argument—the goal of this freshman composition course.

As noted above, generally speaking, I believe that using different ways of presenting the same information can be a valuable pedagogical tool for explaining concepts of rhetoric and composition. Thus, I take a little time to cover presentation principles, including the need for images, and have my students write essays analyzing visual objects, such as advertisements and website designs, to provide a variety of contexts for the same concepts, thus facilitating, I hope, transfer of their writing knowledge. Even so, I hesitate at "fully integrating [multimodal composing] into the work" (p. 391) of first-year composition, especially of the self-expressive sort, thus taking away time from other principles of composition necessary for the development of my students' "academic" writing.

I hesitate for two connected reasons. One is that most of the "composing" that most of these students will do in later classes and on the job, at least in the near future, will be print-based (although see Alex Reid for an opposing opinion). Yes, they may use data-related visuals later on, but most of the writing in freshman composition is not data driven.

The second is that one learns what one practices, and one learns to the extent that one practices. My students need as much time as possible with the English language, with developing their vocabulary, with learning academic textual conventions. Any time that takes away from that practice is to their detriment academically and careerwise. Think about it. Can you imagine a multi-ball training regime in which a basketball player spends time playing tennis, soccer, volleyball, and handball?

A few resources:
Survey of Multimodal Pedagogies in Writing Programs (Composition Studies)
Taking a Traditional Composition Program "Multimodal (Christine Tulley)
Multimodal Writing (Teaching Writing Using Blogs, Wikis ...)
Standards Related to Digital Writing (Teaching Writing Using Blogs, Wikis ...)
Thinking about Multimodal Assessment (Digital Writing, Digital Teaching)
Center for Digital Storytelling

Works cited:
Johns, A. M. (1998). The visual and the verbal: A case study in macroeconomics. English for Specific Purposes, 17, 29-46. 183-197.
Miller, T. (1998). Visual persusasion: A comparison of visuals in academic texts and the popular press. English for Specific Purposes, 17, 29-46.
Shin, D.-S. & Cimasko, T. (2008). Multimodal composition in a college ESL class: New tools, new traditional norms. Computers and Composition, 25, 376-395.
Van Leeuwen, Theo. (2003). A multimodal perspective on composition. In Titus Ensink & Christoph Sauer (Eds.), Framing and perspectivising in discourse (pp. 23–61). Philadelphia: John Benjamins Publishing Company.