by Tanya Clement, Sam Turner, and Trent Wintermeier

AVAnnotate allows users to build digital exhibits of annotated audiovisual (AV) artifacts. AVAnnotate leverages public code and document repositories (GitHub), IIIF (International Interoperable Image Framework) framework, and the AVAnnotate web application to create and share IIIF manifests and user-generated annotations of audio and video materials (Clement et al.). Tanya Clement, Brumfield Labs, and Performant Solutions developed for educators and scholars to enhance the usability and discoverability of AV materials, making it an excellent tool for both research and classroom use. This contribution is targeted towards first-year writing instructors, but may be adapted by any educator seeking methods for teaching students how to critically engage multimodal texts and audiovisual materials. We offer educators a sample AVAnnotate project using artifacts from the SpokenWeb collection and documentation as well as a lesson plan for incorporating AVAnnotate into the first-year writing classroom.

Educational researchers have long asked what knowledge teachers need to teach students in a contemporary context, and that context is increasingly one wherein “texts are elaborately multimodal, constructed not just of print but of image, sound, and movement” (Ryan et al. 477; Kress & van Leeuwen, 2001). Multimodal learning environments challenge teachers in a number of areas (O’Mara, 2006; Walsh et al., 2007), and educators and researchers have responded by creating frameworks for multimodal analysis and critical thinking (e.g., Hodge and Kress, 1988; Jewitt 2002, 2006, 2008; Jewitt & Kress 2003; O'Halloran & Lim-Fei 2011; Unsworth 2001, 2007, 2008; O'Halloran et al. 2015). In the first-year college writing classroom in particular, teachers face pressure to prepare students for a broad range of collegiate and post-graduate experiences in the ever-shifting technological landscape. Critical engagement with multimodal arguments is thus a necessity for rhetorical education, though teacher training inconsistently prepares educators to enact this type of composition pedagogy. Sharing resources–like pedagogy collections and ready-made tools–is then one way to address this challenge for teachers.

AVAnnotate is a particularly useful tool for multimodal composition pedagogy as it requires students to extend beyond literacies stemming from text- and print-based resources. Some of the foundational texts frequently cited as influential in the field’s explicit turn toward multimodality in the late-1990s and early-2000s called for an integration of audio (Comstock and Hocks; Danforth et al.) and video (Welch 1999) in the composition classroom. Following this avenue, Mary E. Hocks and Michelle Comstock’s article, “Composing for Sound: Sonic Rhetoric as Resonance,” argue that sonic literacy is important because writing is not simply text-based but always involves multiple semiotic modes (see: Ceraso “Sonic Scenes of Writing,” Sound Never Tasted So Good; Hawhee “Rhetoric’s Sensorium”; Hawk and Stuart “English Composition as Sonic Practice”). They continue to explain how rhetoric and writing studies scholarship and pedagogy “has worked to unseat the privileged place of text and alphabetic literacy as the center of composing processes” (135). Asking students to compose sound-based or video-based assignments and to use new media technologies such as AVAnnotate to accomplish this work increases their ability to use multimodal methods for communicative, expressive, and rhetorical modes.

To use AVAnnotate, students must generate annotations, which Clement and Wintermeier have described as an “affordance” of the tool that promotes engagement with interpretive invention and other aspects of rhetorical listening (2024/5; Ratcliffe 2005). Using annotation to facilitate multimodal composing, students become more aware of how their listening practices contribute to different identifications, readings, and interpretations of audiovisual materials. Ratcliffe identifies these “rhetorical listening moves” as essential for cross-cultural communication (78). As Clement and Wintermeier explain, student creators are “situated” listeners and “produc[e] annotations for their AV object as they view and listen to their recording by noting significant attributes that resonate with them . . . these annotations also guide how an end-user engages the AVAnnotate project” (n.p.). Ultimately, annotation affords the relationship between student creators, the AVAnnotate software, and end users who engage with discursive and communicative methods offered by the technology’s interface. Bursztajn-Illingworth and Wintermeier outline an example of these goals in the “Curricular Context” section of their AVAnnotate DLF Toolkit where they describe a class in which students used AVAnnotate to share annotations that drew attention to how a video of an interview involved the interviewer using specific body language to generate specific responses of their interviewee (Bursztajn-Illingworth and Wintermeier, n.p.). AVAnnotate projects demonstrate how students can use annotations to generate multimodal arguments within digital contexts.

In the classroom, AVAnnotate can be used to facilitate students’ analysis of audio and visual materials. By inviting students to slow down their reading and research practices, engage in close listening and viewing via critical annotation, and offering a platform for students to build their analysis of a given artifact, the lesson plan below offers teachers a way to engage students in multimodal literacy using AVAnnotate.

Lesson Plan

90-minute First-Year Composition Annotation Discussion and Development

This lesson is designed to be used in a first-year writing classroom that engages multimodal texts and literacies. This lesson plan will 1) teach students how to critically analyze and engage audiovisual texts, and 2) offer students a structure for multimodal composing methods via the AVAnnotate software.

This lesson plan is based around an example AVAnnotate project that we developed to showcase different methods of annotation that teachers and students may be interested in utilizing and applying. The sample AVAnnotate project we share here was developed using the 2022 SpokenWeb reading and discussion featuring UAlberta Writer in Residence Ifeoma Chinwuba and Edmonton Poet Laureate Titilope Sonuga, however, it is important to note that the annotation methods we showcase in this project can be applied to any audio and visual materials that teachers and students are interested in engaging. We developed this lesson plan assuming that instructors and students have already been primed to the basics of engaging multimodal texts (according to a given curriculum or course objective), and offer AVAnnotate as one method of doing so.

Objectives for student learning:

Students will…

  1. Develop a clear sense of how multimodal composition methods afford alternative routes of critical expression across digital contexts, especially through the process of annotation,

  2. Engage with institutionally-available audiovisual resources across campus, including how to access and utilize them for research purposes, and

  3. Understand the rhetorical considerations involved in annotating different elements of audiovisual materials and artifacts.

1. Introduce the Assignment

  • Explain the outline and goals of the assignment. Students will generate annotations of a multimodal text using either a) the sample audio provided in the AVAnnotate project, or b) other audiovisual material at https://ualberta.aviaryplatform.com/collections/1783, institutional holdings, or online archives.

  • Break students into groups of 4-5, depending on class size. Each student group should decide to use one (1) audio or video material; meaning each student in the group will collaboratively annotate the same individual object.

  • Discuss the AVAnnotate templates students will be using during this lesson

  • Questions students should be able to answer before annotating (Step 2):

    • What is the structure of annotations for the AVAnnotate software?

    • What is the difference between annotation and transcription?

    • How does the relationship between the annotation and tag impact the content of the exhibit?

2. Develop Annotations

  • Start by assigning students to listen/watch the multimodal text and discussing a) the current annotation sets provided, and b) ideas for new annotations/annotation sets. The sample project contains 3 annotation sets: Edmonton Poets Transcript; Edmonton Poets Captions; Readings and Discussion.

  • Identify what else students could pay attention to and annotate; are there ways to pay attention differently to generate annotation sets around…

    • Medium/material (Zoom recording, audio noise, glitching)

    • Environment (background noise, non-linguistic sounds)

    • Structure (content vs. transitions)

    • Tone/affect (inflection, mood)

    • Your course’s topic/themes

  • Give groups ~30 minutes to develop their new annotations using the AVAnnotate templates

  • Refer students to AVAnnotate's documentation for guidance on how to use basic AVAnnotate functions, including tags, time stamps and annotations

  • In each group, confirm that students are following these instructions:

    • Students should collectively decide on what to annotate (the quality of the audio; body language in the video; transcript of one speaker; etc.) and why

    • Students should collectively decide on what tags they will use to represent these annotations

    • Students should collectively decide on the amount of detail, length, and other aesthetic considers of the annotations

    • Students should collectively decide on how to break-up the audio (by time; tag; annotation feature; etc.)

  • Finally, students will develop their annotations utilizing the provided AVAnnotate templates

3. Reflection

  • Ask students to discuss in their group what they found interesting in the process or content of their annotations and why. What was difficult about this practice?

  • Reflecting on all of the annotations together, what do you notice is different between each student annotation set?

  • If there is time, have students share major findings with the rest of the class

4. Developing an AVAnnotate Project (optional)

  • Having students develop a full AVAnnotate project (collaboratively or individually) can be a productive activity in that students are able to listen/view their AV material alongside their collaborative annotations

  • While there isn’t enough time in a 50-minute period to complete a project, there are instructions online that students can refer to. See the AVAnnotate “How to Use” page.

  • The basic outline of this can be communicated at the end of the class:

    • First, students will log into the AVAnnotate application. A GitHub account is required to authenticate users in the AVAnnotate software

    • Second, students will take the audiovisual file URL (ending in .mp3 or .mp4) or the IIIF and input that into the software

    • Third, students will either

      • 1. Upload their annotation files individually in .xlsx, .csv, or .tsv format

      • 2. Upload their annotations collectively in .xlsx, .csv, or .tsv format

    • Fourth, students have the option to build out their exhibit in the AVAnnotate software, including images, context, or other details they wish to include

    • Fifth, an AVAnnotate project will be built and viewable as a static and shareable exhibit that can be turned in online or presented in-person during the next class