My Multimodal Text

Link to Tiff's Multimodal Text

Semiotic Techniques Used in My Multimodal Text

I aimed to convey meaning through a multimodal approach, utilising a wide variety of modes to create and enhance my text.

I wanted to steer away from more ordinary text, so I downloaded a Tudor-style font to create a more realistic typography; this was used for the main title, as well as throughout the presentation. Other forms of texts used included speech bubbles. This was a useful tool inspired by Inanimate Alice and Horrible Histories, meaning I could add extra information and instructions for/to the viewer. A significant way I used speech bubbles to create a semiotic turn, conflicting the content of the verbal direction with the direction of the speech bubbles. This was for comedic and ironic effect, such as when Tom verbally suggests  we move on to the next section and continue working, followed by his speech bubble suggesting we take a break and go for some lunch. To continue with postmodernism, I also used conflicting styles of imagery. The majority of the adventure is in a fictional cartoon style, this is to show the unrealistic notion of traveling back in time. This was juxtaposed with the real-life footage I filmed of my hand opening its palm to show the keys that the viewer has collected. I created a brief dream-like fade effect by focussing my camera at the last moment, so the keys are blurry to begin with before becoming clear and sharp. I did this to show the ‘real’ human side of the text; that is to say to remind the reader that the Tudors too were real, and even to touch on the moral side of the viewer saving a life from execution.

Throughout the multimodal text I used a variety of audio, including vocal speech, sound effects, and music. I layered this audio to create a richer depth of sound. In one place, the slide which takes place by the large cooking fire, I wanted the viewer to really feel like they were by a fire, so I used this multi-layered approach to layer the sound of a ‘roaring fire’, with that of a ‘crackling fire’, then recorded my vocal speech of Tom the Tudor Fly, and lastly added his ‘buzzing’ sound. In addition to this, I enhanced the image of the fire with a filter to make it more orange and brighter, using GIMP, now that I was slightly more comfortable using it (see previous post). Adding to this the layered animation clips of moving fire, and a flaming text spelling the word ‘HOT’, this hopefully created an appropriate paradigmatic atmosphere to help the viewer feel encompassed by their surroundings in a fiery setting.

I used special effects as a postmodernistic tool, placing video clips and imagery at interesting angles to entice the viewer in to a new way of seeing things. I experimented with different camera angles, using overhead footage of a great Tudor building to simulate flying with Tom the Tudor Fly, and to see the building from his perspective. His perspective was also seen with the ‘fly-on-the-wall’ –esque image of the kitchen, before zooming in to human level. I also utilised a variety of transitions to move smoothly between slides and to help the reader feel like they are ‘flying’ with Tom.

I used my character of Tom the Tudor Fly as a signifier for the direction and place the reader travels to. Tom frequently directs the viewer to ‘follow me!’, which the reader is able to do through the extensive use of animations and graphics. Tom is almost constantly in motion; I did this by creating my own erratic ‘fly-inspired’ custom flight path animations, and often setting these to repeat themselves so Tom is always buzzing and on-the-move. This heavily animated aspect of my text really helps to make the text feel ‘alive’, and this also helped with the interactive aspect which I admired in Inanimate Alice. To enhance this interactive element further, I used triggers, such as when the viewer has to locate and click on the signpost to the castle in order to see the next part of the animation and continue the story.

I created postmodern style tensions in my text by having Tom speaking directly to the reader. I also used techniques such as pathetic fallacy to create atmosphere; in the more intense slide where we hear of the cook’s imminent execution, I used animated rainclouds, lightning, and the sound of thunder. To add to this I created a shadow on the entire video to make it seem darker, and added multiple layers of tense, clashing music. To get these videos and sounds, I used a YouTube to mp3 converter to acquire the authentic Tudor music in my text, and a different converter to download YouTube clips to my desktop in order to publish them in the text.

Who or what will play the lead character?

I have decided to have a lead character to 'guide' the reader through the adventure. This character will speak, meaning I can keep my actual text to a minimum and focus more on the more unusual modes such as sound and music. 

I have considered various characters to act as the ‘guide’ for the reader on this adventure, originally deliberating a Tudor child, of whom the reader (as a child themselves) would be able to directly relate to. I then decided to go down a more ‘gruesome’ route, as I have found this would be more interesting and appealing to a young reader; something for them to go ‘ewww!’ at. I considered a rat and a flea, before eventually deciding on a fly. By choosing a small animal it is easier to have a sense of perspective, the reader is able to appreciate big and small backgrounds, but I mainly liked the idea of having something with wings, so I can create a variety of more unusual angles for the multimodal text, such as flying over a room or scenery. With a fly I can also more modes, such as a 'buzzing' sound effect and hovering custom animation.

After extensively searching for cartoon fly images, I was unable to find anything quite suitable; I wanted a fly with a bit of character to it, so I decided to create my own. This way I could personify it more and make it more relevant to the story by adorning the fly with a typical hat from the Tudor era! I started by drawing my own fly, but I found it difficult to get rid of the off-colour background to paste it seamlessly onto a background picture.

I have ultimately decided to create my own design on the computer itself. This meant I could easily edit the fly in order to change its facial expressions, adding that extra bit of character. I downloaded GIMP (The GNU Image Manipulation Program) and used this as an aid to create and enhance my Tudor fly. This software is brand new to me and I have been finding it relatively difficult to manipulate, but it is useful to try a new way of adding colours and using filters.

Further inspiration: Horrible Histories books

Thinking further into the field of educational history for children, I have looked at the hugely successful and critically acclaimed books (and recently television series), Horrible Histories, a series I myself grew up reading as a child. These are books designed to teach children about key times in history, while still being fun-to-read and containing a juicy bit of gore and gruesomeness.

A collection of some of the Horrible Histories books

It is important to note however that although I intend my text to be thematically similar with a similar 'horrible history' semantic field, the typology and general formal of my text will be vastly different from the Horrible Histories books. These books are written in a conventional black-and-white, words-printed-on-paper format, whereas my text will be the polar opposite to this. However, in the books there are frequent cartoons with amusing quotation bubbles, as well as some fun diagrams. These break up the text into smaller chunks, giving the reader a light break. The author also draws attention to important names and facts by using bold typography.

A look inside the books shows funny cartoons and bold typography...

Looking at the content and style of Horrible Histories has also helped me to find a way of incorporating social realism into my text, as Horrible Histories have proven that gruesomeness is something which can really appeal to children. Therefore, I have decided to touch on one of the most pervasive aspects of social realism: death. The Tudors were notorious for their public executions and their high death-by-execution rate, and therefore my chosen history period will be the Tudor era. I have decided that the ‘adventure’ in my text will be having to find clues, in order to save an innocent person from getting beheaded.

Furthering my ideas for a text

Thinking more about genres, target audience, and intended outcomes, I have decided to have a definitive educational aspect to my multimodal text. I want my readers to actually learn something, and an interactive multimodal text is the perfect platform to enhance a child's learning experience. I was inspired by the primary school children on my recent placement, who used an interactive maths learning platform called 'Mathletics'. I was surprised to see how genuinely excited the children would get when the teacher set a 'Mathletics' assignment for homework, and how a huge amount of the children chose to go on 'Mathletics' during their free-time on the computers.

'Mathletics' is multimodal and interactive, giving 'animated
support' to children, and encouraging them to learn

In my text I too will incorporate a subject from the National Curriculum, as core learning in these subjects is vital. My target audience will be junior primary school children. Subject-wise, thinking back to my own primary schooling, I personally found the subject of history was often taught in quite a dull fashion, and I feel perhaps this is an area which I could improve on with a multimodal history text. I want my readers to realise that learning about history can be fun and exciting! I have therefore decided to take the viewer back in time for an adventure.

As stated in an earlier post, I want to incorporate an aspect of social realism in my text. Genre-wise, my text will be a fictional adventure which will include dimensions of fantasy such as time travel, but it will also be a include non-fictional educational facts, perhaps even a tool that could be used in schools, and thus for this latter aspect I feel it is important that my reader experiences a real-life perspective. This seems a good idea in theory but it has become somewhat of a challenge in practice as I have been furthering my ideas and beginning to design the storyboard for my text. How does one incorporate some hard-hitting social realism into a light-hearted text designed for relatively young children?

Semiotic Analysis: Picturebook Example: 'Winnie the Witch'

I have looked studied one of my favourite picture books from my childhood, to gain a better understanding of the use of different modes in traditional paper books.

As with most picture books, the pictures illustrate the events which are described in the words. This is a large book (slightly bigger than A4 vertically), and the size of the illustrations are very large also. This is in contrast to the relatively small typography and simple typeface of the writing, and therefore this gives the effect of the pictures being the main focus of the book, thus having the effect of an illustrated story. In accordance with this, the illustration in Winnie the Witch takes a semiotic turn, with the illustration carrying extra details which the text does not itself explain, such a visual jokes and irony.

The authors' use of colour juxtaposition is beautiful. They effectively show a stark contrast of the colour black against every other colour. Winnie's cat Wilbur is black, and her entire house is black too, therefore the majority of the picture book is dark and gloomy in colour, creating postmodernistic tension.

However, Winnie continuously trips over her Wilbur, and so she finally decides to magically turn his fur rainbow. This is the first burst of colour we see in the book. The character of Wilbur the cat actually brings an aspect of social realism into this picture book, as Wilbur comes to terms with being taunted and treated as an outcast, first because of his black fur, and then because of his rainbow fur. This can actually link with deeper societal issues such as racism and discrimination.

This book is in its essence about the about power of colour. The colour is used to show characterization and is used as such throughout the book as we have just seen. The over-use of black used at the beginning, juxtaposed with the rainbow ending when Winnie changes the colours of her entire house so that Wilbur can stay black, really expresses the atmosphere and mood of the whole house; gloomy, cold, and dangerous to begin with, and cheerful, warm and safe at the end.

Multimodal text: 'Inanimate Alice'

I have looked at Inanimate Alice as a key example of an interactive multimodal fictional text, similar to the one I will create. Inanimate Alice uses a wide variety of modes to convey the meaning of the text, such as the use of text, drawings, photographs, sound, videos, and animations. These modes often act to support and enhance each other, such as the dangerous atmosphere created by the text and sound, coupled with the graffiti warnings. The stairs in this episode act as a prominent signifier, leading the viewer (and Alice) to the next section, and the text at times is in line with the stairs, angling the viewer and Alice upwards.

The use of bright, over-saturated colour when Alice climbs the stairs effectively connotes danger, and this is juxtaposed with her memories of Moscow which are white/grey-washed and appear to be more hazy dream-like memories, showing the past.

The music is tense and repetitive, reflecting how Alice and the viewer are feeling, and this is vital to build atmosphere and tension. This is coupled with sound effects such as the use of white noise and noise interference between slides, creating an increasingly unsettling atmosphere.

The text is also used as an effective mode as it moves as if Alice herself is moving, as she ‘hauls’ herself onto the step the text makes a slow hauling motion. When Alice’s friends talk this is portrayed through speech bubbles, this is useful to differentiate between speakers and is more interesting than simply using speech marks.

There are also a variety of transitions utilised, swiping up, down, left, right, with the image of a Russian doll as well as others sweeping into the screen. There is the use of layering of many different images (a plane, a journey line, a map, and a pink filter in one example), this creates a deeper and richer quality to the text. There is also a definite interactive element, especially with the use of the hand which lets the viewer choose where to look on the stairs.