The Coded Picture

In the recent book, Codes, editor Paul Lynde describes Victorian artists practice of embedding "problem" narratives into their complex paintings. Clues to this seemingly innocent (at first glance) little scene are found in the glove lying on the floor, the hat on the table (he is a visitor!) and the paradisical exterior, to name a few. All such clues rely on a deep but shared cultural vocabulary of images and symbols, which taken together, describe the situation and comment on it. The fun is in the decoding of course, and sharing a moral point of view.

In film, the "problem" picture is a fully developed trope; i.e. the whodunit. How many thrillers and mysteries have opened with a clue-laden, but unhinted scenario, only later to be fully explicated at the climax of the movie? A shorter, and now widely-seen version of the same idea is the basketball passing clip, which viewed naievely, is a snooze, but on second view surprises (prepared) viewers (spoiler alert) with a perfectly clear walk-through of a man in a gorilla mask. What's impressive is that almost nobody catches the anomalous walker-by on first look, and the lesson is that we see, not necessarily what is there, but what we expect to see there.

This is structurally similar to "Where's Waldo" and those embedded word drawings that used ot be a standby of the old color comics page in the Sunday paper. Looking for hidden clues in a still, as above, or a motion sequence,

From a constructivist perspective, "problem pictures" could be a valuable tool, perhaps in unexplored ways. How much more could be done with short clips, set up with an invitation to "decode" them? What about applying a previously-learned scheme to a real-world scene, then discussing it in small group for a while? Reversing it could also work: where a scene is viewed prior to the presentation of the scheme or theory.

Applied Research: The Worked Example

It is hard to find any empirically tested media tropes, but here is one: the worked-example effect, predicted by cognitive load theory (Sweller, 1988). According to him, this is the best known and most widely studied of the cognitive load effects.
"A worked example is a step-by-step demonstration of how to perform a task or how to solve a problem” (Clark, Nguyen, Sweller, 2006, p. 190). Studying worked examples is an effective instructional strategy to teach complex problem-solving skills (van Merriënboer, 1997). This is because example-based instruction provides expert mental models, to explain the steps of a solution for novices.

Researchers Sweller and Cooper looked first at the teaching of algebra. Early on, they saw that merely showing worked problems was not any better, learning-wise, than merely having students solve problems on their own. With sufficient structuring, however, which reduces cognitive load, the method appeared to work better than unaided solutions.  Importantly, for media designers, they found that (paper) display of both  text and diagrams reduced cogntive load, as we covered in an earlier post (Split-attention effect). Sweller later proposed applications for animation, (closer to home for us).

Not for beginners
The folks this strategy works best for are the more experienced students, and just a little "worked" material seems to be sufficient, follwed by multiple experiences of unaided solving. All of this is related to a more general body of thought, coming out of Dewey, Piaget, Montessori, Vygotsky, and others, called Problem-based learning (PBL) where students collaboratively solve problems and discuss their experiences as they go, or immediately following the exercise.This is not a matter of dumping the non-swimmers into the deep end. PBL assumes some earlier, didactic training in various problem solving strategies and heuristic reasoning first. In this view, the teacher's role quickly shifts however, after providing sufficient theorectical and procedural structure, to that of a facilitator of active collaboration by students facing a challenging, practical problem. The payoff is solving in context, with the complexity of the real-world reflected in the process as well as the outcome.

This techniqie has been applied extensively in professional training for physicians and engineers, and is one I'm entertaining for the training of non-profit professionals in recruiting, fund-raising and enrollment tasks.

Most of the citations I've found are in classroom or lab settings. Applying the technique to video and online will take some imagination. Clearly, the media will have to model a process as well as introduce specific "challenge" materials to be solved.

.

Notes on Mutimedia Theory (2)

Here are the constituents of we might call modern multimedia theory, and their founders (... and for which we can thank Richard E. Meyer for lacing together in his wonderful book, Multimedia Theory)

Dual Coding Theory, Pavio
Working Memory, Baddeley
Cogntive Load, Sweller
Generative Theory, Wittrock

For all of these, perhaps the overriding component is this last, the generative notion of the mind (mind defined, for now, as simply as "what the brain does") as an actor, building meanings, testing them against evidence and knitting them into what has already been stored away in long-term memory.

The old model of the brain was a passive one. It fit well into the industrial model of schooling, with boxes and bells and students sorted by age. Sitting in those classrooms, we took in knowledge when it was placed in front of us, and our labor was that of mere retention for regurgitation at testing time. The emphasis was on all the social and spatial arrangements around the student, about teaching and not about learning.

This is important, I think, because we still see that old passive brain model reflected in schooling and in much of "adult education" and industrial training today. "Sit 'n git" still rules in too many places when we really should know better.

An active brain model is better suited to these times when machines are taking over for teachers in much of the non-school teaching establishment (....oh, the machines are coming to the schools alright, but we will probably hold out there for decades.)  The question for MM designers and educational technologists is what ought we be doing with these machines to make the best use of what we now know about human learning?

If the active brain is the foundation, dual coding theory is the first story of the emergent model. There are two channels for us humans; one for processing visual information, the other, auditory. Scholars have gathered a considerable body of knowledge about the interactions of these two channels and the media objects they encounter.

Oddly, a third channel, the haptic/kinesthetic, does not seem to have made much of a stir in mainstream, academic discourse yet, even though a century of experience with Montessori's constructivist ideas seems to be well-supported by findings on the value of movement, physical media, collaboration and discovery, not just for children but life-long learners. The interface and game designers are way ahead of the academicians, though, at least on the haptic dimension. (A good topic for a later inquiry.)

 Working memory and cognitive load are closely interlinked. More on that next time.

Multimedia Theory (1)

There is no grand unified theory for MM yet, but there are some well-accepted sub-theories that seem to hang together pretty well, and which could offer some practical advice for the multimedia designer.

Cognitive Load Theory is one such sub theory, established in the late '90's. Based on the metaphor of a computer's processing limitations as applied to the brain, the theory predicts that when humans are overloaded, their ability to process new information (learn)is impaired. The so-called split-attention effect was cited as evidence for the theory (a decrease in learning efficiency when exhibits carry redundant information). This choking down of our mental processes is more than just a matter of taking in too much information over time, like water slowing as it moves through a narrowed pipe. The brain is not a passive container, but an active processor in learning; we don't learn what we perceive, but what we construct out of what we perceive. This active process of building up new mental models, then relating them to our previous knowledge takes up mental resources. Any redundancy or overloading that inhibits this internalizing process defeats its own educational objectives.

Learning is not a result of being exposed to an exhibit, or simply recalling a memory, it is work, and takes energy and resources. "Exhibit minimalism" is the best practice. Hence, "cogntive load" suggests the following MM rules of thumb:

Don't put animation, narration and text on the screen at the same time. This provides distraction and makes learning harder.

Don't add extraneous visual or auditory details to an explanation. Same as above.

Don't place on screen (or page) text apart from the visuals you are trying to describe. "Working memory" is limited too, so the less one has to carry around the exhibit in recall, the better off they are.

Likewise, don't separate such explanatory items in time.