Conceptual Issues in Dual Coding Theory

Despite its impressive empirical successes, Dual Coding Theory has by no means gained universal acceptance, and alternative explanations of the relevant phenomena continue to be proposed and defended. Perhaps the most important reason for this is a perceived incompatibility between the theory (and the notion of mental representation it deploys) and the computational approach to the mind that many regard as fundamental to cognitive science. Paivio himself (1986) explicitly rejects the idea that the human mind can be understood on the model of a computer, and presents Dual Coding Theory as the basis of an alternative approach to cognition. Nevertheless, an argument for a Dual Coding as opposed to a common coding account of memory performance is not ipso facto an argument against the computational view of the mind and the existence and theoretical necessity of a mentalese type code. Some cognitive scientists, conscious of the (in some respects, complementary) merits of both Dual Coding Theory and computationalism, would clearly like to reconcile the two viewpoints. It is worth considering whether or how this can be done in a satisfactory way.

Before that, however, there are certain conceptual issues internal to Dual Coding Theory itself that should be examined. Indeed, the concern that these have never been very adequately addressed (by Paivio or, indeed, anyone else) may well be another important source of dissatisfaction with the theory (Lockhart, 1987; but see Paivio, 2007). We will return in later sections to more general worries about the nature and possibility of imagery representation per se, but we should also ask just how we should understand the notion of a code (as Paivio uses it^[1]), and about how such codes are to be differentiated, characterized, and counted.

Indeed, it seems obvious to some commentators (e.g., Kintsch, 1977; Flanagan, 1984) that Paivio, through focusing too much on visual imagery, has counted wrongly, and that if there is an imagery code at all, there must be one for each of the senses: a visual imagery code, an auditory one, an olfactory one, and so on. Indeed, Flanagan (1984) always talks of “six-code theory” rather than dual coding. This may not be an altogether misleading characterization of Paivio's actual view – he frequently refers to imagery as “modality specific” representation – but he clearly regards the distinction by sense mode between different types of image to be orthogonal to the more central distinction of Dual Coding Theory, between the image and the verbal (symbolic) codes. One might go even further than this (although Paivio himself resists doing so) and argue that what we actually have is a unitary, multi-modal imagery system, that handles auditory, haptic, olfactory, etc. memories, as well as visual ones, in an integrated way. We can certainly accept that non-visual aspects of our imagery experience may play a much larger role in our mental life than is commonly acknowledged (Newton, 1982) without endorsing the idea of “six-code” theory. Despite the ingrained bias toward focusing upon its visual aspect, it is quite plausible that imagery is best understood as fundamentally multi-modal. Indeed, the idea that mental images are, and must always remain, strictly segregated by sense mode (such that images of different modalities might interact associatively, but never merge) is compelling only if we regard the senses as independent, segregated channels through which sense impressions of different types are funneled into the mind, and if we also think of mental imagery as arising within those sense channels, or as directly derived from their immediate outputs. This, of course, is the view of perception and imagery that has come down to us from Empiricist philosophy: the perspective from which mental images were taken to be either “decaying sense” (Hobbes, 1651), or copies of former sense impressions (Hume, 1740 p. 1; see Matthews, 1969).^[2]

This Empiricist view of perception continues to be very influential, but it is by no means universally accepted, and many arguments, both empirical and theoretical, have been advanced against it (Morgan, 1977). ^[3] Some theorists, indeed, completely reject the idea of functionally independent senses, and argue that perception can only be correctly understood if the sense organs are viewed not as independent channels, but as components of a single, integrated, perceptual system (Stoffregen & Bardy, 2001; Walk & Pick, 1981; L.E. Marks, 1978; von Hornbostel, 1927). On this view, perceptual experience is inherently multi-modal (although, of course, we often focus our interest on just one particular modal aspect of it), and there is every reason to think that quasi-perceptual experience, imagery, is the same (Thomas, 1987, 1999b n.6). From this perspective, there is no justification whatsoever for segregating imagery codes by sense mode.

However, even for those (no doubt the majority) who continue to conceive of the five (or so) senses as distinct and more or less passive input channels, it does not follow that imagery might not be multi-modal, that the sight and the scent of the rose that we imagine might not be best understood as aspects of a single, integrated imaginative experience rather than separate adventitiously associated visual and olfactory images.^[4] After all, it is quite conceivable that the representations we experience as imagery, and that get stored in long term memory, might arise not in the sensory channels themselves, but, instead, are synthesized at or beyond the point where the deliverances of the senses are brought together to form our polysensory experience of the world. This is consistent both with certain influential contemporary views about perception and imagery (e.g., Pylyshyn 1978, 1999, 2003b) and with some very traditional philosophical perspectives, such as those of Aristotle and Kant.^[5] (It might, however, be harder to reconcile with other influential contemporary views of the nature of imagery, such as Kosslyn's quasi-pictorial theory of visual imagery (1980, 1994). Kosslyn seems inclined to embrace some form of Dual Coding Theory (Kosslyn, Holyoak, & Huffman, 1976; Kosslyn, 1994 p. 335), but perhaps he really ought to prefer the “six-code” version.)

Let us now consider the verbal code. Paivio himself has always made it quite clear that he is thinking of inner speech (and the like) in natural language (in English, or whatever one's native tongue happens to be). However, as mentalese (or “propositional representation”) came to play a key role in cognitive theory in general, and, in particular, to displace covert speech as the central theoretical concept of the dominant common coding view of memory, it seemed natural to some cognitive theorists to construe the two codes of Dual Coding Theory as imagery and mentalese rather than imagery and English (Baylor, 1973; Kosslyn, Holyoak, & Huffman, 1976; Kieras, 1978). In similar vein, Anderson (1983) suggested a tri-code theory, with imagery, verbal, and “propositional” memory codes.

It is not clear to what extent the established empirical advantages of the imagery/English version of Dual Coding Theory (as an account of memory phenomena) would carry over to an imagery/mentalese version (let alone a tri-code theory). However, quite apart from any such concerns, and even if we think (as many do) that there are other good and independent reasons to believe in the existence of a mentalese code, there are still problems with the imagery/mentalese version of Dual Coding. On any viable construal of Dual Coding Theory , the two codes, whatever they might be, are on a par with one another, playing parallel and complementary explanatory roles. One would expect entities with such similar explanatory roles to also be on the same level within the theory's ontology. That seems to be the case for mental images and mental (natural language) words, but it does not seem to be true for images and mentalese representations. Both mental imagery and inner speech are familiar and common phenomena of conscious experience. They are psychological explananda as well as explanantia. By contrast, mentalese representations are explanantia only. Mentalese is not a phenomenon. It is not directly experienced (if it were it would surely have been known about long before 1975!).^[6] Indeed, we have no reason whatsoever to think that mentalese exists unless we believe that it is essential for the adequate explanation of our cognitive powers. Many cognitive scientists do believe this, of course, but from their perspective there are very good reasons to believe (and it is generally believed) that not just cognition in general, but imagery in particular cannot be fully explained without appeal to mentalese. Some hold that mentalese is necessary to account for the intentionality of imagery (Fodor, 1975; Rollins, 1989; Tye, 1991; Kosslyn, 1994 p. 6); others hold that mentalese descriptions are actually constitutive of imagery (Simon, 1972; Baylor, 1973; Moran, 1973; Pylyshyn, 1978, 2002b, 2003b; Kieras, 1978; Hinton, 1979). On either of these views, the imagery/mentalese version of Dual Coding Theory (and Anderson's tri-code theory) mixes ontological and explanatory levels in a disconcertingly awkward and asymmetrical way: on the one hand the mentalese code functions in the theory as a more or less the equal partner of imagery, paralleling its role in the explanation of memory phenomena; on the other, mentalese is more fundamental, grounding the representational power of imagery, or perhaps even its very being.

It may be possible, however, to reconcile some form of Dual Coding Theory with computationalism (and its mentalese, “propositional” representations) in a more elegant and satisfactory manner. It has been suggested that both of Paivio's codes, both mental images and mental words, might, in a more or less symmetrical way, emerge from, be constructed on the basis of, or otherwise be equivalently dependent upon, underlying, unconscious mentalese representations (Morris & Hampson, 1983; Marschark et al., 1987; Marschark & Hunt, 1989). Paivio's actual position, after all, is that the elements of the verbal code are themselves a form or imagery: auditory and/or vocal-motor images of the spoken words of one's native language (or possibly occasionally even visual images of written words) (Paivio, 1971, 1986 p. 57). If this is the case, there seem to be least two elegant and principled ways in which one might conceive of the relation between Paivio's codes and mentalese. One might argue (following Pylyshyn) that just as visual images are ultimately reducible to mentalese representations, so too are the auditory (or whatever) images that constitute mental words. Alternatively, one might argue (following Fodor or Kosslyn) that although imagery may not be reducible to mentalese, nevertheless, all types of mental image, verbal or otherwise, can function as mental representations only inasmuch as they have semantic properties (meanings or intentionality), and they have these semantic properties only in virtue of associated mentalese representations. Although neither of these options are Dual Coding Theory quite as Paivio conceives it (he firmly rejects computationalism and Fodor style mentalese), they can bring essentially the same explanatory resources to bear on the memory phenomena that form the main evidentiary basis for the theory.

But if verbal representations are themselves a form of imagery, as Paivio holds, another problem arises for Dual Coding Theory (quite regardless of whether one thinks its codes must somehow be underpinned by mentalese). All the relevant representations now seem to be imagery of some sort, and we have already argued that we should not differentiate imagery codes on the basis of sensory modality. In what sense, then, can we now justify talking of two distinct codes? Paivio himself holds that the necessary duality arises because there is a greater degree of functional integration within codes than between codes (Paivio, 1986). The idea, apparently, is that associative connections between different verbal images are stronger, or more numerous, or form a more richly interconnected network, than do associative connections between verbal images and non-verbal ones (and vice-versa). However, it is by no means clear that this difference of degree is an adequate basis for differentiating the codes, or even that it is real. After all, Paivio frequently insists (and the overall theory requires) that there are rich associative interconnections between the two codes, and is it by no means obvious (nor, to the best of my knowledge, is it empirically established) that, for instance, my mental image of the word ‘cat’ is more strongly or richly associated with mental images of other words (‘fur,’ ‘claws,’ ‘meow,’ ‘dog’) than it is with mental images of cats, cat fur, cat claws, etc.

There may, however be another way of suitably differentiating the codes. Regardless of whether their representational power is intrinsic, or is grounded in mentalese (or something else), non-verbal images represent (or are normally taken or used to represent) whatever they are images of – my image of the Eiffel Tower represents the Eiffel Tower for me, my auditory image of the sound of a trumpet represents the sound of a trumpet, and so on. By contrast, verbal images (except for the rare occasions when we are thinking about rather than merely in language) function symbolically: they are not used to represent what they are images of (i.e., some spoken or written word or words), but, rather, what the imagined word or words means (whatever one may take the meaning relation to be). My auditory image of the words ‘Eiffel Tower’ is not normally used to represent the words ‘Eiffel Tower’ (as they might be spoken on some occasion), but, rather, the Eiffel Tower itself. Mental images (as elements of the imagery code) represent what they are images of; mental words (when functioning as elements of the verbal code) represent what the words mean. The two codes thus employ quite different representational modes. This is not a mere difference of degree, and (although the matter needs further exploration) it seems possible that it might be an adequate basis for the duality that Paivio's theory requires.