Grapheme-colour synaesthesia improves detection of embedded shapes, but without pre-attentive ‘pop-out’ of synaesthetic colour

Jamie Ward, Clare Jonas, Zoltan Dienes, Anil Seth

Abstract

For people with synaesthesia letters and numbers may evoke experiences of colour. It has been previously demonstrated that these synaesthetes may be better at detecting a triangle made of 2s among a background of 5s if they perceive 5 and 2 as having different synaesthetic colours. However, other studies using this task (or tasks based on the same principle) have failed to replicate the effect or have suggested alternative explanations of the effect. In this study, we repeat the original study on a larger group of synaesthetes (n = 36) and include, for the first time, an assessment of their self-reported colour experiences. We show that synaesthetes do have a general advantage over controls on this task. However, many synaesthetes report no colour experiences at all during the task. Synaesthetes who do report colour typically experience around one third of the graphemes in the display as coloured. This is more consistent with theories of synaesthesia in which spatial attention needs to be deployed to graphemes for conscious colour experiences to emerge than the interpretation based on ‘pop-out’.

1. Introduction

People with grapheme-colour synaesthesia experience reliable colour sensations whenever they see letters and/or numbers (Hubbard & Ramachandran 2005; Ward & Mattingley 2006), and sometimes when they hear speech (Baron-Cohen et al. 1993; Paulesu et al. 1995) or think about letters or numbers (Dixon et al. 2000). Ramachandran & Hubbard (2001a) reported an influential experiment to demonstrate the authenticity of grapheme-colour synaesthesia, termed the ‘embedded shapes task’. They studied two synaesthetes who were shown arrays of achromatic graphemes for a brief period (1 s). Some of the graphemes were arranged into one of four shapes (diamond, square, rectangle or triangle). For example, there might be a triangle made up of Hs against a random background of Ps and Fs. The two synaesthetes did significantly better than the control group (81% correct in synaesthetes versus 59% correct in controls), suggesting that they may have seen the achromatic graphemes as coloured, thus enabling them to see the embedded shape. One reason why this result was considered a convincing demonstration for the authenticity of synaesthesia is that superior performance on a perceptual task is hard to fake.

This finding was replicated by Hubbard et al. (2005); five of their six synaesthetes performed significantly better than controls. However, Rothen & Meier (2009) failed to replicate the result in a group of 13 synaesthetes. Other studies have used visual search paradigms in which a single target (e.g. 2), rather than an embedded shape, must be detected among an array of distractor graphemes (e.g. 5s) and response times are measured. As in the embedded shapes task, the stimuli are physically achromatic but assumed to generate synaesthetic colours, thus facilitating their detection. Studies using this and related paradigms have yielded mixed results. Some show no benefit at all (n = 23 participants in the following studies combined: Edquist et al. 2006; Sagiv et al. 2006; Gheri et al. 2008), although some single case studies do show a benefit (Smilek et al. 2001, 2003; Palmeri et al. 2002; Laeng et al. 2004).

There are several issues at stake here beyond the replicability of Ramachandran & Hubbard (2001a). First of all, the embedded shapes test has been widely publicized as offering strong proof of the authenticity of synaesthesia and its perceptual nature. These fundamental claims have been cast into doubt by some researchers (Gheri et al. 2008). Second, the results of Ramachandran & Hubbard (2001a) pose important questions for theories of perception and attention outside the domain of synaesthesia.

For people without synaesthesia, searching for a shape or other target is enhanced if the colour of the target differs from the surrounding distractors (e.g. Treisman & Gelade 1980). The standard explanation for this is that the colour information is processed automatically (pre-attentively) and in parallel across all the items in the display, so the target shape appears to ‘pop out’. In situations in which colour does not discriminate between targets and distractors (e.g. all are achromatic, or some distractors are the same colour as the target) then participants are assumed to engage in a more time-consuming strategy in which the focus of attention moves from location to location until the target shape is found. This is termed ‘serial search’. Thus, better performance by synaesthetes is often interpreted as a greater reliance on faster ‘pop-out’ and less reliance on slower serial search (Ramachandran & Hubbard 2001b, 2003b). However, this theory as applied to non-synaesthetic visual search assumes that colour and shape are processed independently. This assumption does not hold for synaesthesia given that some amount of grapheme processing must be required for the colour to be induced. As such it is unrealistic to expect synaesthetic colours to behave ‘just like real colours’ on these tasks.

There are at least two possible ways that the mixed findings could be resolved. The first assumes that attention and serial search is required in synaesthesia, just as it is in visual search for feature conjunctions of colour and shape in non-synaesthetes. In such situations, synaesthetes may experience a small proportion of graphemes as being coloured (i.e. those graphemes within the window of attention) and this could offer them a modest advantage in the absence of pop-out. It may enable local grouping on the basis of colour (e.g. detecting one edge of a triangle), or may facilitate rejection of distractors. It is to be noted that previous studies have not assessed what synaesthetes actually claim to see in these tasks. By definition, all grapheme-colour synaesthetes claim to see colours under free viewing conditions, but this may not hold true for large arrays of graphemes presented with brief exposure. The second way of resolving these mixed results is to assume that there are individual differences between synaesthetes. One noted difference is between synaesthetes who experience colours subjectively bound to the observed grapheme (so-called ‘projectors’), versus those who experience the colour in their mind's eye (so-called ‘associators’, for whom the colours are often bound to a ‘copy’ of the seen letter on some ‘inner screen’; Dixon et al. 2004; Ward et al. 2007). Many of the demonstrations of superior performance in embedded shapes/visual search have come from projectors (but see Smilek et al. 2001, 2003; Palmeri et al. 2002; Edquist et al. 2006), leading to the suggestion that projectors experience synaesthetic colours pre-attentively but the more common associators experience them post-attentively (e.g. Dixon & Smilek 2005). Ward et al. (2007) offer a different interpretation of this distinction. They suggest that both types of synaesthesia require attention for accurate binding of colour to grapheme, but that projectors are more likely to be aware of synaesthetic colours (in brief presentation) because, for these individuals, their synaesthetic percepts are in the same spatial location as the attended stimulus itself. Other types of grapheme-colour synaesthesia require a shifting/dividing of attention between the location of the stimulus and the location of the colour, and this comes at a cost (slower identification of synaesthetic colours, less awareness of synaesthetic colours when attention is directed elsewhere).

Our present experiment is based closely on the experiments of Ramachandran & Hubbard (2001a) and Hubbard et al. (2005). As in the preliminary study by Ramachandran & Hubbard (2001a), we used stylized 5s and 2s that are the mirror image of each other.1 These stimuli have been extensively reproduced elsewhere to demonstrate the phenomenon of synaesthesia (e.g. Ramachandran & Hubbard 2001b, 2003a), because low-level visual features cannot be used to disambiguate the graphemes (both consist of two vertical and three horizontal lines). In addition, we asked synaesthetes to report what they saw on a trial-by-trial basis (e.g. what percentage of graphemes appeared coloured?) and we considered individual differences in the perceived location of synaesthetic colours (projectors versus associators).

2. Material and methods

(a) Participants

There were 36 grapheme-colour synaesthetes tested (mean age = 34.3 years, range = 12–65; 3 males). In addition, there were 36 control participants who reported no synaesthesia (mean age = 33.9 years, range = 14–61; 3 males). All synaesthetes passed a measure of consistency for the colour associations (see the electronic supplementary material). Synaesthetes were classified as projectors if they reported that their synaesthetic colours appeared to be located on or very close to the page (during unconstrained viewing of text) both in our initial questionnaire and in a subsequent illustrated questionnaire (Skelton et al. 2009).

(b) Materials

The arrays were presented in a single block of 56 trials. The shapes were made up of 6 to 10 target graphemes, and there were 41 distractor graphemes. Four shapes were used: triangle, diamond, rectangle and square. Each grapheme was 0.33° × 0.41° in size, and the embedded shapes filled an area approximately 3.1–4.4° wide and 2.3–3.4° high. The display did not make full use of the screen but instead used the central 11.7° × 8.6° area which was indicated by a black outline. The embedded shapes were presented in different locations in this area and not just close to the centre. All displays consisted of black graphemes on a white background. As several different sized monitors were used throughout testing, the distance between participant and monitor was varied so that the visual angle of the display was constant (e.g. for an 18 inch monitor the viewing distance was 104 cm). By restricting our choice of graphemes to 5s and 2s we were unable to control the colour experiences (e.g. to ensure a red target grapheme, against green distractor graphemes) although we did ascertain that the colours for 5 and 2 were perceived to be different by the synaesthetes.

(c) Procedure

Participants were given a single practice trial in which the stimulus was presented for as long as they wished. They were then informed that subsequent arrays would be presented for one second only. They were informed that the first half of the block consisted of shapes made of 2s, and the second half of shapes made of 5s (an instruction screen informed them of the change at the midway point). Before performing the task, synaesthetes were assured that ‘Some people may not experience any colours when doing the task and this is fine. It is still important data for us and it doesn't mean that you don't have synaesthesia.’ This was included to discourage synaesthetes from reporting colour as a demand artefact.

The procedure on an individual trial was as follows. The participant pressed any button to start the first trial. They were free to move their eyes across the array. After one second, the array disappeared and was replaced by instructions that prompted participants to answer three questions. These were not timed. First, they had to choose the shape that they thought had been presented from the four alternatives, guessing if unsure. Second, they were asked to rate the vividness of any synaesthetic colour experience on a 1 (=no colour) to 6 (=very vivid colour) scale. Finally, assuming that they saw a colour (i.e. an intensity rating more than or equal to 2) they were asked to estimate the percentage of digits in the array that they saw as coloured. (There is evidence from non-synaesthetes that accuracy on this kind of task is generally high; Treisman 2006.) Controls were instructed to ignore the last two questions. After answering the questions, they pressed a button to start the next trial.

3. Results

Synaesthetes obtained a mean score of 41.4 per cent (s.d. = 16.9) compared with a score of 31.5 per cent (s.d. = 9.9) obtained from controls. Chance performance is 25 per cent. An independent samples t-test revealed that synaesthetes performed significantly better than controls on this task (t(70) = 3.04, p < 0.005). While the highest-scoring control obtained a score of 50 per cent, there were 10 synaesthetes who obtained a score of 50 per cent or more. The effect size is medium (Cohen's d = 0.68) and supports the original findings of Ramachandran & Hubbard (2001a; see also Hubbard et al. 2005). The distribution of scores for the two groups is shown in figure 1 (see also the electronic supplementary material).

Figure 1.

The distribution of scores (out of 56) for synaesthetes (black bars) and controls (unfilled bars).

Figure 2 shows the number of trials (out of 56) for which each synaesthete reported some synaesthetic experience of colour. The most common report was of no experiences of synaesthetic colour on any trial. The next most common report was that 100 per cent of trials contained some synaesthetic colour. That is, the distribution is bimodal. It is unusual to find synaesthetes claiming to see a roughly equal amount of coloured and non-coloured. Those synaesthetes classed as projectors reported more coloured trials than other grapheme-colour synaesthetes (Mann–Whitney U = 48, p < 0.005; a non-parametric test was used owing to the non-normal distribution). In fact, every projector that we tested reported some experience of colour when performing the task (100%; 9/9) compared with around a third of the remaining synaesthetes (37%; 10/27).

Figure 2.

The number of trials (out of 56) in which synaesthetes claimed to have some experience of colour. The most common pattern was to report no colour experiences at all. The next most common pattern was to report 100 per cent of trials as containing some colour. Synaesthetes who report colour experiences subjectively projected on the page are shown in black.

For those synaesthetes who do claim to see colour, the average percentage of coloured graphemes reported was 30.8 per cent (s.d. = 29.5). Figure 3 contains a possible depiction of what that might look like. The average intensity of colours (on coloured trials) was reported to be 2.9 (s.d. = 0.84) on a 1–6 scale. For these analyses, the grand average was weighted across participants rather than across trials. Thus, a synaesthete who reports colours for 14 trials would have the same contribution to the mean as a synaesthete who reports colours on all 56 trials. The percentage of coloured graphemes and their intensity did not differ between projectors versus other grapheme-colour synaesthetes reporting colour on this task. Projectors report 30 per cent (s.d. = 33.6) of graphemes as being coloured versus 31.5 per cent (s.d. = 27.1) for other grapheme-colour synaesthetes (t(17) = 0.1, n.s.), and the mean intensity ratings were 3.2 (s.d. = 0.8) and 2.7 (s.d. = 0.8), respectively (t(17) = 1.3, n.s.). Thus, being a projector increases the likelihood that colours will be experienced on a trial, but it does not increase the proportion of graphemes in the array that are judged to be coloured, nor the intensity of those colours. If projectors were experiencing colours pre-attentively but associators were experiencing them only after serial search then we would have expected projectors to report more coloured graphemes per trial (as opposed to, or in addition to, more coloured trials).

Figure 3.

A depiction of what a synaesthete might see in this task based on the phenomenological reports of this study.

How does self-reported colour experience relate to objective task performance? Synaesthetes who reported more than 80 per cent of trials as coloured (n = 15, mean = 42.7% correct) were compared with those experiencing less than 20 per cent as coloured (n = 18, mean = 40.5% correct), but there was no difference between these groups (t(31) = 0.4, n.s.). Similarly, projectors did not outperform other grapheme-colour synaesthetes on this task (projectors = 43.9% correct; other synaesthetes = 40.7%; t(34) = 0.5, n.s.). Although this suggests no relationship between synaesthetic phenomenology and task performance, it should be borne in mind that experiencing colour per se may not be sufficient for performing the task. For instance, if only 31 per cent of graphemes on individual trials are coloured then this may or may not be helpful, depending on whether the critical graphemes comprising the shape are perceived as coloured. While we have no way of knowing which actual graphemes were perceived as being coloured, there was a small number (n = 5) of synaesthetes who claimed to perceive the majority of graphemes in the array as coloured (i.e. at least 50% of the graphemes). These synaesthetes did outperform other synaesthetes who experienced colour in a more local/limited fashion (n = 14, t(17) = 3.29, p < 0.005) and synaesthetes who reported no colour at all during the task (n = 17, t(20) = 2.37, p < 0.05), the means for these three groups being 62, 34.3 and 41.3 per cent correct, respectively (the latter two groups did not differ significantly, p > 0.1). As such, performance on this task can be enhanced by the presence of synaesthetic colour but particularly when the colour is distributed across many graphemes. In order to ascertain how these synaesthetes were able to perceive so many colours they were contacted again, shown the stimuli material as before, and asked whether the synaesthetic colours across the display appeared instantly, all in one go, or whether they appeared section-by-section over time. All reported the colours appearing piecemeal. A typical reply was: ‘I definitely do NOT see all the colours in one go. I have to attend to the symbols/shapes or process them in some way, and then it has a colour attributed to it. It's not like I could be looking somewhere else, and in the corner I see a shape made out of shapes of one colour.’

4. Discussion

It has previously been found that synaesthetes are better able to detect an embedded shape comprising target graphemes among distractor graphemes (Ramachandran & Hubbard 2001a; Hubbard et al. 2005). However, the effect has not always been found by other research groups using tasks that are conceptually related to the embedded shapes task (e.g. Edquist et al. 2006; Gheri et al. 2008; Rothen & Meier 2009). A variety of explanations have been proposed for this discrepancy, such as that cases showing superior performance are statistical outliers (Rothen & Meier 2009) or project their colours externally (Dixon & Smilek 2005). Some researchers have even used negative evidence on a task related to embedded shapes to question the credibility of synaesthesia per se (Gheri et al. 2008). We aimed to discriminate between these competing accounts. Given that any non-trivial explanation of superior performance by synaesthetes is related to the assumption that they are able to use their synaesthetic colours during the task, we also asked our synaesthetes to report the presence/absence of colour experience on a trial-by-trial basis. Our results demonstrate that synaesthetes, on the whole, do significantly outperform controls on this task, consistent with Ramachandran & Hubbard (2001a) and Hubbard et al. (2005). Superior performance was not linked to the number of trials in which synaesthetic colour was experienced, but was related to the proportion of graphemes that were noted to be coloured. Many synaesthetes reported no colours at all during the task, and those who did report colours typically reported that only a minority of graphemes in the array were coloured. The latter is inconsistent with the notion that synaesthetic colours are triggered pre-attentively across a large portion of the visual field, and is more consistent with the notion that synaesthetic colours are induced within a circumscribed locus of attention.

Different components of vision (e.g. colour, shape, motion) are processed by partially independent mechanisms in the visual stream and it is suggested that attention, operating over spatial representations of objects in the visual field, may be required to bind together these different attributes (Treisman & Gelade 1980; Treisman 1988). If a red ‘A’ is shown to a (non-synaesthete) participant, grapheme recognition mechanisms may detect ‘A’ and colour-sensitive mechanisms may detect red but spatial attention may be required for these attributes to be combined into a perceptual whole that is treated as (and experienced as) a single object rather than a collection of parts. It is suggested (e.g. Robertson 2003) that attention may serve essentially the same purpose in synaesthesia, except that the binding is between a veridical component (the achromatic letter ‘A’) and an illusory component (a synaesthetic sensation of red). Synaesthesia is enhanced when the inducing stimulus is near the focus of attention (Sagiv et al. 2006) or when attention is deployed on a perceptually easy versus hard task (Mattingley et al. 2006).

It has been suggested that synaesthetes classified as projectors may be able to experience synaesthetic colour without attending to (and consciously identifying) the inducing grapheme (Smilek et al. 2001; Wagar et al. 2002; Dixon & Smilek 2005). Alternatively, attention to the inducing grapheme is needed to experience colours, but projectors are more likely to be aware of the induced colours because they are experienced in the same spatial location as the attended stimuli rather than appearing ‘in their mind's eye’ (Ward et al. 2007). In the present study, projectors were more likely to be aware of synaesthetic colours during the task, but they tended to report that a minority of graphemes in the array were coloured rather than perceiving the whole array as coloured so that the embedded shape ‘pops out’. This is consistent with the view that projectors (like other grapheme-colour synaesthetes) require spatial attention for the conscious binding of synaesthetic colour to grapheme, but they tend to be more aware of synaesthetic colours because the colours are (by definition) in the locus of attention.

There is no convincing evidence from our study of ‘pop-out’ in grapheme-colour synaesthesia. The percentage of correct trials is far lower than one would expect if colours were perceived across the array (e.g. based on data from Hubbard et al. 2005). Moreover, the majority of synaesthetes do not report colours across the entire array and, for those who do, the colours are noted to emerge as their focus of attention shifts around the array. This is perhaps not surprising given that synaesthetic colours depend (to some degree) on perception of the associated grapheme, unlike regular visual search paradigms in which colour perception is independent of grapheme processing. Nevertheless synaesthetes do outperform controls on this task, so what might explain this, if not pop-out? If synaesthetic colours emerge within a window of attention but are not necessarily restricted to single graphemes then this could allow localized grouping within that region (based on the Gestalt similarity principle). Just seeing two red graphemes in a vertical or slanted line would provide important clues as to the shape's identity. This may explain why some synaethetes who do well on the embedded shapes test (e.g. cases CHP and AAD in Hubbard et al. 2005) do not necessarily do well on regular visual search (reported as CP and AD in Sagiv et al. 2006). This kind of mechanism may also facilitate the search process (where to look next) by directing attention to a specific location or preventing return of fixation to the inspected area. The extent to which this mechanism depends on participants being aware of the synaesthetic colours is debatable. It is possible that synaesthetic colours could influence performance even if the participant is not aware of them, if, for instance, the colours are in a different spatial location.

Most models of vision, attention and awareness aim to offer an account not only of perception in the neurotypical population but also try to explain atypical vision, attention and awareness arising from genetic differences or acquired brain damage. The form of synaesthesia studied here affects as many as 1 to 2 per cent of the adult population (Simner et al. 2006), and there is an expectation that we will understand synaesthesia in terms of an adaptation/variation of some basic mechanism rather than as a stand-alone entity (Bargary & Mitchell 2008; Ward 2008). With regards to the issue of binding different perceptual attributes (e.g. graphemes and colours) into a consciously experienced whole, Treisman (2005) lists three potential mechanisms that could be relevant to synaesthesia. First, different attributes may appear to be experientially linked due to synchrony of firing between two different neural populations. At present there is no evidence from synaesthesia on this issue. Second, there are ‘conjunction detectors’ in the brain consisting of, for example, neurons that respond when certain auditory and visual stimuli are presented together (Stein & Stanford 2008). One suggestion is that, in synaesthesia, such ‘conjunction detectors’ may be over-activated by unimodal stimuli such that an auditory stimulus activates audio-visual mechanisms (Goller et al. 2009); or, similarly, neural populations normally responding either to graphemes or to colours have failed to differentiate their response pattern during development (Baron-Cohen et al. 1993). This kind of mechanism is likely to be highly relevant to synaesthesia, but it does not account for the circumstances in which synaesthetes are shown to be aware or unaware of their synaesthetic colours. At least in visually complex arrays, the third type of binding mechanism discussed by Treisman (2005) appears to be crucial—namely, attention to the spatial location of the inducing stimulus.

Acknowledgements

This research was supported by a pump-priming grant to A.S. from the Department of Informatics, University of Sussex. Kristina Spenner, Kate Spurr and Jennifer Ware assisted in data collection for this study.

Footnotes

References

View Abstract