Are icons pictures or logographical words? Statistical, behavioral, and neuroimaging measures of semantic interpretations of four types of visual information

This dissertation is composed of three studies that use statistical, behavioral, and neuroimaging methods to investigate Chinese and English speakers’ semantic interpretations of four types of visual information including icons, single Chinese characters, single English words, and pictures. The goal is to examine whether people cognitively process icons as logographical words.

By collecting survey data from 211 participants, the first study investigated how differently these four types of visual information can express specific meanings without ambiguity on a quantitative scale. In the second study, 78 subjects participated in a behavioral experiment that measured how fast people could correctly interpret the meaning of these four types of visual information in order to estimate the differences in reaction times needed to process these stimuli. The third study employed functional magnetic resonance imaging (fMRI) with 20 participants selected from the second study to identify brain regions that were needed to process these four types of visual information in order to determine if the same or different neural networks were required to process these stimuli.

Findings suggest that 1) similar to pictures, icons are statistically more ambiguous than English words and Chinese characters to convey the immediate semantics of objects and concepts; 2) English words and Chinese characters are more effective and efficient than icons and pictures to convey the immediate semantics of objects and concepts in terms of people’s behavioral responses, and 3) according to the neuroimaging data, icons and pictures require more resources of the brain than texts, and the pattern of neural correlates under the condition of reading icons is different from the condition of reading Chinese characters.

In conclusion, icons are not cognitively processed as logographical words like Chinese characters although they both stimulate the semantic system in the brain that is needed for language processing. Chinese characters and English words are more evolved and advanced symbols that are less ambiguous, more efficient and easier for a literate brain to understand, whereas graphical representations of objects and concepts such as icons and pictures do not always provide immediate and unambiguous access to meanings and are prone to various interpretations.