index.html   Frameworks for understanding diagrams:
Enriching the debate



This paper responds to renewed interest in the centuries old question of what is a diagram.  Existing status of our understanding of diagrams is seen as unsatisfactory and confusing particularly when responding to problems of accounting for the variety of diagrams, handling change in diagrams in a well formed way, and all of this in the context of semantically mixed diagrams. Existing frameworks (Peirce, Bertin, Engelhardt) for understanding diagrams are discussed and evaluated against these problems. These frameworks are then contrasted for similarities and differences with our proposal which is based on, SySpM, Symbolic and Spatial Mapping. Potential and contributions of the proposed framework are briefly mentioned. Keywords: SySpM: Symbolic, Spatial, Mapping. WF: Well-formedness.


1. Introduction

Diagrams are a very old form of representing and making sense of the world around us. Some 2500 years ago people have used clay tablets to express boundaries, groupings, and routes, Figure 1.


Fig 1: Clay Tablet map from Ga-Sur, Kirkuk, 2,500 B.C.

However, despite this close and rich involvement of diagrams in human history people are now again still asking centuries old question “What is a diagram?”, [26]. [39] asks the question “What are arrows?”. [1] ask “What constitutes a diagram?”. [4] adds to these calls by questioning our existing understanding of diagrams arguing that centuries held assumption that “a drawing-is a drawing-is a drawing” is progressively shown to be invalid.


The reemergence of questions of understanding diagrams does not come as a surprise especially when we consider the complexity of the field under investigation. [14] has identified three main problems that need good answers to advance our understanding of diagrams. We list them here but a full discussion found in the given reference.


Variety of diagram types

Handling change in diagrams and maintaining their well formedness

Semantically mixed diagrams


2. Existing approaches to understanding diagrams

Existing ways of understanding diagrams is divided mainly into two broad camps. One is the scientific ways and the other are the paradigmatic/philosophical ways. Both ways are introduced and evaluated against the problems.

3. The scientific approaches

[17] summarises in broad terms the basic approaches currently adopted for understanding of diagrams. These are:

Theoretical approaches: the nature of diagrams - examined by art, philosophy, mathematics and graphic design; the syntax and semantics of diagrams; formal properties of diagrammatic notations, formal theories of human communication and human-computer interaction through diagrams; the computational tractability of diagrams.

Psychological approaches: visual perception; diagram comprehension and inference; the interplay between diagrammatic representations and human reasoning (including individual cognitive differences); generation and manipulation of diagrams for inference purposes; diagrams in education; the cognitive effectiveness of diagrams.

Computational approaches: diagram interpretation and parsing; visual language grammars; visual program execution; the generation and manipulation of diagrams for displaying the results of executions; data visualization; diagram animation; diagrammatic interfaces; intelligent diagrammatic reasoning systems.

3.1 Evaluation of theoretical approaches

Humans construct theories in order to explain, predict and master some aspect of the natural world. They do this by abstracting the world they theorise about into a certain perspective which has then to be tested and proven. Theoretical abstraction tends to reject competing theories as either incorrect or having sharp differences with each other. As a consequence what is seen as valuable or interesting by some is found irrelevant and ignored by others. Theoretical approaches also inclined towards producing formal accounts of diagrams. This makes such an approach less suitable for addressing the contexts of informal, everyday diagramming, or when diagramming is part of thinking, because all of the contexts evolve around several different types of diagrams and require a certain level of rule relaxation. Furthermore, such diagrams are often drawn for different purposes and each purpose is guided by a particular perspective. Theoretical approaches tend to reject other perspectives. [5] argue that an interdisciplinary field such as diagrams cannot afford to concentrate only on formal analyses as a way of accounting for the nature of diagrams. They point out how existing accounts of understanding diagrams have ignored issues of interpretation because they are less easily formalised. Purely theoretical approaches not only create problems for accounting for the diversity of diagrams and how software is to be developed to support this diversity, but also limit the scope of research into diagrams. [9] argue that researchers tend to focus their attention on specific types of diagrams which are supported by certain well known and established formalisms and ignore other more useful types of diagrams which are less formalised. This feature of theoretical approaches limits their suitability for addressing the variety of diagrams problem. [5] also show in their survey of the different ways of accounting for the nature of diagrams a lack of approaches that are concerned with performance of diagrams. They attribute this limitation to the fact that performance is less easily formalised. This feature of theoretical ways limits their capability to account for well formedness, which forms part of the problem of handling change.

3.2 Evaluation of psychological approaches

The focus of this approach is on the psychological processes that occur when people see and interpret diagrams. [11] and [12] studied the comparisons of quantitative information and the relative goodness of a set of physical dimensions for ratio judgements. Based on the psychological studies of integral and separable dimensions, [6], proposed the proximity compatibility hypothesis to account for a variety of graphics problems, especially those that involve information integration. Another approach has focused on the perceptual and cognitive processes involved in graph and diagram comprehension, [21]; [7]; [32]. In these studies the perceptual and cognitive processes involved in graph comprehension are analysed and modeled by computational systems such as production systems and semantic networks. On the basis of psychological studies, several researchers have developed computer systems that can automate the design of graphics, [23]. Psychological accounts are directly linked and constructed from stimuli received from our senses. [15] defines a diagram as a “visual language sign”. [34] drawing on the theory of Percepts and Concepts, sees diagrams in terms of sensory data perceived and conceptual relations (interpretations) are formed: “a conceptual graphs describes the way percepts are assembled”. The emphasis of these approaches on perceptual or visual perspectives reduces their support for geometric relationships between the visual elements that make up a diagram. This makes the psychological approach incapable of accounting for the diversity of diagram types and their meanings and consequently very little support is given to accounting for mixed diagrams. Any meaningful significance of visual features of diagrams is left for the observer to work out and this becomes a problem when considering the wide variety of diagrams. [9] argue that trying to understand diagrams from a psychology and cognition perspective gives very little guidance on how to construct them. This limits the potential for psychological ways of understanding to address the problem of handling change. The same authors criticise these approaches for their limited support for mixed or complex diagrams. [42] points out that most of the psychological studies into the nature of diagrams have focused on difficulty factors and perceptual and cognitive processes involved in diagram comprehension. Zhang proposes that a better approach should consider the relationship between task and display rather than treating them separately as most psychological approaches seem to do. This suggests a limitation of psychological approaches in dealing with and making sense of the problem of the variety of diagrams.

3.3 Evaluation of computational approaches

Computational systems are often developed based on psychological studies with the aim of automating the generation of diagrams, [42]. This aspect of computational approaches makes them less likely to support the contexts of everyday life and of diagramming as part of thinking. Such contexts require the activity of diagramming to involve interpretation and reinterpretation rather than generating a diagram as a final product or output. Often within these contexts diagrams contain instances where rules of well formedness are relaxed to aid the thinking process. Computational approaches tend to apply rules of well formedness in a rigid way thus making them less suitable in supporting such features. [9] also add that existing drawing packages seldom support the activity of marking the significance of a distance between two points. This suggests the limitation of computational approaches in accounting for the problem of variety of diagrams. In general automatic generation of diagrams bypasses most of the problems of variety of diagrams, handling change, and mixed diagrams. This is because it assumes that most of these problems have already been solved and dealt with by the creator of the diagram. Computational approaches place a great deal of significance on the issue of efficiency. This emphasis leads to the generated diagram missing most or some of the meaningful spatial and visual features that were of importance to the diagrammer. This affects the part of thinking context. Finally, computational approaches are driven by and operate on data supplied by an external source. This limits the suitability of these approaches in recognising the problem of variety of diagrams mentioned above. This criticism is justified because data variations are huge and thus they offer no basis for classifying the variety of diagrams they generate.

The analysis of the scientific ways of understanding diagrams shows they do not have the potential to account well for the problems mentioned above.


4. Paradigmatic/philosophical frameworks

These approaches are seen as frameworks of understanding because they attempt to bring together a number of ways of understanding diagrams. [35] argues that integrated approaches will open up broad new horizons for cartographic research. Our own analysis of [5] list of the various approaches to classifying diagrams points towards a shift in the thinking of the research community towards deeper and more meta issues. For long periods research seemed to focus on treating diagrams as cognitive, or perceptual, or graphical things. However, from the 1990s onwards the focus seems to have shifted towards finding ways of bridging the gap between the various frameworks. Three main frameworks for understanding diagrams discussed under this category are:

Semiotics — Peirce

Semiology of Graphics — Bertin

Notational Schemas — Engelhardt


Each of these three frameworks is discussed and evaluated.


4.1 Semiotics — Peirce

Semiotics is the science of signs, with sign considered to be a relationship between an expression and its referent, content or what the expression refers to. Semiotics derives from studies in linguistics, but is broader than spoken or written language and can be seen as an approach to understanding all the ways people communicate with each other. The field of study was initially proposed by the Swiss linguist Ferdinand de Saussure and the American pragmatist Charles Sanders Peirce. The two worked separately but had a common interest aimed at developing a better understanding of how and why certain structures were able to generate and convey meaning. The importance and influence of Peirce’s work for graphic designers, cartographers, and others who have interest in the theoretical foundations of diagrams is acknowledged here and is introduced in the next section. [31] declared that anything, including diagrams, can be a sign as long as someone, in some situation, interprets it as standing for something other than itself. His work on semiotics is well known across many disciplines, especially in the field of studying diagrams. Peirce’s work is based around his model known as “The Semiotic Triad”. Peirce used this model of signs to develop his system of classification of signs based on how they convey meaning.

Peirce’s understanding of signs, including diagrams, is best illustrated by making direct reference to his definition of a sign:

 “A Sign, or Representamen, is a First which stands in such a genuine triadic relation to a Second, called its Object, as to be capable of determining a Third, called its Interpretant, to assume the same triadic relation to its Object in which it stands itself to the same Object”. (Peirce, 1902)

A representamen divides into three parts:

Mark: any sensory data pattern in any modality that someone or something is capable of perceiving. This is the (not necessarily physical) form of the sign. The element is known by Peirce (1932) as Firstness and is described as follows: “Firstness is any possible quality of feeling taken by itself, whether ‘the color of magenta, the odor of attar, the sound of a railway whistle, the taste of quinine, the quality of the emotion upon contemplating a fine mathematical demonstration, the quality of the feeling of love, etc.’”

Token: a classification of some mark as an instance of some type. This is about the sense made of the sign in the mind of the observer (this can be another sign). This element is known by [30] as Secondness and is described as follows: “Secondness presents itself as brute fact, as struggle and opposition, shock, surprise, effort and resistance. It is the hard, uncontrolled givenness which we encounter in experience.”

Type: a general pattern associated with some schema or rule for classifying and relating the marks in the environment or to aspects of experience. This element is about the thing to which the sign refers and is referred to by [30] as Thirdness. It is about binding Firstness and Secondness. In Peirce’s words: “The beginning is first, the end second, the middle third.”


The importance and influence of semiotics, as [35] points out, is in its potential to integrate the different traditional ways of understanding diagram’s such as visual perception and visual cognition.


4.2 Evaluation of Peirce

Peirce’s work delivers a broad brush approach to what constitute the differences between the various signs other than what they refer to. This feature of this approach makes it ill suited to dealing with the problem of the variety of diagram’s types that transcend application constraints. A further implication of Peirce’s approach is that it implies that any meaning of a sign is a property of the human interpreter. Within this context the approach attributes differences between diagrams to human inferences and as such is incapable of dealing with the problem of variety of diagrams. This is because differences or even similarities between diagrams are not purely accounted for by cognitive but there is also the spatial aspect that contributes to these differences and similarities. These limitations extend the inappropriateness of semiotics to dealing with the problem of mixed diagrams. Furthermore, if a sign is vague or contains errors the interpreter is unable to usefully recognise the meaning of the sign. This makes the approach ill suited to dealing with the problem of handling change when rules are relaxed or with the context of everyday life diagramming where diagrams are often vague and ill structured. Under Peirce’s framework such diagrams pose difficulties because the mapping between the Firstness and Secondness is less likely to materialise. The approach is lacking in support for handling changes in diagrams.

4.3 Semiology of Graphics — Bertin

[3], a French cartographer and information designer described one of the first comprehensive theories of graphic variables and their uses. Bertin identified and classified a basic set of “visual variables” (Shape, Size, Colour (hue), Texture, Position, Orientation, and Movement). Bertin considered these visual variables to be the fundamental units of visual communication and to form the basis of all forms of visual coding. His classification of the variables was according to the scale of information that they are most appropriate for encoding (i.e., nominal, ordinal, and quantitative). Table 1 shows a summary of Bertin’s classification. Use of these variables is also limited by the ability of the human eye to see differences. Each variable has a minimum spread and a maximum length. The concept of visual variables, as described by Bertin, has since been modified and expanded by [34]; [12]; [23]. Bertin’s work influenced others, such as [22], who classified diagrams based on structural properties rather than content. They produced 11 categories of visual representations: Graphs, Tables, Graphical tables, Time charts, Networks structure, Diagrams, Maps, Cartograms, Process diagrams, Icons, and Pictures. [35] states that Bertin considered the viewing of these visual variables to be a pre-attentive process, that is, they create a mental image or communicate meaning before any internal image schemata held by the map viewer are brought into play. Table 1 shows Bertin’s original set of variables and their association with quantitative, ordered and nominal aspects [35].


Table 1. Bertin’s original list of visual variables, in [35]      


Level of measurement

Visual variable












Colour (value)








Colour (hue)














4.4 Evaluation of Semiology of Graphics

Although Bertin’s matrix of associating the various visual variables does differentiate between diagrams, nevertheless, it does not give a root reason for the differences or similarities between the various types of diagrams. The matrix proposes how to create effective diagrams as far as human’s perceptual and visual processing is concerned. As far as Bertin’s framework is concerned there are no structural differences between a list, a graph, and a hierarchical diagram. Their differences are all to do with the content of their information. The inappropriateness of Bertin’s framework concerned is further compounded when mixed diagrams are considered. Bertin’s framework has been criticised by others such [8] arguing that [3] account of graphics and other accounts such as that of [22] to produce generative frameworks of visual representations are too general to be useful, particularly in areas such as diagrammatic knowledge acquisition. This critique points to the framework’s inability to deal with the problem of variety of diagrams.



4.5 Notational Schemas — Engelhardt

Engelhardt (2002) argues that most existing approaches to understanding graphics are cover either only specific aspects of graphics or only certain types of graphics. Engelhardt suggests a “Comprehensive and Unifying Framework” which, he argues, has the ability to bring together both extremes.


The framework evolves around three major parts:

Syntax of spatial structure: this is about spatial/visual descriptions of a graphic.

Type of correspondence: this is to do with the type of relationship or correspondence between a graphic and its content.

Type of graphic representation: this is in terms of primary or hybrid schemas, see below.


Other aspects of Engelhardt’s framework that are similarly of interest are the notions of “Primary types of graphics” and “Hybrid types of graphics”. Engelhardt’s framework accounts for differences between visual representations based mainly on two aspects mainly visual vocabulary and visual grammar. The other part of Engelhardt’s framework is the compositional schema. This is broken down into two parts, primary (basic) and hybrid (complex) schemas. The basic compositional schema is defined as “the systematic ways in which several components are combined with each other into a meaningful composition”.


4.6 Evaluation of Notational Schemas

Engelhardt’s framework places a strong emphasis on a spatially/graphically oriented description of diagrams. This is evident in its approach of recognising the arrangement of certain spatial or visual components into meaningful structures dictated by some content. Such an approach will have difficulty in dealing with the problem of the variety of diagram types. We give two reasons for this difficulty. One is that the framework is a descriptive one rather than accounting for purpose or meaning, and as such it is concerned with differences and/or similarities between application oriented diagrams rather than what lies underneath these diagrams. Second, a spatially oriented account will actually compound the difficulty of the problem of the diversity of diagrams rather than easing it. The basis for this claim is that we recognise the richness of the spatial aspect and there is a huge scope for human creativity to manipulate the spatial aspect so as to produce unlimited graphical arrangements to serve a very wide range of applications. The framework also has difficulty in dealing with the problem of mixed diagrams. In such diagrams we often find the same spatial feature expressing different meanings. This puts the framework under considerable difficulty to account for such diagrams. Some of these difficulties are evident in Figure 2.





Fig 2. Fish distribution (Engelhardt 2002)




This diagram is seen under Engelhardt’s framework as a “Primary” type diagram and assigned as a “Grouping” diagram. However, the description does not account for a number of significant features of the diagrams such as:


The significance of the different sizes of the fish.

The similarity of some of the objects (types of fish) in the clusters.

The two spaces on the diagram.

The different decreasing numbers of objects in each cluster.

The significance of the decrease in numbers of objects in the clusters.

Whether the single fish is a cluster or not.


Engelhardt makes no reference to how his framework copes with handling change.

5. Summary of the four paradigmatic / philosophical frameworks

Philosophical realism focuses on some external reality and assumes that we can understand it in terms of its ‘essence’. It usually postulates a single external reality, which assumes overriding importance in research and, to a greater or lesser extent, this gives the research a reductionist flavour. The narrow focus of most of these research communities means that they can address certain problems but cannot integrate them with consideration of other problems. Philosophical nominalism maintains that there are no external types or generalities and that we create (or ‘name’) our own individual types. Diagrammatic research informed by this stance tends to be in the form of surveys to try to find out what types of diagrams people use in practice, ending up with taxonomies without underlying theories. But it would not offer us a solid foundation neither for understanding what it means when people think with diagrams, or for developing software to aid such thinking. All it could offer, at least in its pure forms, would be to come up with a list of what people do and the types of diagram that people use, and to try to implement those piecemeal in software. The problem with the former is that there is no way of separating out either what is contingent upon the specific application or the cultural context from what is essentially diagrammatic. The problem with the latter is that the software produced would be merely a clumsy cobbling together of different diagramming operations, leading to a facility lacking in coherence and in which the user finds it difficult to choose which facility to use at any one time.

6. SySpM: A proposed framework for understanding diagrams

Full discussion of this framework and its contributions is beyond the scope of this paper and can be found in [9]. Here we briefly introduce the most important elements of the framework. The central thrust of this framework is: Separating Symbolic from Spatial but allowing for their Mapping. The framework is based on the notion that Sy is distinct and separate from Sp and is irreducible to it. From this framework individual SySpM’s could be constructed. The term “a SySpM” is used to denote a distinct (particular) collection of Sy and a distinct collection of Sp and a distinct Mapping between the two Sy and Sp collection. Synonymous with this term are drawing styles or types of diagrams. We grouped the Sy and Sp terms into primary and secondary lists. The former includes Sy and Sp terms that are used in the simplest or basic form of the SySpM. This category of Sy and Sp is also in all instances of diagrams of the relevant SySpM. The latter includes terms that are used in complex diagrams of the relevant SySpM and it also includes things that have to do with the relationships or interaction between the various Sy things of the relevant SySpM. Development of each SySpM also includes list of Sp things that do not map onto any Sy known as redundant Sp things. These could then become available for other SySpM’s to use to give mixed diagrams, list of constraints and why they occur, and list of events or changes relevant to the SySpM under consideration. Special features of a SySpM use redundant Sp features to bring in secondary Sp features rather than another SySpM as mixed diagrams. An example of this in a box and arrows SySpM is when lines are allowed to cross other lines. This happens when one line is given a kink or a gap to indicate clearly that one is passing over or under the other rather than connecting to it, e.g. electronic circuits. Sub types are diagrams in which the original and simple SySpM is constrained/ complicated because of specific needs usually associated with a type of application. This is achieved by brining in an extra Sy constraint which, owing to Mapping, also gives a different Sp feel to the diagram. There are at least three different ways of Sy sub types in a Box and Arrows SySpM: Networks, Lists, and Trees. For each SySpM there might be special cases that do not 'fit' well. Many spatial applications involve several of such special cases such as holes, discontinuities, and other irregularities. We need to identify these and explain the problems they give, that is, what constraints they break, either spatial (as is the case here) or symbolic.


A SySpM could contain features that are outside the range of its base symbolic types. This recognises that each SySpM will be able to express only a subset of the symbol level, not all of it. To express the whole wide range of things at the Sy level requires several different SySpM’s. Within this context two types of mixedness are identified. One is when several SySpM’s are present in a diagram but none dominates the overall meaning of the diagram. An example of this is the Napoleon’s march on Moscow diagram. This type of mixedness is referred to “True mixed diagrams”. The other type is referred to as “Augmented diagrams”. This type of mixedness has one SySpM occupying a primary importance whilst other SySpM’s are added in and have secondary importance. This type of mixedness occurs when redundant Sp features of a SySpM are used to bring in Sy from other SySpM’s. For example, in a Box and Arrows SySpM thickness of lines could be used to bring in quantitative value from Bar Chart SySpM.


In [9] eight SySpM’s were developed namely:


Boxes and Arrows

Communicating Similarity

Map of Objects

Set Membership

Bar Charts

Route Maps

Contour Maps

Surface Coverage


These SySpM’s are not seen as a final list but rather to demonstrate the potential of SySpM as a framework in accounting for a diverse range of types of diagrams.

7. Relationship to other frameworks of understanding diagrams

In this section a review of how the SySpM framework for understanding diagrams relates to these three extant frameworks is considered. The discussion of the frameworks will be in terms of the parallels and differences they have with the SySpM framework. Two tables are shown to present the content of the contrasts. One table lists the parallels whilst the other lists the differences.


7.1 Semiotics



The Triadic System


Type: Combination of Mark and Token

The general concept of mixed SySpM and sub types

Token: instance of a type

A meta approach specific to understanding diagram that conveys symbolic meaning

A meta approach for understanding Diagrams, Photographs, Paintings, Images

Diagrams are seen in a human context

Places a central emphasis on the role of the human as an interpreter





The Triadic System

To do with diagrams in themselves, but within human context

To do with human-human communication. Nothing is a sign unless it is interpreted as a sign declares Peirce (1931)

Starts with Symbolic aspect

Starts with Sensory

Makes clear distinction between diagrams and arts or photographs. The distinction is based on the notion of the symbolic aspect being the qualifying aspect of diagrams rather than aesthetic aspects as in art

Conflates all forms of signs under one framework defined by the triadic system

All three aspects of Symbolic, Spatial and Mapping are equally meaningful, important and inseparable

Tokens by themselves, apart from an interpreter, have no meaning

All three aspects of Symbolic, Spatial and Mapping are equally meaningful, important and inseparable

Meaning of a sign is divided unequally between the three parts of the triadic system of signs: “Firstness and Thirdness can be more or less vague; Secondness alone is always precise.”

Purposeful and deliberate definitions and mappings in each SySpM.

Arbitrariness or convention governs the relationship between a sign and its object: “A diagram is a (sign) which is predominantly an icon of relations and is aided to be so by conventions. Indices are also more or less used” (1976)

Each SySpM is defined and worked out in detail and with precise and distinct mappings taking into account not the final product as such but rather how it has evolved to become a diagram

Takes a broad view of what a sign is, signs are seen as a final product

Offers a richer and more elegant framework of what constitutes a mixed diagram based on the concept of redundant Sp features.

Offers a weak framework for understanding mixed diagrams in terms of collections of signs

Handling change is included

Offers nothing or very little in the way of handling change or dynamics of diagrams

Diagrams are seen as part of human functioning rather than as given constructs

Diagrams (signs) are seen as given

SySpM is considered to be an intuitive framework for understanding diagrams

Understanding diagrams and signs in general relies on prior human experiences and knowledge

Allows for rule relaxation and hence facilitates fluidity of interpretation or construction of diagrams and aids creativity

If a sign (according to Peirce’s definition) is vague then it may inhibit users thinking unless further signs or additional information become available

Offers a framework for developing diagramming software

Offers no direction for how diagram software could be developed




The contrast, as shown by the tables, shows certain parallels but a greater number of differences. The reason for this is built in the differences between the philosophical dispositions. The framework of SySpM delivers an understanding of diagrams that is based on the notion of Aspects and their mappings. This is a higher level of understanding than what semiotics is concerned with. Whilst Peirce’s theory of signs struggles with interpreting ambiguous or not well known visual signs, the framework of SySpM on the other hand provides the clarity needed. Under Peirce’s framework such signs or diagrams pose difficulties because the mapping between the Firstness and Secondness is less likely to materialise. SySpM provides intuitive mappings between Sy and Sp in distinct ways across the SySpM’s developed. This enables clarity and precision of distinct ways of interpretation.


7.2 Semiology of Graphics



Visual Variables

Meta framework

Attempts to provide a meta framework for understanding diagrams

Aspects are irreducible and inseparable to each other

Separates (tacitly) spatial/visual things from each other

Distinct and deliberate mappings between collections of Sy and Sp things

General association, based on human visual processing capabilities, between the visual and spatial variables and quantitative, nominal, and ordered features





Visual Variables

Explicit recognition of the richness of spatial and symbolic aspects

Limited recognition of spatial and symbolic things

Mappings between Symbolic (Sy) and Spatial (Sp) aspects is intuitive

Mapping between spatial features and quantitative is driven by the ability of human visual processing capabilities

Both Sy and Sp are equally important and irreducible to each other or any other aspect

Elevates visual and graphical features (Blackwell and Engelhardt, 1998)

Separates Sy from Sp, see section on the choice of terminologies for both aspects

Conflates sensory, symbolic, and spatial phenomena

Flexible by making distinct differentiation between the various SySpM’s and other features such as rule relaxation

Rigid in its approach: “Only implantation by point can manifest absolute quantities in x” (Bertin, 1981)

Recognises and promotes the richness of the spatial aspect

Constrains the richness of the spatial aspect of the world around us to a limited number of visual variables

Handling change is supported

Static description of diagrams

Offers a framework for developing diagramming software

Offers no direction for how diagram software could be developed



In contrast with semiotics, the work of Bertin seems to provide a closer contrast with the framework of SySpM, hence the small number of differences. This could be explained by pointing out the emphasis of semiology of graphics on diagrams that communicate meaning rather than on all forms of visual representations. However, there still remain fundamental differences between the two frameworks, particularly when semiology elevates certain chosen spatial and sensory features. Bertin’s matrix of associating the various visual variables is intended to produce better graphics rather than to describe differences and/or similarities between various diagrams. With this approach in mind Bertin’s framework finds it difficult to make distinctions between a list, a graph, and a hierarchical diagram. This indifference is a source of the problem of the variety of diagram types. However, under SySpM precise distinction are formed between all types of diagrams, based on their unique Sy, Sp and M. SySpM provides a basis for extending and enriching Bertin’s framework by considering the richness of Sy and how this may be expressed through distinct Sp and M.



7.3 Notational Schemas



Notational Schemas

Spatial things

Component schemas


Compositional schemas


Primary types

Mixed SySpM’s

Hybrid types





Notational Schemas

Symbolic and Spatial are both equally important and needed

Strong emphasis on spatial constraints

Trans application

Application oriented

Diagrams that convey symbolic meaning

Includes pictures that do not convey symbolic meaning

Each SySpM is distinctly different from other SySpM’s

Primary types in themselves contain mixed symbolic meanings: a Map has at least Routes and Objects.

Irreducibility and equal importance of the symbolic, spatial and their mappings.

Strong emphasis on the visual/spatial

Distinction between spatial components that have Sy meaning and those that do not have Sy meaning (redundant)

Components do not make a distinction between meaningful and meaningless (redundant) components

Handling change is supported

Static description of diagrams

Offers a framework for developing diagramming software

Offers no direction for how diagram software could be developed


The issues and the overall approach of notational schemas seem to extend and enrich those discussed in Richards (2000). The framework is spatially oriented and this makes it differ from the framework of SySpM.


8. Summary

Although there are certainly parallels between the framework of SySpM and the three extant frameworks, nevertheless, there are fundamental differences between them. The differences are centred around what is seen as by each framework as important to our understanding of diagrams. The SySpM framework distinguishes itself from the rest of the other frameworks by explicitly acknowledging the notion of aspects namely the Symbolic and Spatial aspects, and their mappings as a good way of understanding diagrams. The framework enabled the development of a number of notions which helped to deal with these problems, namely:


The notion of Aspects and their irreducibility

The concept of the basic SySpM

The concept of more complex versions of A SySpM

Sub types

Primary and secondary Sy and Sp things

Redundant Sp features

Mixed and augmented diagrams

Relaxation of rules

Handling change


Potential of extending the framework to include other aspects such as sensory (colour) and dynamics of diagramming.


All of these notions give a measure of the goodness of the framework, particularly in helping to deal with the problems of variety of diagram types, mixed diagrams, and handling change.


9. The SySpM framework and the three problems

The SySpM framework deals with the problem of the variety of diagram types at two levels. One is at the level of individual SySpM’s. The other level is within each individual SySpM where a wide range of different diagrams can be accounted for through the notions of sub types, special features, true and augmented diagrams. The deliberate, distinct, and rich account of what constitutes Sy, Sp, and M for each SySpM allows not only the potential to accounting for a very wide range of diagrams but also in differentiating them with precision.


The problem of mixed diagrams is dealt with through the notions of augmented and true mixed diagrams. [14] demonstrated the potential of the eight SySpM’s in accounting for four complex and semantically mixed real world diagrams:


A Quantified Flow Chart

A Pressure – Volume Graph

Diagram depicting Napoleon’s March on Moscow

An Ordnance Survey Map


Change in diagrams is give precision and flexibility at the same time. Precision because each editing operation is specific to the type of SySpM under consideration and distinguished from other SySpM’s. Flexibility is accommodated by the notion of rule relaxation.


10. Conclusion

This paper responded to renewed interest in our understanding of diagrams. Whilst existing definitions are based on a specific perspective the framework proposed in this paper pursues a much richer approach. The proposed framework is contrasted with other main approaches of understanding diagrams. The merits of the framework are discussed in relation to three main problems concerning diagrams. The work carried introduced in this paper is part of an ongoing research which aims to develop drawing software based on the framework of SySpM capable of handling the challenges presented by the three main problems.



9. References


  1. Anderson M and McCartney R (2003) Diagram processing: Computing with diagrams. Artificial Intelligence 145, pp 181- 226. ELSEVIER.


  2. Basden A, Brown A J, Tetlow S D A, Hibberd P R (1996) Design of a user interface for a knowledge refinement. tool International Journal of Human Computer Studies, vol.45, pp.157-183.


  3. Bertin J, (1983), “Semiology of graphics”. University of Wisconsin press.


  4. Bishop I (1994) In: Visualisation in modern cartography, Ed. A M MacEachren and D R Fraser Taylor. Pergamon 1994.


  5. Blackwell A F and Engelhardt Y (1999) A Meta-Taxonomy for Diagram Research. In P. Olivier, M. Anderson & B. Meyer (Eds.), Diagrammatic Representation and Reasoning, Springer-Verlag.


  6. Carswell C M and Wickens C D (1988) Comparative graphics: History and applications of perceptual integrality theory and the proximity compatibility hypothesis. Technical Report, Institute of Aviation, University of Illinois at Urbana- Champaign.


  7. Casner S M and Larkin J H (1989) Cognitive efficiency considerations for good graphic design. Proceedings of the Eleventh Annual Conference of the Cognitive Science Society, Ann Arbor Michigan.


  8. Cheng P (1996) Diagrammatic Knowledge Acquisition: Elicitation, Analysis and Issues. In N. R. Shadbolt, H. O'Hara, & G. Schreiber (Eds.), Advances in Knowledge Acquisition: 9th European Knowledge Acquisition Workshop, EKAW'96. (pp. 179-194). Berlin: Springer-Verlag.


  9. Cheng P, Cupit J, (2001), Supporting diagrammatic knowledge acquisition: an ontological analysis of Cartesian graphs. Int. J. Human – Computer – Interaction Studies, 54.


  10. Cheng P, Lowe R, Scaife M (2001) Cognitive Science Approaches To Understanding Diagrammatic Representations. In Thinking with Diagrams, ed. Alan F. Blackwell, Kluwer Academic Publishers, Dordrecht


  11. Cleveland W S (1985) The Elements Of Graphic Data. Wadsworth Publishing.


  12. Cleveland, W.S. and McGill, R. (1986) An experiment in graphical perception. Int. J. Man-Machine Studies 25, 491-500.


  13. Engelhardt Yuri (2002), The Language of Graphics, PhD thesis, 2002, Institute for Logic, Language & Computation, University of Amsterdam. ISBN 90-5776-089-4


  14. Fathulla K (2006), Understanding Diagrams Based on Symbolic and Spatial Mapping. PhD thesis, University of Salford, United Kingdom.


  15. Garland K (1979) Some general characteristics present in diagrams denoting activity, event and relationship. Information Design Journal, 1(1), 15-22.


  16. Gibson J J (1977) The theory of affordances. In: Shaw R, Bransford J (eds): Perceiving, Acting and Knowing Erlbaum, Hilsdale, NJ.


  17. Gurr C (2000) Tutorial T1 - Diagrammatic Reasoning and Visual Languages, IEEE Symposium on Visual Languages, Seattle, Washington, September 10-14, 2000


  18. Horn R E, (2001) Visual Language and Converging Technologies in the Next 10-15 Years (and Beyond). The National Science Foundation Conference on Converging Technologies (Nano-Bio-Info-Cogno) for Improving Human Performance.


  19. Jobson J, Rahman Z, and Woodell G A (2001) The Spatial Aspect of Color and Scientific Implications of Retinex Image Processing, SPIE International Symposium on AeroSense, Proceedings of the Conference on Visual Information Processing X.


  20. Kim J, Hahn J and Hahn H (2000) How Can We Understand a System with (so) Many Diagrams? Cognitive Integration Processes in Diagrammatic Reasoning, Information Systems Research, Vol. 11, No. 3, 284-303.


  21. Larkin J H and Simon H A (1987), "Why a diagram is (sometimes) worth a ten thousand words. Cognitive Science, 11, pp.65-99.


  22. Lohse G L, et al,(1994), “A classification of visual information”, Communications of the ACM, Vol. 37, No. 12.


  23. Mackinlay J D (1998) Automating the design of graphical presentation of relational information. In: Maybury, M T, Wahlster, W (eds): Readings in intelligent user interfaces.


  24. Markowitz V and Shoshani A (1998), On the Correctness of Representing Extended Entity-Relationship Structures in the Relational Model. SIGMOD Conference 1989: 430-439.


  25. Neilson I, Lee J (1994) Conversation with graphics: implications for the design of natural language/graphics interfaces. Int J Human-Computer Studies, vol.40 pp.509-541.


  26. Osborn J R (2005) Theory Pictures as Trails: Diagrams and the Navigation of Theoretical Narratives. Cognitive Science Online, Vol.3.2, pp. 14-44.


  27. Peirce C S (written around 1897, republished in 1932). Elements of Logic. In C. Hartshorne and P. Weiss (Eds.), The collected papers of C.S. Peirce. Harvard University Press.


  28. Peirce, C S (1902) Logic, Regarded As Semeiotic. (The Carnegie application of 1902)


  29. Peirce, C S (1931) Collected Writings (8 Vols.). (Ed. Charles Hartshorne, Paul Weiss & Arthur W Burks). Cambridge, MA: Harvard University Press


  30. Peirce, C S (1931-36) In "Some Consequences of Four Incapacities," Collected Papers of Charles Sanders Peirce, 8 vols., ed. Charles Hartshorne and Paul Weiss (vol. 1-6), Arthur W. Burks (vols. 7-8), Cambridge: Harvard University Press.


  31. Peirce, C S (1976) (NEM) New Elements of Mathematics, ed. by Carolyn Eisele, 4 vols., Mouton, The Hague, 1976.


  32. Pinker R S (1990), A theory of graph comprehension. In R Freedle, Ed. Artificial Intelligence and Future of Testing. Hillsdale, NJ: Lawrence Erlbaum Associates.


  33. Rowe P (1987) Design Thinking. Cambridge: MIT Press.


  34. Shneiderman, B, Card, S.K, and Mackinlay, J.D (1999) Readings in information visualization : using vision to think (Morgan Kaufmann series in interactive)


  35. Sluter R S (2001) NEW THEORETICAL RESEARCH TRENDS IN CARTOGRAPHY. Brazilian Journal of Cartography, a publication of SBC - Brazilian Society of Cartography, Geodesy, Photogrammetry and Remote Sensing.


  36. Sowa J F (1984) Conceptual Structures: Information Processing in Mind and Machine, Addison-Wesley, Reading, MA.


  37. Taylor M M, McCann M M, Tuori M I (1984) The Interactive Spatial Information System. DCIEM Report No 84-R-22 Defence and Civil Institute of Environmental Medicine, Ontario.


  38. Tufte E R (1983) The Visual Display of Quantitative Information. Graphics Press.


  39. Tversky B (2001) Spatial Schemas in Depictions. In Spatial Schemas and Abstract Thought. Gattis, M. (ed.), Cambridge, MA:MIT Press.


  40. Winograd T (1995) From programming environments to environments for designing. Communications of the ACM, vol.38 pp.65-74.


  41. Wood D (1993) The power of maps. Routledge, London.


  42. Zhang J 1(1996) A Representational Analysis of Relational Information Displays. International Journal of Human- Computer Studies.


E-mail the editors
Pour écrire à la rédaction

© 2007, Applied Semiotics / Sémiotique appliquée