Frameworks for understanding diagrams: Enriching the debate
Kamaran FATHULLA, Andrew BASDEN
Abstract
This paper responds to renewed interest in the centuries old question of what is a diagram.
Existing status of our understanding of diagrams is seen as unsatisfactory and confusing
particularly when responding to problems of accounting for the variety of diagrams, handling
change in diagrams in a well formed way, and all of this in the context of semantically mixed
diagrams. Existing frameworks (Peirce, Bertin, Engelhardt) for understanding diagrams are
discussed and evaluated against these problems. These frameworks are then contrasted for
similarities and differences with our proposal which is based on, SySpM, Symbolic and Spatial
Mapping. Potential and contributions of the proposed framework are briefly mentioned.
Keywords: SySpM: Symbolic, Spatial, Mapping. WF: Well-formedness.
1. Introduction
Diagrams are a very old form of representing and making sense of the world around us. Some
2500 years ago people have used clay tablets to express boundaries, groupings, and routes,
Figure 1.
Fig 1: Clay Tablet map from Ga-Sur, Kirkuk, 2,500 B.C.
However, despite this close and rich involvement of diagrams in human history people are now
again still asking centuries old question “What is a diagram?”, [26]. [39] asks the question
“What are arrows?”. [1] ask “What constitutes a diagram?”. [4] adds to these calls by
questioning our existing understanding of diagrams arguing that centuries held assumption that
“a drawing-is a drawing-is a drawing” is progressively shown to be invalid.
The reemergence of questions of understanding diagrams does not come as a surprise especially
when we consider the complexity of the field under investigation. [14] has identified three main
problems that need good answers to advance our understanding of diagrams. We list them here
but a full discussion found in the given reference.
Variety of diagram types
Handling change in diagrams and maintaining their well formedness
Semantically mixed diagrams
2. Existing approaches to understanding diagrams
Existing ways of understanding diagrams is divided mainly into two broad camps. One is the
scientific ways and the other are the paradigmatic/philosophical ways. Both ways are introduced
and evaluated against the problems.
3. The scientific approaches
[17] summarises in broad terms the basic approaches currently adopted for understanding of
diagrams. These are:
Theoretical approaches: the nature of diagrams - examined by art, philosophy, mathematics and
graphic design; the syntax and semantics of diagrams; formal properties of diagrammatic
notations, formal theories of human communication and human-computer interaction through
diagrams; the computational tractability of diagrams.
Psychological approaches: visual perception; diagram comprehension and inference; the
interplay between diagrammatic representations and human reasoning (including individual
cognitive differences); generation and manipulation of diagrams for inference purposes;
diagrams in education; the cognitive effectiveness of diagrams.
Computational approaches: diagram interpretation and parsing; visual language grammars;
visual program execution; the generation and manipulation of diagrams for displaying the results
of executions; data visualization; diagram animation; diagrammatic interfaces; intelligent
diagrammatic reasoning systems.
3.1 Evaluation of theoretical approaches
Humans construct theories in order to explain, predict and master some aspect of the natural
world. They do this by abstracting the world they theorise about into a certain perspective which
has then to be tested and proven. Theoretical abstraction tends to reject competing theories as
either incorrect or having sharp differences with each other. As a consequence what is seen as
valuable or interesting by some is found irrelevant and ignored by others. Theoretical approaches
also inclined towards producing formal accounts of diagrams. This makes such an approach less
suitable for addressing the contexts of informal, everyday diagramming, or when diagramming is
part of thinking, because all of the contexts evolve around several different types of diagrams
and require a certain level of rule relaxation. Furthermore, such diagrams are often drawn for
different purposes and each purpose is guided by a particular perspective. Theoretical
approaches tend to reject other perspectives. [5] argue that an interdisciplinary field such as
diagrams cannot afford to concentrate only on formal analyses as a way of accounting for the
nature of diagrams. They point out how existing accounts of understanding diagrams have
ignored issues of interpretation because they are less easily formalised. Purely theoretical
approaches not only create problems for accounting for the diversity of diagrams and how
software is to be developed to support this diversity, but also limit the scope of research into
diagrams. [9] argue that researchers tend to focus their attention on specific types of diagrams
which are supported by certain well known and established formalisms and ignore other more
useful types of diagrams which are less formalised. This feature of theoretical approaches limits
their suitability for addressing the variety of diagrams problem. [5] also show in their survey of
the different ways of accounting for the nature of diagrams a lack of approaches that are
concerned with performance of diagrams. They attribute this limitation to the fact that
performance is less easily formalised. This feature of theoretical ways limits their capability to
account for well formedness, which forms part of the problem of handling change.
3.2 Evaluation of psychological approaches
The focus of this approach is on the psychological processes that occur when people see and
interpret diagrams. [11] and [12] studied the comparisons of quantitative information and the
relative goodness of a set of physical dimensions for ratio judgements. Based on the
psychological studies of integral and separable dimensions, [6], proposed the proximity
compatibility hypothesis to account for a variety of graphics problems, especially those that
involve information integration. Another approach has focused on the perceptual and cognitive
processes involved in graph and diagram comprehension, [21]; [7]; [32]. In these studies the
perceptual and cognitive processes involved in graph comprehension are analysed and modeled
by computational systems such as production systems and semantic networks. On the basis of
psychological studies, several researchers have developed computer systems that can automate
the design of graphics, [23]. Psychological accounts are directly linked and constructed from
stimuli received from our senses. [15] defines a diagram as a “visual language sign”. [34]
drawing on the theory of Percepts and Concepts, sees diagrams in terms of sensory data
perceived and conceptual relations (interpretations) are formed: “a conceptual graphs describes
the way percepts are assembled”. The emphasis of these approaches on perceptual or visual
perspectives reduces their support for geometric relationships between the visual elements that
make up a diagram. This makes the psychological approach incapable of accounting for the
diversity of diagram types and their meanings and consequently very little support is given to
accounting for mixed diagrams. Any meaningful significance of visual features of diagrams is
left for the observer to work out and this becomes a problem when considering the wide variety
of diagrams. [9] argue that trying to understand diagrams from a psychology and cognition
perspective gives very little guidance on how to construct them. This limits the potential for
psychological ways of understanding to address the problem of handling change. The same
authors criticise these approaches for their limited support for mixed or complex diagrams. [42]
points out that most of the psychological studies into the nature of diagrams have focused on
difficulty factors and perceptual and cognitive processes involved in diagram comprehension.
Zhang proposes that a better approach should consider the relationship between task and display
rather than treating them separately as most psychological approaches seem to do. This suggests
a limitation of psychological approaches in dealing with and making sense of the problem of the
variety of diagrams.
3.3 Evaluation of computational approaches
Computational systems are often developed based on psychological studies with the aim of
automating the generation of diagrams, [42]. This aspect of computational approaches makes
them less likely to support the contexts of everyday life and of diagramming as part of thinking.
Such contexts require the activity of diagramming to involve interpretation and reinterpretation
rather than generating a diagram as a final product or output. Often within these contexts
diagrams contain instances where rules of well formedness are relaxed to aid the thinking
process. Computational approaches tend to apply rules of well formedness in a rigid way thus
making them less suitable in supporting such features. [9] also add that existing drawing
packages seldom support the activity of marking the significance of a distance between two
points. This suggests the limitation of computational approaches in accounting for the problem
of variety of diagrams. In general automatic generation of diagrams bypasses most of the
problems of variety of diagrams, handling change, and mixed diagrams. This is because it
assumes that most of these problems have already been solved and dealt with by the creator of
the diagram. Computational approaches place a great deal of significance on the issue of
efficiency. This emphasis leads to the generated diagram missing most or some of the
meaningful spatial and visual features that were of importance to the diagrammer. This affects
the part of thinking context. Finally, computational approaches are driven by and operate on data
supplied by an external source. This limits the suitability of these approaches in recognising the
problem of variety of diagrams mentioned above. This criticism is justified because data
variations are huge and thus they offer no basis for classifying the variety of diagrams they
generate.
The analysis of the scientific ways of understanding diagrams shows they do not have the
potential to account well for the problems mentioned above.
4. Paradigmatic/philosophical frameworks
These approaches are seen as frameworks of understanding because they attempt to bring
together a number of ways of understanding diagrams. [35] argues that integrated approaches
will open up broad new horizons for cartographic research. Our own analysis of [5] list of the
various approaches to classifying diagrams points towards a shift in the thinking of the research
community towards deeper and more meta issues. For long periods research seemed to focus on
treating diagrams as cognitive, or perceptual, or graphical things. However, from the 1990s
onwards the focus seems to have shifted towards finding ways of bridging the gap between the
various frameworks. Three main frameworks for understanding diagrams discussed under this
category are:
Semiotics — Peirce
Semiology of Graphics — Bertin
Notational Schemas — Engelhardt
Each of these three frameworks is discussed and evaluated.
4.1 Semiotics — Peirce
Semiotics is the science of signs, with sign considered to be a relationship between an expression
and its referent, content or what the expression refers to. Semiotics derives from studies in
linguistics, but is broader than spoken or written language and can be seen as an approach to
understanding all the ways people communicate with each other. The field of study was initially
proposed by the Swiss linguist Ferdinand de Saussure and the American pragmatist Charles
Sanders Peirce. The two worked separately but had a common interest aimed at developing a
better understanding of how and why certain structures were able to generate and convey
meaning. The importance and influence of Peirce’s work for graphic designers, cartographers,
and others who have interest in the theoretical foundations of diagrams is acknowledged here
and is introduced in the next section. [31] declared that anything, including diagrams, can be a
sign as long as someone, in some situation, interprets it as standing for something other than
itself. His work on semiotics is well known across many disciplines, especially in the field of
studying diagrams. Peirce’s work is based around his model known as “The Semiotic Triad”.
Peirce used this model of signs to develop his system of classification of signs based on how
they convey meaning.
Peirce’s understanding of signs, including diagrams, is best illustrated by making direct
reference to his definition of a sign:
“A Sign, or Representamen, is a First which stands in such a genuine triadic
relation to a Second, called its Object, as to be capable of determining a
Third, called its Interpretant, to assume the same triadic relation to its Object
in which it stands itself to the same Object”. (Peirce, 1902)
A representamen divides into three parts:
Mark: any sensory data pattern in any modality that someone or something is capable of
perceiving. This is the (not necessarily physical) form of the sign. The element is known
by Peirce (1932) as Firstness and is described as follows: “Firstness is any possible
quality of feeling taken by itself, whether ‘the color of magenta, the odor of attar, the
sound of a railway whistle, the taste of quinine, the quality of the emotion upon
contemplating a fine mathematical demonstration, the quality of the feeling of love, etc.’”
Token: a classification of some mark as an instance of some type. This is about the sense
made of the sign in the mind of the observer (this can be another sign). This element is
known by [30] as Secondness and is described as follows: “Secondness presents itself as
brute fact, as struggle and opposition, shock, surprise, effort and resistance. It is the hard,
uncontrolled givenness which we encounter in experience.”
Type: a general pattern associated with some schema or rule for classifying and relating
the marks in the environment or to aspects of experience. This element is about the thing
to which the sign refers and is referred to by [30] as Thirdness. It is about binding
Firstness and Secondness. In Peirce’s words: “The beginning is first, the end second, the
middle third.”
The importance and influence of semiotics, as [35] points out, is in its potential to
integrate the different traditional ways of understanding diagram’s such as visual
perception and visual cognition.
4.2 Evaluation of Peirce
Peirce’s work delivers a broad brush approach to what constitute the differences between the
various signs other than what they refer to. This feature of this approach makes it ill suited to
dealing with the problem of the variety of diagram’s types that transcend application constraints.
A further implication of Peirce’s approach is that it implies that any meaning of a sign is a
property of the human interpreter. Within this context the approach attributes differences
between diagrams to human inferences and as such is incapable of dealing with the problem of
variety of diagrams. This is because differences or even similarities between diagrams are not
purely accounted for by cognitive but there is also the spatial aspect that contributes to these
differences and similarities. These limitations extend the inappropriateness of semiotics to
dealing with the problem of mixed diagrams. Furthermore, if a sign is vague or contains errors
the interpreter is unable to usefully recognise the meaning of the sign. This makes the approach
ill suited to dealing with the problem of handling change when rules are relaxed or with the
context of everyday life diagramming where diagrams are often vague and ill structured. Under
Peirce’s framework such diagrams pose difficulties because the mapping between the Firstness
and Secondness is less likely to materialise. The approach is lacking in support for handling
changes in diagrams.
4.3 Semiology of Graphics — Bertin
[3], a French cartographer and information designer described one of the first comprehensive
theories of graphic variables and their uses. Bertin identified and classified a basic set of “visual
variables” (Shape, Size, Colour (hue), Texture, Position, Orientation, and Movement). Bertin
considered these visual variables to be the fundamental units of visual communication and to
form the basis of all forms of visual coding. His classification of the variables was according to
the scale of information that they are most appropriate for encoding (i.e., nominal, ordinal, and
quantitative). Table 1 shows a summary of Bertin’s classification. Use of these variables is also
limited by the ability of the human eye to see differences. Each variable has a minimum spread
and a maximum length. The concept of visual variables, as described by Bertin, has since been
modified and expanded by [34]; [12]; [23]. Bertin’s work influenced others, such as [22], who
classified diagrams based on structural properties rather than content. They produced 11
categories of visual representations: Graphs, Tables, Graphical tables, Time charts, Networks
structure, Diagrams, Maps, Cartograms, Process diagrams, Icons, and Pictures. [35] states that
Bertin considered the viewing of these visual variables to be a pre-attentive process, that is, they
create a mental image or communicate meaning before any internal image schemata held by the
map viewer are brought into play. Table 1 shows Bertin’s original set of variables and their
association with quantitative, ordered and nominal aspects [35].
Table 1. Bertin’s original list of visual variables, in [35]
|
Level of measurement
|
|
Numerical
|
Ordinal
|
Nominal
|
|
X
|
X
|
X
|
|
X
|
X
|
|
|
|
X
|
|
|
|
X
|
|
|
|
|
X
|
|
|
X
|
|
|
|
|
X
|
4.4 Evaluation of Semiology of Graphics
Although Bertin’s matrix of associating the various visual variables does differentiate between
diagrams, nevertheless, it does not give a root reason for the differences or similarities between
the various types of diagrams. The matrix proposes how to create effective diagrams as far as
human’s perceptual and visual processing is concerned. As far as Bertin’s framework is
concerned there are no structural differences between a list, a graph, and a hierarchical diagram.
Their differences are all to do with the content of their information. The inappropriateness of
Bertin’s framework concerned is further compounded when mixed diagrams are considered.
Bertin’s framework has been criticised by others such [8] arguing that [3] account of graphics
and other accounts such as that of [22] to produce generative frameworks of visual
representations are too general to be useful, particularly in areas such as diagrammatic
knowledge acquisition. This critique points to the framework’s inability to deal with the problem
of variety of diagrams.
4.5 Notational Schemas — Engelhardt
Engelhardt (2002) argues that most existing approaches to understanding graphics are cover
either only specific aspects of graphics or only certain types of graphics. Engelhardt suggests a
“Comprehensive and Unifying Framework” which, he argues, has the ability to bring together
both extremes.
The framework evolves around three major parts:
Syntax of spatial structure: this is about spatial/visual descriptions of a graphic.
Type of correspondence: this is to do with the type of relationship or correspondence
between a graphic and its content.
Type of graphic representation: this is in terms of primary or hybrid schemas, see below.
Other aspects of Engelhardt’s framework that are similarly of interest are the notions of
“Primary types of graphics” and “Hybrid types of graphics”. Engelhardt’s framework accounts
for differences between visual representations based mainly on two aspects mainly visual
vocabulary and visual grammar. The other part of Engelhardt’s framework is the compositional
schema. This is broken down into two parts, primary (basic) and hybrid (complex) schemas. The
basic compositional schema is defined as “the systematic ways in which several components are
combined with each other into a meaningful composition”.
4.6 Evaluation of Notational Schemas
Engelhardt’s framework places a strong emphasis on a spatially/graphically oriented description
of diagrams. This is evident in its approach of recognising the arrangement of certain spatial or
visual components into meaningful structures dictated by some content. Such an approach will
have difficulty in dealing with the problem of the variety of diagram types. We give two reasons
for this difficulty. One is that the framework is a descriptive one rather than accounting for
purpose or meaning, and as such it is concerned with differences and/or similarities between
application oriented diagrams rather than what lies underneath these diagrams. Second, a
spatially oriented account will actually compound the difficulty of the problem of the diversity
of diagrams rather than easing it. The basis for this claim is that we recognise the richness of the
spatial aspect and there is a huge scope for human creativity to manipulate the spatial aspect so
as to produce unlimited graphical arrangements to serve a very wide range of applications. The
framework also has difficulty in dealing with the problem of mixed diagrams. In such diagrams
we often find the same spatial feature expressing different meanings. This puts the framework
under considerable difficulty to account for such diagrams. Some of these difficulties are evident
in Figure 2.

Fig 2. Fish distribution (Engelhardt 2002)
This diagram is seen under Engelhardt’s framework as a “Primary” type diagram and assigned as
a “Grouping” diagram. However, the description does not account for a number of significant
features of the diagrams such as:
The significance of the different sizes of the fish.
The similarity of some of the objects (types of fish) in the clusters.
The two spaces on the diagram.
The different decreasing numbers of objects in each cluster.
The significance of the decrease in numbers of objects in the clusters.
Whether the single fish is a cluster or not.
Engelhardt makes no reference to how his framework copes with handling change.
5. Summary of the four paradigmatic / philosophical frameworks
Philosophical realism focuses on some external reality and assumes that we can understand it in
terms of its ‘essence’. It usually postulates a single external reality, which assumes overriding
importance in research and, to a greater or lesser extent, this gives the research a reductionist
flavour. The narrow focus of most of these research communities means that they can address
certain problems but cannot integrate them with consideration of other problems. Philosophical
nominalism maintains that there are no external types or generalities and that we create (or
‘name’) our own individual types. Diagrammatic research informed by this stance tends to be in
the form of surveys to try to find out what types of diagrams people use in practice, ending up
with taxonomies without underlying theories. But it would not offer us a solid foundation neither
for understanding what it means when people think with diagrams, or for developing software to
aid such thinking. All it could offer, at least in its pure forms, would be to come up with a list of
what people do and the types of diagram that people use, and to try to implement those
piecemeal in software. The problem with the former is that there is no way of separating out
either what is contingent upon the specific application or the cultural context from what is
essentially diagrammatic. The problem with the latter is that the software produced would be
merely a clumsy cobbling together of different diagramming operations, leading to a facility
lacking in coherence and in which the user finds it difficult to choose which facility to use at any
one time.
6. SySpM: A proposed framework for understanding diagrams
Full discussion of this framework and its contributions is beyond the scope of this paper and can
be found in [9]. Here we briefly introduce the most important elements of the framework. The
central thrust of this framework is: Separating Symbolic from Spatial but allowing for their
Mapping. The framework is based on the notion that Sy is distinct and separate from Sp and is
irreducible to it. From this framework individual SySpM’s could be constructed. The term “a
SySpM” is used to denote a distinct (particular) collection of Sy and a distinct collection of Sp
and a distinct Mapping between the two Sy and Sp collection. Synonymous with this term are
drawing styles or types of diagrams. We grouped the Sy and Sp terms into primary and
secondary lists. The former includes Sy and Sp terms that are used in the simplest or basic form
of the SySpM. This category of Sy and Sp is also in all instances of diagrams of the relevant
SySpM. The latter includes terms that are used in complex diagrams of the relevant SySpM and
it also includes things that have to do with the relationships or interaction between the various Sy
things of the relevant SySpM. Development of each SySpM also includes list of Sp things that
do not map onto any Sy known as redundant Sp things. These could then become available for
other SySpM’s to use to give mixed diagrams, list of constraints and why they occur, and list of
events or changes relevant to the SySpM under consideration. Special features of a SySpM use
redundant Sp features to bring in secondary Sp features rather than another SySpM as mixed
diagrams. An example of this in a box and arrows SySpM is when lines are allowed to cross
other lines. This happens when one line is given a kink or a gap to indicate clearly that one is
passing over or under the other rather than connecting to it, e.g. electronic circuits. Sub types are
diagrams in which the original and simple SySpM is constrained/ complicated because of
specific needs usually associated with a type of application. This is achieved by brining in an
extra Sy constraint which, owing to Mapping, also gives a different Sp feel to the diagram. There
are at least three different ways of Sy sub types in a Box and Arrows SySpM: Networks, Lists,
and Trees. For each SySpM there might be special cases that do not 'fit' well. Many spatial
applications involve several of such special cases such as holes, discontinuities, and other
irregularities. We need to identify these and explain the problems they give, that is, what
constraints they break, either spatial (as is the case here) or symbolic.
A SySpM could contain features that are outside the range of its base symbolic types. This
recognises that each SySpM will be able to express only a subset of the symbol level, not all of
it. To express the whole wide range of things at the Sy level requires several different SySpM’s.
Within this context two types of mixedness are identified. One is when several SySpM’s are
present in a diagram but none dominates the overall meaning of the diagram. An example of this
is the Napoleon’s march on Moscow diagram. This type of mixedness is referred to “True mixed
diagrams”. The other type is referred to as “Augmented diagrams”. This type of mixedness has
one SySpM occupying a primary importance whilst other SySpM’s are added in and have
secondary importance. This type of mixedness occurs when redundant Sp features of a SySpM
are used to bring in Sy from other SySpM’s. For example, in a Box and Arrows SySpM
thickness of lines could be used to bring in quantitative value from Bar Chart SySpM.
In [9] eight SySpM’s were developed namely:
Boxes and Arrows
Communicating Similarity
Map of Objects
Set Membership
Bar Charts
Route Maps
Contour Maps
Surface Coverage
These SySpM’s are not seen as a final list but rather to demonstrate the potential of SySpM as a
framework in accounting for a diverse range of types of diagrams.
7. Relationship to other frameworks of understanding diagrams
In this section a review of how the SySpM framework for understanding diagrams relates to
these three extant frameworks is considered. The discussion of the frameworks will be in terms
of the parallels and differences they have with the SySpM framework. Two tables are shown to
present the content of the contrasts. One table lists the parallels whilst the other lists the
differences.
7.1 Semiotics
Parallels
|
|
The Triadic System
|
|
Type: Combination of Mark and Token
|
The general concept of mixed SySpM
and sub types
|
Token: instance of a type
|
A meta approach specific to
understanding diagram that conveys
symbolic meaning
|
A meta approach for understanding Diagrams,
Photographs, Paintings, Images
|
Diagrams are seen in a human context
|
Places a central emphasis on the role of the
human as an interpreter
|
|
|
The Triadic System
|
To do with diagrams in themselves, but
within human context
|
To do with human-human communication.
Nothing is a sign unless it is interpreted as a
sign declares Peirce (1931)
|
Starts with Symbolic aspect
|
Starts with Sensory
|
Makes clear distinction between
diagrams and arts or photographs. The
distinction is based on the notion of the
symbolic aspect being the qualifying
aspect of diagrams rather than aesthetic
aspects as in art
|
Conflates all forms of signs under one
framework defined by the triadic system
|
All three aspects of Symbolic, Spatial
and Mapping are equally meaningful,
important and inseparable
|
Tokens by themselves, apart from an
interpreter, have no meaning
|
All three aspects of Symbolic, Spatial
and Mapping are equally meaningful,
important and inseparable
|
Meaning of a sign is divided unequally
between the three parts of the triadic system of
signs: “Firstness and Thirdness can be more or
less vague; Secondness alone is always
precise.”
|
Purposeful and deliberate definitions
and mappings in each SySpM.
|
Arbitrariness or convention governs the
relationship between a sign and its object: “A
diagram is a (sign) which is predominantly an
icon of relations and is aided to be so by
conventions. Indices are also more or less
used” (1976)
|
Each SySpM is defined and worked out
in detail and with precise and distinct
mappings taking into account not the
final product as such but rather how it
has evolved to become a diagram
|
Takes a broad view of what a sign is, signs are
seen as a final product
|
Offers a richer and more elegant
framework of what constitutes a mixed
diagram based on the concept of
redundant Sp features.
|
Offers a weak framework for understanding
mixed diagrams in terms of collections of
signs
|
Handling change is included
|
Offers nothing or very little in the way of
handling change or dynamics of diagrams
|
Diagrams are seen as part of human
functioning rather than as given
constructs
|
Diagrams (signs) are seen as given
|
SySpM is considered to be an intuitive
framework for understanding diagrams
|
Understanding diagrams and signs in general
relies on prior human experiences and
knowledge
|
Allows for rule relaxation and hence
facilitates fluidity of interpretation or
construction of diagrams and aids
creativity
|
If a sign (according to Peirce’s definition) is
vague then it may inhibit users thinking unless
further signs or additional information become
available
|
Offers a framework for developing
diagramming software
|
Offers no direction for how diagram software
could be developed
|
The contrast, as shown by the tables, shows certain parallels but a greater number of differences.
The reason for this is built in the differences between the philosophical dispositions. The
framework of SySpM delivers an understanding of diagrams that is based on the notion of
Aspects and their mappings. This is a higher level of understanding than what semiotics is
concerned with. Whilst Peirce’s theory of signs struggles with interpreting ambiguous or not
well known visual signs, the framework of SySpM on the other hand provides the clarity needed.
Under Peirce’s framework such signs or diagrams pose difficulties because the mapping between
the Firstness and Secondness is less likely to materialise. SySpM provides intuitive mappings
between Sy and Sp in distinct ways across the SySpM’s developed. This enables clarity and
precision of distinct ways of interpretation.
7.2 Semiology of Graphics
|
|
Visual Variables
|
|
Attempts to provide a meta framework for
understanding diagrams
|
Aspects are irreducible and inseparable
to each other
|
Separates (tacitly) spatial/visual things from
each other
|
Distinct and deliberate mappings
between collections of Sy and Sp things
|
General association, based on human visual
processing capabilities, between the visual and
spatial variables and quantitative, nominal,
and ordered features
|
|
|
Visual Variables
|
Explicit recognition of the richness of
spatial and symbolic aspects
|
Limited recognition of spatial and symbolic
things
|
Mappings between Symbolic (Sy) and
Spatial (Sp) aspects is intuitive
|
Mapping between spatial features and
quantitative is driven by the ability of human
visual processing capabilities
|
Both Sy and Sp are equally important
and irreducible to each other or any
other aspect
|
Elevates visual and graphical features
(Blackwell and Engelhardt, 1998)
|
Separates Sy from Sp, see section on the
choice of terminologies for both aspects
|
Conflates sensory, symbolic, and spatial
phenomena
|
Flexible by making distinct
differentiation between the various
SySpM’s and other features such as rule
relaxation
|
Rigid in its approach: “Only implantation by
point can manifest absolute quantities in x”
(Bertin, 1981)
|
Recognises and promotes the richness
of the spatial aspect
|
Constrains the richness of the spatial aspect of
the world around us to a limited number of
visual variables
|
Handling change is supported
|
Static description of diagrams
|
Offers a framework for developing
diagramming software
|
Offers no direction for how diagram software
could be developed
|
In contrast with semiotics, the work of Bertin seems to provide a closer contrast with the
framework of SySpM, hence the small number of differences. This could be explained by
pointing out the emphasis of semiology of graphics on diagrams that communicate meaning
rather than on all forms of visual representations. However, there still remain fundamental
differences between the two frameworks, particularly when semiology elevates certain chosen
spatial and sensory features. Bertin’s matrix of associating the various visual variables is
intended to produce better graphics rather than to describe differences and/or similarities
between various diagrams. With this approach in mind Bertin’s framework finds it difficult to
make distinctions between a list, a graph, and a hierarchical diagram. This indifference is a
source of the problem of the variety of diagram types. However, under SySpM precise
distinction are formed between all types of diagrams, based on their unique Sy, Sp and M.
SySpM provides a basis for extending and enriching Bertin’s framework by considering the
richness of Sy and how this may be expressed through distinct Sp and M.
7.3 Notational Schemas
Parallels
|
|
Notational Schemas
|
|
Component schemas
|
|
Compositional schemas
|
|
Primary types
|
|
Hybrid types
|
|
|
Notational Schemas
|
Symbolic and Spatial are both equally
important and needed
|
Strong emphasis on spatial constraints
|
|
Application oriented
|
Diagrams that convey symbolic
meaning
|
Includes pictures that do not convey symbolic
meaning
|
Each SySpM is distinctly different from
other SySpM’s
|
Primary types in themselves contain mixed
symbolic meanings: a Map has at least Routes
and Objects.
|
Irreducibility and equal importance of
the symbolic, spatial and their
mappings.
|
Strong emphasis on the visual/spatial
|
Distinction between spatial components
that have Sy meaning and those that do
not have Sy meaning (redundant)
|
Components do not make a distinction
between meaningful and meaningless
(redundant) components
|
Handling change is supported
|
Static description of diagrams
|
Offers a framework for developing
diagramming software
|
Offers no direction for how diagram software
could be developed
|
The issues and the overall approach of notational schemas seem to extend and enrich those
discussed in Richards (2000). The framework is spatially oriented and this makes it differ from
the framework of SySpM.
8. Summary
Although there are certainly parallels between the framework of SySpM and the three extant
frameworks, nevertheless, there are fundamental differences between them. The differences are
centred around what is seen as by each framework as important to our understanding of
diagrams. The SySpM framework distinguishes itself from the rest of the other frameworks by
explicitly acknowledging the notion of aspects namely the Symbolic and Spatial aspects, and
their mappings as a good way of understanding diagrams. The framework enabled the
development of a number of notions which helped to deal with these problems, namely:
The notion of Aspects and their irreducibility
The concept of the basic SySpM
The concept of more complex versions of A SySpM
Sub types
Primary and secondary Sy and Sp things
Redundant Sp features
Mixed and augmented diagrams
Relaxation of rules
Handling change
Constraints
Potential of extending the framework to include other aspects such as sensory (colour) and
dynamics of diagramming.
All of these notions give a measure of the goodness of the framework, particularly in helping to
deal with the problems of variety of diagram types, mixed diagrams, and handling change.
9. The SySpM framework and the three problems
The SySpM framework deals with the problem of the variety of diagram types at two levels. One
is at the level of individual SySpM’s. The other level is within each individual SySpM where a
wide range of different diagrams can be accounted for through the notions of sub types, special
features, true and augmented diagrams. The deliberate, distinct, and rich account of what
constitutes Sy, Sp, and M for each SySpM allows not only the potential to accounting for a very
wide range of diagrams but also in differentiating them with precision.
The problem of mixed diagrams is dealt with through the notions of augmented and true mixed
diagrams. [14] demonstrated the potential of the eight SySpM’s in accounting for four complex
and semantically mixed real world diagrams:
A Quantified Flow Chart
A Pressure – Volume Graph
Diagram depicting Napoleon’s March on Moscow
An Ordnance Survey Map
Change in diagrams is give precision and flexibility at the same time. Precision because each
editing operation is specific to the type of SySpM under consideration and distinguished from
other SySpM’s. Flexibility is accommodated by the notion of rule relaxation.
10. Conclusion
This paper responded to renewed interest in our understanding of diagrams. Whilst existing
definitions are based on a specific perspective the framework proposed in this paper pursues a
much richer approach. The proposed framework is contrasted with other main approaches of
understanding diagrams. The merits of the framework are discussed in relation to three main
problems concerning diagrams. The work carried introduced in this paper is part of an ongoing
research which aims to develop drawing software based on the framework of SySpM capable of
handling the challenges presented by the three main problems.
9. References
- Anderson M and McCartney R (2003) Diagram processing: Computing with diagrams.
Artificial Intelligence 145, pp 181- 226. ELSEVIER.
- Basden A, Brown A J, Tetlow S D A, Hibberd P R (1996) Design of a user interface for a
knowledge refinement. tool International Journal of Human Computer Studies, vol.45,
pp.157-183.
- Bertin J, (1983), “Semiology of graphics”. University of Wisconsin press.
- Bishop I (1994) In: Visualisation in modern cartography, Ed. A M MacEachren and D R
Fraser Taylor. Pergamon 1994.
- Blackwell A F and Engelhardt Y (1999) A Meta-Taxonomy for Diagram Research. In P.
Olivier, M. Anderson & B. Meyer (Eds.), Diagrammatic Representation and Reasoning,
Springer-Verlag.
- Carswell C M and Wickens C D (1988) Comparative graphics: History and applications
of perceptual integrality theory and the proximity compatibility hypothesis. Technical
Report, Institute of Aviation, University of Illinois at Urbana- Champaign.
- Casner S M and Larkin J H (1989) Cognitive efficiency considerations for good graphic
design. Proceedings of the Eleventh Annual Conference of the Cognitive Science
Society, Ann Arbor Michigan.
- Cheng P (1996) Diagrammatic Knowledge Acquisition: Elicitation, Analysis and Issues.
In N. R. Shadbolt, H. O'Hara, & G. Schreiber (Eds.), Advances in Knowledge
Acquisition: 9th European Knowledge Acquisition Workshop, EKAW'96. (pp. 179-194).
Berlin: Springer-Verlag.
- Cheng P, Cupit J, (2001), Supporting diagrammatic knowledge acquisition: an
ontological analysis of Cartesian graphs. Int. J. Human – Computer – Interaction Studies,
54.
- Cheng P, Lowe R, Scaife M (2001) Cognitive Science Approaches To Understanding
Diagrammatic Representations. In Thinking with Diagrams, ed. Alan F. Blackwell,
Kluwer Academic Publishers, Dordrecht
- Cleveland W S (1985) The Elements Of Graphic Data. Wadsworth Publishing.
- Cleveland, W.S. and McGill, R. (1986) An experiment in graphical perception. Int. J.
Man-Machine Studies 25, 491-500.
- Engelhardt Yuri (2002), The Language of Graphics, PhD thesis, 2002, Institute for Logic,
Language & Computation, University of Amsterdam. ISBN 90-5776-089-4
- Fathulla K (2006), Understanding Diagrams Based on Symbolic and Spatial Mapping.
PhD thesis, University of Salford, United Kingdom.
- Garland K (1979) Some general characteristics present in diagrams denoting activity,
event and relationship. Information Design Journal, 1(1), 15-22.
- Gibson J J (1977) The theory of affordances. In: Shaw R, Bransford J (eds): Perceiving,
Acting and Knowing Erlbaum, Hilsdale, NJ.
- Gurr C (2000) Tutorial T1 - Diagrammatic Reasoning and Visual Languages, IEEE
Symposium on Visual Languages, Seattle, Washington, September 10-14, 2000
- Horn R E, (2001) Visual Language and Converging Technologies in the Next 10-15
Years (and Beyond). The National Science Foundation Conference on Converging
Technologies (Nano-Bio-Info-Cogno) for Improving Human Performance.
- Jobson J, Rahman Z, and Woodell G A (2001) The Spatial Aspect of Color and Scientific
Implications of Retinex Image Processing, SPIE International Symposium on AeroSense,
Proceedings of the Conference on Visual Information Processing X.
- Kim J, Hahn J and Hahn H (2000) How Can We Understand a System with (so) Many
Diagrams? Cognitive Integration Processes in Diagrammatic Reasoning, Information
Systems Research, Vol. 11, No. 3, 284-303.
- Larkin J H and Simon H A (1987), "Why a diagram is (sometimes) worth a ten thousand
words. Cognitive Science, 11, pp.65-99.
- Lohse G L, et al,(1994), “A classification of visual information”, Communications of the
ACM, Vol. 37, No. 12.
- Mackinlay J D (1998) Automating the design of graphical presentation of relational
information. In: Maybury, M T, Wahlster, W (eds): Readings in intelligent user
interfaces.
- Markowitz V and Shoshani A (1998), On the Correctness of Representing Extended
Entity-Relationship Structures in the Relational Model. SIGMOD Conference 1989: 430-439.
- Neilson I, Lee J (1994) Conversation with graphics: implications for the design of natural
language/graphics interfaces. Int J Human-Computer Studies, vol.40 pp.509-541.
- Osborn J R (2005) Theory Pictures as Trails: Diagrams and the Navigation of Theoretical
Narratives. Cognitive Science Online, Vol.3.2, pp. 14-44.
- Peirce C S (written around 1897, republished in 1932). Elements of Logic. In C.
Hartshorne and P. Weiss (Eds.), The collected papers of C.S. Peirce. Harvard University
Press.
- Peirce, C S (1902) Logic, Regarded As Semeiotic. (The Carnegie application of 1902)
- Peirce, C S (1931) Collected Writings (8 Vols.). (Ed. Charles Hartshorne, Paul Weiss &
Arthur W Burks). Cambridge, MA: Harvard University Press
- Peirce, C S (1931-36) In "Some Consequences of Four Incapacities," Collected Papers of
Charles Sanders Peirce, 8 vols., ed. Charles Hartshorne and Paul Weiss (vol. 1-6), Arthur
W. Burks (vols. 7-8), Cambridge: Harvard University Press.
- Peirce, C S (1976) (NEM) New Elements of Mathematics, ed. by Carolyn Eisele, 4 vols.,
Mouton, The Hague, 1976.
- Pinker R S (1990), A theory of graph comprehension. In R Freedle, Ed. Artificial
Intelligence and Future of Testing. Hillsdale, NJ: Lawrence Erlbaum Associates.
- Rowe P (1987) Design Thinking. Cambridge: MIT Press.
- Shneiderman, B, Card, S.K, and Mackinlay, J.D (1999) Readings in information
visualization : using vision to think (Morgan Kaufmann series in interactive)
- Sluter R S (2001) NEW THEORETICAL RESEARCH TRENDS IN CARTOGRAPHY.
Brazilian Journal of Cartography, a publication of SBC - Brazilian Society of
Cartography, Geodesy, Photogrammetry and Remote Sensing.
- Sowa J F (1984) Conceptual Structures: Information Processing in Mind and Machine,
Addison-Wesley, Reading, MA.
- Taylor M M, McCann M M, Tuori M I (1984) The Interactive Spatial Information
System. DCIEM Report No 84-R-22 Defence and Civil Institute of Environmental
Medicine, Ontario.
- Tufte E R (1983) The Visual Display of Quantitative Information. Graphics Press.
- Tversky B (2001) Spatial Schemas in Depictions. In Spatial Schemas and Abstract
Thought. Gattis, M. (ed.), Cambridge, MA:MIT Press.
- Winograd T (1995) From programming environments to environments for designing.
Communications of the ACM, vol.38 pp.65-74.
- Wood D (1993) The power of maps. Routledge, London.
- Zhang J 1(1996) A Representational Analysis of Relational Information Displays.
International Journal of Human- Computer Studies.
|