direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Sie sind hier

TU Berlin

Page Content

Gesture research at the Research Center for Semiotics


Back to previous page

Everyday numerous gestures of different kinds are produced , for example, to describe a rollercoaster, to point something out or to emphasize the spoken word. While official sign language has already been relatively throughly examined from a modern linguistic and semiotic perspective, the everyday gestures of hearing people have only recently been allocated similar specific research importance. In 1995 a lecture at the Research Center for Semiotics (RCS) at the TU Berlin founded joint gesture research with the Free University Berlin (documented in Schmauser and Noll 1999).

IRP Gesture recognition with sensor gloves

The interdisciplinary research project (IRP) "Gesture Recognition with Sensor Gloves" (1994-98) formed the center of the institutionalized research activities. It unitied the three TU study areas robotics (Prof Dr-Eng. Günter Hommel), microsensorics (Prof Dr-Eng. Ernst Obermeier) and semiotics (Prof Dr Roland Posner). The aim of the IRP was the sensor-supported recognition of significant gestures, which are called  "emblems" in gesture research. Such emblems are well known and can be effortlessly translated into linguistic expression. Everyday examples are the beckoning over of a person  ("Come here!"), tapping of the temple or winding motion next to it ("You're crazy!") and a drinking motion ("Let's raise our glasses!"); specialist gestures like those of a fire fighter for example formed the second field of investigation.

The use of the patented TU sensor glove to reveal gestures in contrast to a conventional video recording clearly provided more information. A detailed analysis of gestures requires the recording of numerous parameters, which are only completely invisible from a certain point. Alongside the position of hands and their orientation in the space (relative to the rest of the body), these include, in particular, the positions of the individual fingers and finger joints. Moreover, the sensor glove also allows the pressure of movements to be registered – information which an experienced human observer can only deduce indirectly, for example, detected from  the colouring of the knuckles. The sensor recording delivers automatic values for the movement process, which can then be registered mathematically. These allow a formal description of the individual emblems with regard to the variation range within which they are still recognizable. It is thus possible to differenciate formally and exactly between an individual gesture and a gestum (approximately analogous to the difference between phon vs. phonem). In these investigations semiotics delivers the empirically secured substance for robotics and elaborates a semantic of gestures, consequently allowing for the automatic relation of gestures and meanings with each other. Parallel to this physical registering of gestures, the respective linguistic expression has to be collected and categorised,  because it is exactly the body-related phraseologism of a linguistic community which grants insight into the structure of the underlying mental models and their interconnection. 

The Berlin Lexicon of Everyday Gestures

Approximately 150 families of everyday gestures were systematically registered at the RCS by video recording, were then digitalized and were finally put togther in a multimedia database. In this form the entire recorded material is available for secondary research. The entire documentation to-date of the four-year project is the concluding report (Posner, among others, 1999); The Berlin Lexicon of Everyday Gestures is considerably more extensive (Posner, among others, in prep.). Developing the work of Ekman, Johnson und Sparhawk, this emblem lexicon presents the approximately 150 gesture families. Considerable progress in comparison with other work to-date are the systematic processing, which allows the gestures to be classified  in a hierarchical pattern according to their meaning in families and sub-families, and the reader-friendly presentation, which records the extremely complex material in a number of complementary sign systems.

The textual part of each entry describes, from an expression perspective, the form of each structural variant, that is the respective parts of the body carrying out the action, as well its place, orientation and movements. The Hamburg Notation System (HamNoSys), which translates the listed parameters in a lineary sequence of conventional signs, which also allows the expression of metastructural information, was used for the transcription of these morphological details  – for example a mirror image gesture of both hands or the iteration of a gesture. From a content perspective, all meaning variants including their speech-theoretic components and allocated linguistic expressions are listed. Thus the gesture with which imaginary sweat is wipped off the forehead, the locution center of which is "sweating", the illocution "indication" and the perlocution "to display to someon that a tricky situation has just been mastered". A common interjection here is "Phh!", an everyday idiom "That's a load off my mind".

The empirical investigation of the selected gesture families was carried out as a quota sample survey, in which the test participants are chosen to represent a cross section of the Berlin population. The test participants saw video images with selected gestures and answered a questionaire about the meaning and the purpose behind each gesture. The statistical information gathered was processed graphically for the lexicon, so that with the help of diagrams the reader can easily recognize how the interpretation of each gesture can be analysed according to sex, age and origin (east Berlin, west Berlin, foreign).

The information described to-date is complemented by pictorial material. Moment recordings of the video present the respective typical phase of the gesture, so that the combination of this picture and the gesture names  ("wiping sweat from the forehead") guarantees that the gesture is recognized. As a result the reader can often successfully link very abstract texual descriptions of the gesture forms with his/her own knowledge. Further pictorial information shows how the gestures in other pictorial sign systems are presented. Comics and contemporary caricatures from the daily press proved to be a productive treasure trove.

It became clear throughout the course of the first project phase that morphologically and functionally related gestures can only be usefully isolated from each other if their common history is reconstructed. An etymological part in contrast to the original conception shows the development story of each gesture family from the first sources to today, which introduces all documents from text and vase painting to sculpture as possible knowledge sources. This reconstruction is based on the research-leading hypothesis that every gesture has developed from a form-related use-movement so that, for example, every drinking emblem represents a known form of liquid intake in the respective cultural context. Since consequently all representations of everyday performances also give information about these general motorical schemes, it is possible to to trace the historical development way beyond the available sources to phylogeny.

The institutionalization of gesture research

The IRP made a decisive step towards the institutionalization of gesture research through the organisation of the international symposium "The Semantics and Pragmatics of Everyday Gestures" (cf. Müller 1998 and Müller and Posner 1999). Between April 23 and 26 1998 about 30 gesture researchers from around the world met at the RCS to introduce research results to-date and to elaborate the basics for the recording of comparative gesture lexica. The starting point for the conference were the problem fields and possible solutions identified by the IRP. The fundamentals about which body movements should be included in gesture lexica – only gestures in the narrow sense or also whole body movements, only emblems or also speech-accompanying gestures. The focus subjects include pragmatics and cultural variance of gestures, the problem of national lexica, the form and function of deictic gestures, specialist gestures in some examples as well as studies on selected gesture forms in different cultures. In order to connect existing research activities as closely as possible and to work closer together in future, it was decided at the podium discussion to found the International Society for Gesture Studies (ISG) at the following conference. Its official newspaper Gesture: An International Journal of Research, Theory, and History (editors: Adam Kendon and Cornelia Müller) is to appear from  2001 with Verlag John Benjamins (Amsterdam and Philadelphia).

The planned following conference took place from April 1 to 4 2000 at the Universidad Fernando Pessoa in Porto (Portugal). The title was "Gestures: Meaning and Use" and was  organised together with the Technical University Berlin, the University of North Carolina as well as the Universidad Fernando Pessoa (program: www.ufp.pt/gestures, conference report: Edmondson 2000). This second ISGS conference was successful in generating a worldwide focus of attention for gesture research and increasing cooperation. This particularly affects the everyday lexica, which are drawn up by research groups from  Aix-en-Provence (I. Guaïtella), Barcelona (L. Payrató), Berlin (R. Posner), Moscow (G. Kreidlin), Rom (I. Poggi), Sofia (I. Kassabov) and Tokyo (Y. Tohyama and P. Tumarkin) among others. The text versions of the lectures will appear in a conference volume.

The next important event on gesture research was the  "Semiotic Summer Institute" in Urbino (Juli 9 to18 2000), especially with the colloqiums "The Body and Its Stagings" and "The Rhetoric of Gestures" (Centro Internazionale di Semiotica e di Linguistica, Piazza del Rinascimento 7, I–61029 Urbino, Fax: +39-722-2289, E-Mail: semiotica@bib.uniurb.it).

Investigation of artefact-related gestures

In addition to this, a more detailed study of artefact-related gestures is planned. Many detailed analyses of gesture families show how closely their figure and meaning is connected with the allocated artefact. Many emblems emulate the figure of the missing artefact, for example the picture frame emblem, with which a photo tries out a frame, others simulate dealings with artefacts, for example the bottle-opener emblem in the absence of a bottle opener. The development of such emblems reflects the development of each available artefact. At the beginning of the 20th century the telephoning emblem consisted of turning an imaginary crank, it then became a dialling disk and even later a button pad – today's variant is a hand to the ear as a mobile phone. It should also be investigated among other things how use movements, which necessarily adapt to the corresponding artefact, become gestures, whose meaning changes in turn sytematically. When gestures are generalized then the telephone emblem can also mean "We'll call". Another, widespread change in meaning is metaphorization. In the case of the screwing in of a real screw with a screwdriver being replaced by a figure-like screwing movement without replacing these two artefacts, then it can only mean "Give me the screwdriver" (which would be a prerequisite for a certain context), but also "He has a screw loose", in the case of the gesture being produced at a certain place – namely next to the temple of the sender. A semiotic analysis, historical focus and culture-comparative documentation of such changes makes it possible to prove processes of meaning production as very detailed empirically anchored and thus to elucidate a connection between technology and semantics, which is often overlooked.

The DGS 9th international congress was organised, under the leadership of Cornelia Müller and Massimo Serenari, to examine the existing research results on this subject and to bring together scientists from different disciplines (Dresden, Oktober 1999) with a section entitled "Artefact-related Body Movements as a Basis for Gestures". The main focus  was "Machines and History" and therfore offered an optimal framework for investigation into the described interconnection of gesture development and artefact development (see Posner 2000).

Further gesture-related activities of the RCS

The linking-up of gesture research led to the founding of a new international book series with the title Body, Sign, Culture (editors: Hartwig Kalverkämper, Reinhard Krüger and Roland Posner; Berlin: Berlin-Verlag). This series is concerned with all types of signs produced by the body and therefore offers a forum for contributions pointing the way from all relevant disciplines from biology and anthropology to linguistics and theatrical studies.

To start the planned culture comparisons, a cross-section of the IRP tested 150 gesture families already in different cultural circles were shown, in Italy (Massimo Serenari), Japan (Christine Kühn) and New Guinea (Arnold Groh). A joint publication of the results as well as cooperation with Corea is in preparation.

Speech-accompanying gestures were analysed parallel to the study of emblems  – in particular by Sabine Kowal and Daniel O’Connell –  which structure the spoken word or illustrate certain aspects of it (so-called "illustrators"). Typical examples are gestures with which the fisherman shows the length of his catch or someone draws the physical structure of a roller coaster in the air.

Several main seminars examined interjections which are typically produced with certain gestures. An index finger placed over the lips, asking for silence is generally accompanied by "pssst", and a hollowed hand behind the ear usually with a interogatively intonated "eh?". A further research aim is the correct phonetic transcription of these interjections and their general linguistic comparison.

Some dissertations with gesture-related themes are currently being written at the RCS. Massimo Serenari will document the evaluation of the empirical material. One of the aspects is the phase structure of gesture acquisition: when is knowledge about gestures consolidated and from when are metaphorizations understood? The geographical structure is also analogously examined, whether today – 10 years after German unification  – differences between east and west Berlin gestures can still be detected. Moreover, the extent to which foreigners are integrated with regard to their gesture production and reception in the cultural community provides another focus for investigation. A second dissertation topic is emblems with sexual content. It is known from everyday knowledge that many of them are not used to make sexual contact but on the contrary to offend or to break off contact. Christine Petermann empirically investigates which emblems are used in  heterosexual behavior for sexual invitation; her dissertation is entitled  "Emblematic Gestures of Human Sexual Behavior". Because in this area questioning in laboratory conditions is highly likely to lead to a distortion of the answers then the method used was observation of the participant in suitable locations, such as discos and pubs. It showed that the success of gestures was heavily dependant on the basic conditions and that increasingly also women produce initiating sexual gestures (for example the so-called "coward").


Literature

Benthien, Claudia (1999): Im Leibe wohnen. Literarische Imagologie und historische Anthropologie der Haut. Berlin: Berlin-Verlag.

Edmondson, William (2000): Report about the international conference, "Gestures: Meaning and Use". Appears in: Zeitschrift für Semiotik 22, 2.

Müller, Cornelia (1998): Report about the international symposium, "The Semantics and Pragmatics of Everyday Gestures". Zeitschrift für Semiotik 20: 436-439.

Müller, Cornelia and Roland Posner (eds.) (1999): The Semantics and Pragmatics of Everyday Gestures. Berlin: Berlin-Verlag.

Posner, Roland, Reinhard Krüger, Thomas Noll und Massimo Serenari (1999): Concluding report of the interdisciplinary research project  "Gesture Recognition with Sensor Gloves". The semi-semiotic report appears in: TU Berlin papers on linguistics.

Posner, Roland, Reinhard Krüger, Thomas Noll and Massimo Serenari (in prep.): Berlin Lexicon of Everyday Gestures. Berlin: Berlin-Verlag.

Posner, Roland (2000): Report about the 9th international DGS congress. Zeitschrift für Semiotik 22: 115-121.

Schmauser, Caroline and Thomas Noll (eds.) (1999): Körperbewegungen und ihre Bedeutungen. Berlin: Berlin-Verlag.



Contacts

Roland Posner Prof PhD

  • Project leadership
  • Semantics and pragmatics of everyday gestures

Reinhard Krüger Prof PhD 

  • Etymology of everyday gestures

Thomas Noll PhD 

  • Formal description of gestures
  • HamNoSys

Massimo Serenari MA

  • Experimental investigations
  • Presentation of gestures in comics

To top

Zusatzinformationen / Extras

Quick Access:

Schnellnavigation zur Seite über Nummerneingabe