Inside the
Conductor’s Jacket:
Analysis, Interpretation
and Musical Synthesis of Expressive Gesture
by
Teresa Marrin
Nakra
Submitted to the Department
of Media Arts and Sciences,
School of Architecture and
Planning,
in partial fulfillment of
the requirements for the degree of
Doctor of Philosophy
at the
MASSACHUSETTS INSTITUTE OF
TECHNOLOGY
February 2000
copyright Massachusetts Institute
of Technology 2000. All rights reserved.
Thesis Supervisors: Tod Machover
(MIT) and Rosalind W. Picard (MIT)
Thesis Readers: John
Harbison (MIT) and David Wessel (U. C. Berkeley)
Table of Contents
Title,
Abstract, and Committee Members
Dedication
Acknowledgements
Chapter
1: INTRODUCTION
1.1
Vision
1.2
Overview
1.2.1 The slow
asphyxiation of classical music
1.2.2 Why our new
instruments are not expressive enough
1.2.3 What needs
to improve for interactive music to become an art form
1.2.4 Instruments
for Free Gesture
1.3
Motivation
1.3.1 Hyperinstruments,
the Brain Opera, and the Digital Baton
1.3.2 Why continue
with conducting as a model?
1.3.3 Conducting
Technique
1.3.4 Why conducting
might not be a good model for interactive music systems
1.3.5 Interpretive
variation as the key to emotion in music
1.3.6 The Significance
of Music for Us
1.4
Approach
1.4.1 Framing
the Problem
1.4.2 Results
Chapter
2: BACKGROUND AND RELATED WORK
2.1
Conducting and Interpretation Pedagogy
2.2
Previous Conductor Study
2.3
Theories of Expression and Emotion in Music
2.3.1 Leonard
Bernstein
2.3.2 Manfred Clynes
2.3.3 Expression
"Rules" Research
2.4
Theoretical frameworks for mappings between gestures and music
2.4.1 David Efron
2.4.2 Joel Ryan
2.4.3 Teresa Marrin
2.5
Interactive systems for conductors and conductor-like gestures
2.5.1 Hyperinstruments
2.5.2 Radio Baton
2.5.3 The Virtual
Orchestra
2.5.4 A MultiModal
Conducting Simulator
2.5.5 The Conductor
Follower of the MIT Electronic Music Studio
2.5.6 Gesture Recognition
and Computer Vision
2.6
Wearable interfaces for real-time interactive music
2.6.1 BodySynth
2.6.2 BioMuse
2.6.3 Lady’s Glove
2.6.4 DancingShoes
2.6.5 Miburi
2.6.6 Benoit Maubrey’s
Electro-Acoustic Clothing
2.6.7 The Musical
Jacket
2.6.8 Chris Janney’s
HeartBeat
2.6.9 Others
Chapter
3: THE CONDUCTOR’S JACKET SYSTEM
3.1
Background
3.1.1 Preliminary
Investigations
3.2
System Design
3.3
Implementation Details and Issues
3.3.1 Design Criteria
3.3.2 Measures
of expression that were not used
3.3.3 Design/Implementation
Problems
3.4
Data Collection Experiments
3.5
Formatting, Timing, Graphing and Filtering the Data
3.5.1 Non-real-time
filters in Labview
3.5.2 Non-real-time
filters in Matlab
3.5.3 General Issues
with this Data Set
Chapter
4: VISUAL ANALYSIS OF CONDUCTOR DATA
4.1
Interpretive Feature Identification
4.1.1
Use of the left hand for expressive variation
4.1.2
The flatlining effect
4.1.3
The direct, one-to-one correlation between muscle tension and dynamic intensity
4.1.4
Predictive indications
4.1.5
Repetitive signals minimized until new information appears
4.1.6
Treatment of information-bearing vs. non-information bearing gestures
4.1.7
Frequency of unnecessary actions decreases with experience
4.1.8
Clarity of signal during slow, legato passages correlates with experience
4.1.9
Division of labor between biceps, triceps, and forearm
4.1.10
Rate encoding
4.1.11
The link between respiration and phrasing
4.1.12
Large GSR peaks at the beginning of every piece
4.1.13
GSR baseline variance as a strong indicator of experience
4.1.14
Temperature baselines
4.2
Other features for future treatment
Chapter
5: HYPOTHESES OF EXPRESSION
5.1
Interpretation of results from analysis
5.2
Hypotheses of Expression
5.2.1 Efficiency
5.2.2 Intentionality
5.2.3 Polyphony
5.2.4 Signal-to-Noise
Ratio of Expertise
5.2.5 Tri-Phasic
Structure of Communicative Gestures
5.2.6 Bi-Phasic
Pulse Structure
5.2.7 Evolution
of Conducting Gestures
5.2.8 Unidirectional
Rate Sensitivity
5.2.9 Musical Flow
State
5.3
What is Expression?
Chapter
6: THE GESTURE CONSTRUCTION
6.1
System Architecture
6.1.1 Jacket Design
6.1.2 System Design
6.2
Real-time Signal Processing
6.3
Code Interfacing between Filters and Mapping Structures
6.4
C++ Mapping Algorithms
6.4.1 Musical
Algorithms of the Gesture Construction
6.5
Resulting pieces
6.5.1 Etude 1:
Tuning
6.5.2 Etude 2:
One-to-One Relationship
6.5.3 Etude 3:
Beats, Cutoffs, and Crescendo/Diminuendo on Sustain
6.5.4 Bach’s Toccata
and Fugue in D minor
6.5.5 Song for
the End
6.6
Public Performances and Demonstrations
Chapter
7: EVALUATION AND FUTURE WORK
7.1
Evaluation
7.2
Biggest Lessons
7.3
Design Wins
7.4
Future Work
7.4.1 Analytical
Improvements
7.4.2 Hardware
Improvements
7.4.3 Software
Improvements
7.4.4 Artistic
and Theoretical Improvements
Chapter
8: CONCLUSIONS
8.1
Design Issues for Sensor Instruments
8.1.1 The Disembodiment
Problem
8.1.2 Mimesis
8.1.3 Traditional
vs. Digital Instruments
8.1.4 Distinctions
between Musical and Physical Gesture
8.2
A Framework for Future Research
8.2.1 Extensions
to the Conductor’s Jacket Project
8.2.2 Implications
from the Conductor’s Jacket for Other Work
8.3
The Future of Musical Performances
8.3.1 The Need
for New Live Concerts
8.3.2 Possibilities
for Great Art with Sensors
8.4
Coda
REFERENCES
Appendix:
Information on the Conductor’s Jacket datafiles
Developed by Teresa Marrin
Nakra at the Media Laboratory, MIT, Cambridge,
Massachusetts, with support
from Motorola, IBM, and the Things That Think Consortium.
For use by Teresa Marrin Nakra.