Capture the story of Indian dance

BLOG: Heidelberg Laureate Forum

Laureates of mathematics and computer science meet the next generation
Heidelberg Laureate Forum

The dance is the mother of the arts.
Music and poetry exist in time; painting and architecture in space.
But the dance lives at once in time and space.

Curt Sachs

Tanwi Mallick, HLF14 participant: I hail from India  – a country rich in cultural heritage. My research interests are computer vision and in more detail the automated analysis and interpretation of Indian classical dance. Until today only very little work has been done in computer analysis of dance by tracking dance movements. And if so, this research mainly was concentrated on dances like Ballet, Samba and Salsa.

I have picked up this maybe exotic research topic as in India dance has long been deeply interwoven into the social fabric, especially in classical forms, called Indian Classical Dance (ICD). ICD has many variants such as Bharatanatyam, Kathak, Oddisi, Manipuri, and Kuchipuri, which makes it even more interesting. Each variant of ICD creates a very specific sequence of rhythmic patterns synchronized with music. And these special variants are associated with a rich set of body postures and gestures – grammars for the performance have emerged, mythological stories, sculptural depictions, and various other cultural artefacts. In this way, these dance forms embody a correlated and collated collection of knowledge sources.

In short, an ICD consists of visual (posture, movements, and expressions), auditory (music, tempo, rhythm, and intonation) and textual (lyric of the song) information that tell a story through body movements, hand gestures, vocal and instrumental music, facial expression (emotion), costume, and make-up. As the age-old Indian dance treatise summaries:

The body should catch up to the time,
The hand must explain the meaning,
The eyes must speak the emotion,
And the feet must beat the time-measure.


Recent advances in short-range, low-cost depth imaging technology have led to the development of affordable multimedia cameras like Kinect that can detect and track various human movements in real-time.  The research I am involved in is not only a simple tracking. The automated analysis and interpretation of the various dances can be useful in several ways:

  1. Create dance tutoring systems,
  2. Preserve cultural heritage by dance transcription,
  3. Synthesize and create animated avatars,
  4. Interpret the story of an ICD recital, and more.

Dance tutoring systems for example can provide instructional feedback to the dancer and evaluate dance performances in real-time. It can capture the motions of a novice dancer and show the difference in postures and movements with respect to the stored performance of an expert dancer. Here I highlight how the novice dancer goes astray in making a posture with reference to her expert dancer.

Dance tutoring system with automatic instructional feedback
Expert Dancer klein
Expert Dancer
Novice Dancer
Novice Dancer

Analysis and interpretation of dance can help convert the audio-visual information of dance into a script using the well-known labanotation for dance. I am working on automating such dance transcription processes as this might help immensely to preserve the heritage of ICD. And such transcripts can also be very handy to exchange dance ideas between several performers.

Labanotation of a dance posture.
RGB Depth Labanotation
LA – Left Arm, LB – Left Body, LL – Left Leg, LS – Left Support. It is similar for the right part of the human body.

As mentioned before, such analysis of dance movements can help synthesize 3D dancing avatars for visual evaluation of composed choreographies. Certainly such analysis of dance is not only about analysis of images or videos. There are more components such as audio, music, lyric (text), and grammar that together with the visuals make up a complete dance. Plus the real interpretation of a dance involves comprehending the multi-dimensional aspects of it in the context of culture, story, emotion and gesture. Such the recognition of a body movement, hand gesture or facial expression from an image or a video can lead to the discovery of the semantic concept defined by the keyword(s) expressed. And a spatio-temporal sequence of hand gestures and body postures can lead to the discovery of high-level concepts like a mood, words of a poem, or a portion of a mythological story.

Sequence of postures from Bharatanatyam and their corresponding skeletal projection


Currently the analyses of body movements and facial expression are underway. These would subsequently be integrated with the domain knowledge of ICD. In my analysis I mainly focus on the ICD Bharatanatyam and on the recognition of “adavus” (choreographic forms), “hasta-mudras” (hand gestures) and “rasas” (emotions). Detection of percussion beats and their synchronization with movements would establish a complete narrative.

I would be very happy to exchange my current work and ideas with other attendees of the HLF14 and am curious whether there will be someone with further research ideas!

Tanwi Mallick All pictures on courtesy of Tanwi Mallick

 Tanwi Mallick is a PhD candidate at the Department of Computer Science and Engineering at the Indian Institute of Technology Kharagpur, India. Together with her supervisors Prof. Partha Pratim Das and Prof. Arun Kumar Majumdar she does research in the fields of human computer interaction. She is supported in her research by TCS, India.

Avatar photo

Posted by

YOU! We invite attendees of the Heidelberg Laureate forum to support our blog with additional valuable content. Email

Leave a Reply

E-Mail-Benachrichtigung bei weiteren Kommentaren.
-- Auch möglich: Abo ohne Kommentar. +