01 introduction

Multimodal Communicative Behaviors

• Verbal
• Vocal
• Visual

Prior Research on Multimodal

• behavioral
• computational
• interaction
• deep learning

Two core challenges

• Representation
• Joint representations
• Coordinated representations
• Alignment -- The very core
• Can be seen as a specialization of multi-view learning
• Explicit Alignment
• Implicit Alignment
• Translations
• Translating from one modality to another
• Example-based
• Model-driven
• Fusion
• Model-Agnostic Approaches
  • Early Fusion
  • Late Fusion
• Model-Based (Intermediate) Approaches
  • Deep neural networks
  • Kernel-based methods
  • Graph models
• Co-Learning
• Parallel
• Weakly-Parallel

Real world tasks tackled by MMML

Affective Computing