The handbook of multimodal-multisensor interfaces. Volume 2, Signal processing, architectures, and detection of emotion and cognition
Başlık:
The handbook of multimodal-multisensor interfaces. Volume 2, Signal processing, architectures, and detection of emotion and cognition
Yazar:
Oviatt, Sharon, author.
ISBN:
9781970001693
Yazar:
Edisyon:
First edition.
Fiziksel Niteleme:
1 PDF (xxiii, 515 pages) : illustrations.
Seri:
ACM books, #21
ACM books ; #21.
İçindekiler:
Introduction: Trends in intelligent multimodal-multisensorial interfaces: cognition, emotion, social signals, deep learning, and more --
Part I. Multimodal signal processing and architectures
1. Challenges and applications in multimodal machine learning / Tadas Baltrusaitis, Chaitanya Ahuja, Louis-Philippe Morency -- 1.1 Introduction -- 1.2 Multimodal Applications -- 1.3 Multimodal Representations -- 1.4 Co-learning -- 1.5 Conclusion -- Focus questions -- References --
2. Classifying multimodal data / Ethem Alpaydin -- 2.1 Introduction -- 2.2 Classifying multimodal data -- 2.3 Early, late, and intermediate integration -- 2.4 Multiple kernel learning -- 2.5 Multimodal deep learning -- 2.6 Conclusions and future work -- Acknowledgments -- Focus questions -- References --
3. Learning for multimodal and affect-sensitive interfaces / Yannis Panagakis, Ognjen Rudovic, Maja Pantic -- 3.1 Introduction -- 3.2 Correlation analysis methods -- 3.3 Temporal modeling of facial expressions -- 3.4 Context dependency -- 3.5 Model adaptation -- 3.6 Conclusion -- Focus questions -- References --
4. Deep learning for multisensorial and multimodal interaction / Gil Keren, Amr El-desoky Mousa, Olivier Pietquin, Stefanos Zafeiriou, Björn Schuller -- 4.1 Introduction -- 4.2 Fusion models -- 4.3 Encoder-decoder models -- 4.4 Multimodal embedding models -- 4.5 Perspectives -- Focus questions -- References --
Part II. Multimodal processing of social and emotional states --
5. Multimodal user state and trait recognition: an overview / Björn Schuller -- 5.1 Introduction -- 5.2 Modeling -- 5.3 An overview on attempted multimodal stait and trait recognition -- 5.4 Architectures -- 5.5 A modern architecture perspective -- 5.6 Modalities -- 5.7 Walk-through of an example state -- 5.8 Emerging trends and future directions -- Focus questions -- References --
6. Multimodal-multisensor affect detection / Sidney K. D'Mello, Nigel Bosch, Huili Chen -- 6.1 Introduction -- 6.2 Background from affective sciences -- 6.3 Modality fusion for multimodal-multisensor affect detection -- 6.4 Walk-throughs of sample multisensor-multimodal affect detection systems -- 6.5 General trends and state of the art in multisensor-multimodal affect detection -- 6.6 Discussion -- Acknowledgments -- Focus questions -- References --
7. Multimodal analysis of social signals / Alessandro Vinciarelli, Anna Esposito -- 7.1 Introduction -- 7.2 Multimodal communication in life and human sciences -- 7.3 Multimodal analysis of social signals -- 7.4 Next steps -- 7.5 Conclusions -- Focus questions -- References --
8. Real-time sensing of affect and social signals in a multimodal framework: a practical approach / Johannes Wagner, Elisabeth Andre -- 8.1 Introduction -- 8.2 Database collection -- 8.3 Multimodal fusion -- 8.4 Online recognition -- 8.5 Requirements for a multimodal framework -- 8.6 The social signal interpretation framework -- 8.7 Conclusion -- Focus questions -- References --
9. How do users perceive multimodal expressions of affects? / Jean-Claude Martin, Celine Clavel, Matthieu Courgeon, Mehdi Ammi, Michel-Ange Amorim, Yacine Tsalamlal, Yoren Gaffary -- 9.1 Introduction -- 9.2 Emotions and their expressions -- 9.3 How humans perceive combinations of expressions of affects in several modalities -- 9.4 Impact of context on the perception of expressions of affects -- 9.5 Conclusion -- Focus Questions -- References --
Part III. Multimodal processing of cognitive states --
10. Multimodal behavioral and physiological signals as indicators of cognitive load / Jianlong Zhou, Kun Yu, Fang Chen, Yang Wang, Syed Z. Arshad -- 10.1 Introduction -- 10.2 State-of-the-art -- 10.3 Behavioral measures for cognitive load -- 10.4 Physiological measures for cognitive load -- 10.5 Multimodal signals and data fusion -- 10.6 Conclusion -- Funding -- Focus questions -- References --
11. Multimodal learning analytics: assessing learners' mental state during the process of learning / Sharon Oviatt, Joseph Grafsgaard, Lei Chen, Xavier Ochoa --11.1 Introduction -- 11.2 What is multimodal learning analytics? -- 11.3 What data resources are available on multimodal learning analytics? -- 11.4 What are the main themes from research findings on multimodal learning analytics? -- 11.5 What is the theoretical basis of multimodal learning analytics? -- 11.6 What are the main challenges and limitations of multimodal learning analytics? -- 11.7 Conclusions and future directions -- Focus questions -- References --
12. Multimodal assessment of depression from behavioral signals / Jeffrey F. Cohn, Nicholas Cummins, Julien Epps, Roland Goecke, Jyoti Joshi, Stefan Scherer -- 12.1 Introduction -- 12.2 Depression -- 12.3 Multimodal behavioral signal processing systems -- 12.4 Facial analysis -- 12.5 Speech analysis -- 12.6 Body movement and other behavior analysis -- 12.7 Analysis using other sensor signals -- 12.8 Multimodal fusion -- 12.9 Implementation-related considerations and elicitation approaches -- 12.10 Conclusion and current challenges -- Acknowledgments -- Focus questions -- References --
13. Multimodal deception detection / Mihai Burzo, Mohamed Abouelenien, Veronica Perez-Rosas, Rada Mihalcea -- 13.1 Introduction and motivation -- 13.2 Deception detection with individual modalities -- 13.3 Deception detection with multiple modalities -- 13.4 The way forward -- Acknowledgments -- Focus questions -- References --
Part IV. Multidisciplinary challenge topic --
14. Perspectives on predictive power of multimodal deep learning: surprises and future directions / Samy Bengio, Li Deng, Louis-Philippe Morency, Björn Schuller -- 14.1 Deep learning as catalyst for scientific discovery -- 14.2 Deep learning in relation to conventional machine learning -- 14.3 Expected surprises of deep learning -- 14.4 The future of deep learning -- 14.5 Responsibility in deep learning -- 14.6 Conclusion -- References --
Index -- Biographies -- Volume 2 Glossary.
Özet:
The content of this handbook is most appropriate for graduate students and of primary interest to students studying computer science and information technology, human-computer interfaces, mobile and ubiquitous interfaces, affective and behavioral computing, machine learning, and related multidisciplinary majors. When teaching graduate classes with this book, whether in a quarter- or semester-long course, we recommend initially requiring that students spend two weeks reading the introductory textbook, The Paradigm Shift to Multimodality in Contemporary Interfaces (Morgan & Claypool Publishers, Human-Centered Interfaces Synthesis Series, 2015). With this orientation, a graduate class providing an overview of multimodal-multisensor interfaces then could select chapters from the current handbook, distributed across topics in the different sections.
Tür:
Elektronik Erişim:
Abstract with links to full text http://dx.doi.org/10.1145/3107990