Issue

Vol. 19 (2027)

Triboelectric E-Skin Patch Decoding Silent Speech for Advanced Human–Machine Interaction and Augmented Virtual Communication

https://doi.org/10.1007/s40820-026-02346-x
Ziwei Huo
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Nuo Xu
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Yixuan Fu
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Rui Gu
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Chao Liu
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Zhong Lin Wang
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China
Qijun Sun
Beijing Key Laboratory of High‑Entropy Energy Materials and Devices, Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing 101400, People’s Republic of China

Corresponding Author: Qijun Sun

sunqijun@binn.cas.cn

Nano-Micro Letters, Vol. 19 (2027), Article Number: 55

Abstract

Silent speech decoding (lip reading) represents an innovative hands-free communication method for individuals with laryngeal and lingual impairments, addressing critical daily communication challenges. Herein, a novel silent speech decoding system is proposed, which employs a self-powered, low-cost, and conformable triboelectric-based E‑skin patch and a well-trained multi-layer convolutional neural network (CNN) model. We systematically characterize the structural principles and electrical performance of the E-skin patch. A comparative analysis of labial kinematics during vocalized and silent speech is also conducted, which demonstrates strong consistency in time–frequency domain characteristics. The utilized CNN model is trained across 15 lexical categories (200 samples per category), achieving a high accuracy of 98.75% and demonstrating excellent recognition performance. Furthermore, by integrating human–machine interaction (HMI) and virtual reality (VR) technologies, we develop three representative applications, including a real-time lip language visualization system, an orofacial-controlled “Feeding Frenzy” interactive game, and a VR communication platform with avatar dialogue. This technology enables barrier-free communication for voiceless populations, demonstrating significant potential in HMI, augmentative communication systems, virtual reality integration, and biomedical rehabilitation.


Highlights:
1 A triboelectric E-skin patch is developed to capture lip/facial muscle movements for silent speech decoding.
2 The proposed four-layer convolutional neural network architecture assists to achieve a record high recognition accuracy of 98.75%.
3 Featured human–machine interaction/virtual reality applications have been demonstrated, including silent speech display system, interactive game, and virtual communication.

Keywords

Triboelectric sensor E‑skin patch Silent speech decoding Human–machine interaction Virtual communication
Huo, Ziwei, Nuo Xu, Yixuan Fu, Rui Gu, Chao Liu, Zhong Lin Wang, and Qijun Sun. 2026. “Triboelectric E-Skin Patch Decoding Silent Speech for Advanced Human–Machine Interaction and Augmented Virtual Communication”. Nano-Micro Letters 19 (October):55. https://doi.org/10.1007/s40820-026-02346-x.
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX
References
  1. H. Zhou, D. Li, X. He, X. Hui, H. Guo et al., Bionic ultra-sensitive self-powered electromechanical sensor for muscle-triggered communication application. Adv. Sci. 8(15), 2101020 (2021). https://doi.org/10.1002/advs.202101020
  2. T. Zheng, M. Gao, Y. Wang, Y. Luo, H. Zhang et al., Ultra-wide frequency response and high-resolution triboelectric acoustic sensors for constructing multifunctional speech aided system. Nano Energy 135, 110640 (2025). https://doi.org/10.1016/j.nanoen.2024.110640
  3. Y. Zhang, W. Chen, B. Luo, H. Liao, L. Liu et al., Wearable triboelectric sensor for deep learning-assisted silent speech recognition in assistive communication. Chem. Eng. J. 524, 169299 (2025). https://doi.org/10.1016/j.cej.2025.169299
  4. R. Zhang, Machine learning-assisted triboelectric nanogenerator-based self-powered sensors. Cell Rep. Phys. Sci. 5(4), 101888 (2024). https://doi.org/10.1016/j.xcrp.2024.101888
  5. L. Zhang, Y. Liu, X. Sun, Z. Wen, Advances in triboelectric nanogenerators in acoustics: energy harvesting and sound sensing. Nano Trends 8, 100064 (2024). https://doi.org/10.1016/j.nwnano.2024.100064
  6. R. Yin, D. Wang, S. Zhao, Z. Lou, G. Shen, Wearable sensors-enabled human–machine interaction systems: from design to application. Adv. Funct. Mater. 31(11), 2008936 (2021). https://doi.org/10.1002/adfm.202008936
  7. C. Yao, S. Liu, Z. Liu, S. Huang, T. Sun et al., Deep learning-enhanced anti-noise triboelectric acoustic sensor for human-machine collaboration in noisy environments. Nat. Commun. 16, 4276 (2025). https://doi.org/10.1038/s41467-025-59523-6
  8. Q. Yang, W. Jin, Q. Zhang, Y. Wei, Z. Guo et al., Mixed-modality speech recognition and interaction using a wearable artificial throat. Nat. Mach. Intell. 5(2), 169–180 (2023). https://doi.org/10.1038/s42256-023-00616-6
  9. J. Yang, J. Chen, Y. Su, Q. Jing, Z. Li et al., Eardrum-inspired active sensors for self-powered cardiovascular system characterization and throat-attached anti-interference voice recognition. Adv. Mater. 27(8), 1316–1326 (2015). https://doi.org/10.1002/adma.201404794
  10. N. Xu, Y. Wang, Z. Huo, J. Yu, J. Yang et al., Flexible and self-powered paper-based artificial synapse for neuromorphic computing and 3d information transmission. Nano Res. Energy 4(4), e9120187 (2025). https://doi.org/10.26599/nre.2025.9120187
  11. L. Xu, S. Zhong, T. Yue, Z. Zhang, X. Lu et al., AIoT-enhanced health management system using soft and stretchable triboelectric sensors for human behavior monitoring. Ecomat 6(5), e12448 (2024). https://doi.org/10.1002/eom2.12448
  12. Z. Wang, Z. Zhao, Q. Tu, C. Yao, Z. Liu et al., Graphene aerogel-based flexible pressure sensor for physiological signal detection and human-machine interaction. Nano-Micro Lett. 18(1), 308 (2026). https://doi.org/10.1007/s40820-026-02109-8
  13. W. Wang, J. Zhu, H. Zhao, F. Yao, Y. Zhang et al., A reconfigurable omnidirectional triboelectric whisker sensor array for versatile human-machine-environment interaction. Nano-Micro Lett. 18(1), 76 (2025). https://doi.org/10.1007/s40820-025-01930-x
  14. J. Wang, Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing. Front. Neurosci. 18, 1449527 (2024). https://doi.org/10.3389/fnins.2024.1449527
  15. H. Tian, X. Li, Y. Wei, S. Ji, Q. Yang et al., Bioinspired dual-channel speech recognition using graphene-based electromyographic and mechanical sensors. Cell Rep. Phys. Sci. 3(10), 101075 (2022). https://doi.org/10.1016/j.xcrp.2022.101075
  16. Z. Huo, Q. Sun, J. Yu, Y. Wei, Y. Wang et al., Neuromorphic devices assisted by machine learning algorithms. Int. J. Extrem. Manuf. 7(4), 042007 (2025). https://doi.org/10.1088/2631-7990/adba1e
  17. W. Qiao, L. Zhou, J. Zhang, D. Liu, Y. Gao et al., A highly-sensitive omnidirectional acoustic sensor for enhanced human–machine interaction. Adv. Mater. 36(48), 2413086 (2024). https://doi.org/10.1002/adma.202413086
  18. T. Sun, B. Feng, J. Huo, Y. Xiao, W. Wang et al., Artificial intelligence meets flexible sensors: emerging smart flexible sensing systems driven by machine learning and artificial synapses. Nano-Micro Lett. 16(1), 14 (2023). https://doi.org/10.1007/s40820-023-01235-x
  19. J.G. Park, B. Kim, J.Y. Song, H.K. Lee, M.C. Kim et al., Shear thickening and charge-storing interlayer-based all-aerosol-sprayed wearable triboelectric sensor for industrial wireless human-machine interfaces. Nano Energy 124, 109444 (2024). https://doi.org/10.1016/j.nanoen.2024.109444
  20. P. Parashar, L.-C. Shen, Y.-H. Lee, M.K. Sharma, B.K. Nahak et al., A highly flexible self-powered triboelectric sensor array for silent speech recognition and swallowing motion analysis. Small 21(36), e70436 (2025). https://doi.org/10.1002/smll.70436
  21. H. Niu, X. Wei, H. Li, F. Yin, W. Wang et al., Micropyramid array bimodal electronic skin for intelligent material and surface shape perception based on capacitive sensing. Adv. Sci. 11(3), 2305528 (2024). https://doi.org/10.1002/advs.202305528
  22. K. Munirathinam, L. Li, A. Shanmugasundaram, J. Park, D.-W. Lee, Air-breakdown triboelectric nanogenerator inspired by transistor architecture for low-force human-machine interfaces. Nano-Micro Lett. 18(1), 251 (2026). https://doi.org/10.1007/s40820-026-02103-0
  23. Y. Lu, H. Tian, J. Cheng, F. Zhu, B. Liu et al., Decoding lip language using triboelectric sensors with deep learning. Nat. Commun. 13, 1401 (2022). https://doi.org/10.1038/s41467-022-29083-0
  24. Y. Lou, M. Li, A. Yu, J. Zhai, Z.L. Wang, From wave energy to electricity: functional design and performance analysis of triboelectric nanogenerators. Nano-Micro Lett. 17(1), 298 (2025). https://doi.org/10.1007/s40820-025-01811-3
  25. S. Liu, T. Fawden, R. Zhu, G.G. Malliaras, M. Bance, A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction. Sci. Adv. 10(26), eado9576 (2024). https://doi.org/10.1126/sciadv.ado9576
  26. K. Tao, J. Yu, J. Zhang, A. Bao, H. Hu et al., Deep-learning enabled active biomimetic multifunctional hydrogel electronic skin. ACS Nano 17(16), 16160–16173 (2023). https://doi.org/10.1021/acsnano.3c05253
  27. Y. Gu, Y. Luo, Q. Guo, W. Yu, P. Li et al., Empowering human-machine interfaces: self-powered hydrogel sensors for flexible and intelligent systems. Adv. Funct. Mater. 36(5), e09085 (2026). https://doi.org/10.1002/adfm.202509085
  28. Y. Liu, C.K. Yiu, Z. Zhao, W. Park, R. Shi et al., Soft, miniaturized, wireless olfactory interface for virtual reality. Nat. Commun. 14(1), 2297 (2023). https://doi.org/10.1038/s41467-023-37678-4
  29. Y. Dong, L. Duan, X. Mao, T. Gu, Y. Gu et al., Exploring human motions for smart wearables: energy conversion, harvesting and self–powered sensing. Nano Energy 143, 111289 (2025). https://doi.org/10.1016/j.nanoen.2025.111289
  30. L. Xie, Y. Zhang, H. Yuan, M. Zhang, X. Zhang et al., Neural Chinese silent speech recognition with facial electromyography. Speech Commun. 171, 103230 (2025). https://doi.org/10.1016/j.specom.2025.103230
  31. J.H. Yeo, C.W. Kim, H. Kim, H. Rha, S. Han et al., Personalized lip reading: adapting to your unique lip movements with vision and language. Proc. AAAI Conf. Artif. Intell. 39(9), 9472–9480 (2025). https://doi.org/10.1609/aaai.v39i9.33026
  32. J. Liu, M. Liu, C. Sun, Z. Lin, Z. Feng et al., Triboelectric hydrophone for underwater detection of low-frequency sounds. Nano Energy 99, 107428 (2022). https://doi.org/10.1016/j.nanoen.2022.107428
  33. L. Luo, C. Liu, R. Gu, M. Chen, Y. Wang et al., Constructing high-performance and versatile liquid–solid triboelectric nanogenerator with inflatable columnar units. Int. J. Extrem. Manuf. 7(1), 015505 (2025). https://doi.org/10.1088/2631-7990/ad88bd
  34. C. Liu, R. Gu, J. Yang, L. Luo, M. Chen et al., A self-powered dual ratchet angle sensing system for digital twins and smart healthcare. Adv. Funct. Mater. 34(42), 2405104 (2024). https://doi.org/10.1002/adfm.202405104
  35. Z. Lin, G. Zhang, X. Xiao, C. Au, Y. Zhou et al., A personalized acoustic interface for wearable human–machine interaction. Adv. Funct. Mater. 32(9), 2109430 (2022). https://doi.org/10.1002/adfm.202109430
  36. Z. Lin, Y. Wang, S. Nie, M. Zou, F. Xu et al., Island-bridge microstructured triboelectric pressure sensor for effective dynamic epidermal pulse monitoring. Soft Sci. 5(4), 47 (2025). https://doi.org/10.20517/ss.2025.53
  37. Y. Xiong, Y. Liu, J. Yang, M. Chen, Z.L. Wang et al., Bioinspired triboelectric-driven multisensory framework with autonomous cross-modal adaptation. eScience 6(2), 100482 (2026). https://doi.org/10.1016/j.esci.2025.100482
  38. E. Cui, P. Wu, F. Wang, S. Xu, D. Yang et al., Dipole effect enhanced liquid stream-current generator. Matter 9(3), 102632 (2026). https://doi.org/10.1016/j.matt.2025.102632
  39. R. Gu, C. Liu, S. Park, N. Xu, D. Kim et al., Triboelectric nanogenerators: from nano energy to extreme manufacturing. Int. J. Extreme Manuf. 8(2), 023001 (2026). https://doi.org/10.1088/2631-7990/ae1fc1
  40. K.-H. Lee, M.-G. Kim, W. Kang, H.-M. Park, Y. Cho et al., Pulse-charging energy storage for triboelectric nanogenerator based on frequency modulation. Nano-Micro Lett. 17(1), 210 (2025). https://doi.org/10.1007/s40820-025-01714-3
  41. T. Liu, M. Zhang, Z. Li, H. Dou, W. Zhang et al., Machine learning-assisted wearable sensing systems for speech recognition and interaction. Nat. Commun. 16, 2363 (2025). https://doi.org/10.1038/s41467-025-57629-5
  42. S. Lin, Y. Guo, X. Zeng, X. Zhou, Y. Zhang et al., TENG-based self-powered silent speech recognition interface: from assistive communication to immersive AR/VR interaction. Nano-Micro Lett. 18(1), 143 (2026). https://doi.org/10.1007/s40820-025-01982-z
  43. F. Wang, P. Wu, X. Chen, E. Cui, T. Liu et al., Crystal boundary engineering-dominated high-performance chitosan-based triboelectric nanogenerator for self-powered breath-activated ammonia sensors. InfoMat 7(7), e12662 (2025). https://doi.org/10.1002/inf2.12662
  44. T. Kim, Y. Shin, K. Kang, K. Kim, G. Kim et al., Ultrathin crystalline-silicon-based strain gauges with deep learning algorithms for silent speech interfaces. Nat. Commun. 13(1), 5815 (2022). https://doi.org/10.1038/s41467-022-33457-9
  45. D. Kam, G. Gwon, Y. Ra, D. Lee, Y.-S. Kim et al., Mechanically guided 4D printing of a triboelectricity-driven acoustic transducer for self-powered and intelligent voice recognition. npj Flex. Electron. 9, 73 (2025). https://doi.org/10.1038/s41528-025-00451-2
  46. R. Gu, L. Wei, N. Xu, Y. Xiong, Q. Sun et al., Machine learning enhanced self-charging power sources. Adv. Funct. Mater. 35(40), 2505719 (2025). https://doi.org/10.1002/adfm.202505719
  47. S. Mu, S. Li, H. Zhao, Z. Wang, X. Xiao et al., A Platypus-inspired electro-mechanosensory finger for remote control and tactile sensing. Nano Energy 116, 108790 (2023). https://doi.org/10.1016/j.nanoen.2023.108790
  48. H. Tan, Y. Zhou, Q. Tao, J. Rosen, S. van Dijken, Bioinspired multisensory neural network with crossmodal integration and recognition. Nat. Commun. 12(1), 1120 (2021). https://doi.org/10.1038/s41467-021-21404-z
  49. Y. Dai, Y. Li, S. Xuan, Y. Dai, T. Xu, H. Yu, Triboelectric nanogenerator-based flexible acoustic sensor for speech recognition. ACS Appl. Mater. Interfaces 17(7), 11117–11125 (2025). https://doi.org/10.1021/acsami.4c21563
  50. J.P. Lee, H. Jang, Y. Jang, H. Song, S. Lee et al., Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface. Nat. Commun. 15(1), 530 (2024). https://doi.org/10.1038/s41467-023-44673-2
  51. Y. Liu, Y. Xiong, J. Yang, L. Cheng, Z.L. Wang et al., Wearable and self-powered bi-directional interaction and feedback system for immersive pneumatic haptic experiences. Nano Energy 155, 112043 (2026). https://doi.org/10.1016/j.nanoen.2026.112043
Read More
Author biographies are not available.

Most read articles by the same author(s)

1 2 > >>