2010-2019年 西北工业大学 本科-博士

2023年至今，304永利集团，准聘副教授

2020年至2023，304永利集团，准聘助理教授

2019-2020年，百度研究院，人工智能研究员

机器多模态感知与学习：以大脑的多通道知觉为背景，挖掘并探究多模态信息（如图像、声音、触觉、力觉等）在机器（人）感知、交互与学习等方向的潜在问题与方法，让机器具备『多模态智能』。部分研究介绍请观看B站视频（https://www.bilibili.com/video/BV1DK4y1P7Ep?p=2）。

GeWu-Lab实验室网站: https://gewu-lab.github.io/

对客观存在保持好奇心，自驱，刻苦，以做有趣、有温度、有价值的研究为目标！

更多关于实验室介绍，请参见知乎文章：https://zhuanlan.zhihu.com/p/1908530147998565113

2023级多名硕士生（冯若轩，杜恒辉等），已在机器人触觉感知交互，多模态统一学习上形成若干代表性工作，如AnyTouch系列，Crab系列等，曾于北京智源人工智能研究院和腾讯公司等多家研究机构与公司实习。

2022级多名直博生（夏文科，杨泽群等），已在机器人交互学习、多模态交互理论上形成若干代表性工作，曾于上海AI Lab、字节Seed、北京人形机器人创新中心等多家研究机构与公司实习。

2021级直博生卫雅珂（曾赴CMU联培），指导发表多篇CCF-A类论文（如T-PAMI, CVPR Oral文章），荣获国家奖学金，2024年度百度奖学金（全球仅10人，奖金20万）。

2020级博士生与硕士生，发表多篇CCF-A类会议Oral文章。

多名访问同学（RA）于实验室在本科大二、大三阶段，发表CVPR、ICML等一作论文，如访问员工2017级上交本科生钱锐，指导发表多篇CCF-A类论文，后于CUHK MMLab攻读博士学位；访问员工邓安东（上交），指导发表/在投多篇CCF-A类论文，后赴UCF攻读博士学位。

已毕业同学（去向）：

李光耀（2020级博士）：清华大学

许一鑫（2020级硕士）：航天一院

彭小康（2020级硕士）：中央办公厅

本科生课程：《人工智能与Python程序设计》，2020-2026

本科生课程：《人工智能与机器人》，2024-2026

研究生课程：《模式识别与计算机视觉》，2020-2026

北京市自然科学基金面上项目（2026-2028）：多模态物体交互方法研究，主持

智源学者项目（2025-2026）：面向物体操纵的视听触高效协同感知研究，主持

CCF-智谱大模型创新基金（2024-2025）：基于统一学习范式的跨模态信息融合、推理与生成研究，主持

CCF-腾讯犀牛鸟基金（2024-2025）：基于扩散模型的音频视频协同编辑与生成方法研究，主持

中国科协青年人才托举工程项目（2022-2024），主持

国家自然科学基金青年科学基金项目（2022-2024）：自然场景下机器的视听感知与学习，主持

百度研究院（2021-2022）：跨模态迁移学习场景下的可解释性研究，主持

腾讯AI Lab犀牛鸟专项研究计划（2021-2022）：动态视音场景下多说话人跟踪与日志方法研究，主持

304永利集团官网新教师启动金项目（2021-2022）：面向视听信息的多模态认知计算，主持

2026

APPO: Attention-guided Perception Policy Optimization for Video Reasoning

Henghui Du, Chang Zhou, Xi Chen, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

Information-Theoretic Decomposition for Multimodal Interaction Learning

Zequn Yang, Yake Wei, Haotian Ni, Zhihao Xu, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

Wenbo Yu, Wenke Xia, Weitao Zhang, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

When would Vision-Proprioception Policies Fail in Robotic Manipulation?

Jingxian Lu*, Wenke Xia*, Yuxuan Wu, Zhiwu Lu, Di Hu

International Conference on Learning Representations (ICLR)

---

AnyTouch 2: General Optical Tactile Representation Learning for Dynamic Tactile Perception

Ruoxuan Feng, Yuxuan Zhou, Siyu Mei, Dongzhan Zhou, Pengwei Wang, Shaowei Cui, Bin Fang, Guocai Yao, Di Hu

International Conference on Learning Representations (ICLR)

---

2025

MokA: Multimodal Low-Rank Adaptation for MLLMs

Yake Wei, Yu Miao, Dongzhan Zhou, Di Hu

Advances in Neural Information Processing Systems (NeurIPS), Oral Presentation

---

Human-assisted Robotic Policy Refinement via Action Preference Optimization

Wenke Xia, Yichu Yang, Hongtao Wu, Xiao Ma, Tao Kong, Di Hu

Advances in Neural Information Processing Systems (NeurIPS)

---

Efficient Quantification of Multimodal Interaction at Sample Level

Zequn Yang, Hongfa Wang, Di Hu

International Conference on Machine Learning (ICML)

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer

Haotian Ni, Yake Wei, Hang Liu, Gong Chen, Chong Peng, Hao Lin, Di Hu

International Conference on Machine Learning (ICML)

---

Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition

Chengxiang Huang, Yake Wei, Zequn Yang, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction

Wenke Xia, Ruoxuan Feng, Dong Wang, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception

Ruotian Peng, Haiying He, Yake Wei, Yandong Wen, Di Hu

Computer Vision and Pattern Recognition (CVPR)

---

AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

Ruoxuan Feng, Jiangyu Hu, Wenke Xia, Tianci Gao, Ao Shen, Yuhao Sun, Bin Fang*, Di Hu

International Conference on Learning Representations (ICLR)

---

2024

On-the-fly Modulation for Balanced Multimodal Learning

Yake Wei, Di Hu, Henghui Du, and Ji-Rong Wen

IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI)

---

Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation (Oral)

Ruoxuan Feng, Di Hu, Wenke Ma, Xuelong Li

Conference on Robot Learning (CoRL)

---

KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance

Jingxian Lu, Wenke Xia, Dong Wang, Zhigang Wang, Bin Zhao, Di Hu, and Xuelong Li

Conference on Robot Learning (CoRL)

---

Diagnosing and Re-learning for Balanced Multimodal Learning

Yake Wei, Siwei Li, Ruoxuan Feng, and Di Hu

European Conference on Computer Vision (ECCV)

---

Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation

Juncheng Ma, Peiwen Sun, Yaoting Wang, and Di Hu

European Conference on Computer Vision (ECCV)

---

Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes

Yaoting Wang†, Peiwen Sun†, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu

European Conference on Computer Vision (ECCV)

---

Can Textual Semantics Mitigate Sounding Object Segmentation Preference?

Yaoting Wang†, Peiwen Sun†, Yuanchao Li, Honggang Zhang, and Di Hu

European Conference on Computer Vision (ECCV)

---

Boosting Audio Visual Question Answering via Key Semantic-Aware Cues

ACM Conference on Multimedia (ACMMM)

Guangyao Li, HenghuiDu, and Di Hu

---

Unveiling and Mitigating Bias in Audio Visual Segmentation (Oral)

Peiwen Sun, Honggang Zhang, and Di Hu

ACM Conference on Multimedia (ACMMM), Oral Presentation

---

Depth Helps: Improving Pre-trained RGB-based Policy with Depth Information Injection

Xincheng Pang†, Wenke Xia†, Zhigang Wang, Bin Zhao, Di Hu, Dong Wang, and Xuelong Li

International Conference on Intelligent Robots and Systems (IROS)

---

Learning Manipulation by Predicting Interaction

Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, Heming Cui, Bin Zhao, Xuelong Li, Yu Qiao, and Hongyang Li

Robotics: Science and Systems Conference (RSS)

---

MMPareto: Innocent Uni-modal Assistance for Enhanced Multi-modal Learning

Yake Wei, Di Hu

International Conference on Machine Learning (ICML)

---

Enhancing Multi-modal Cooperation via Fine-grained Modality Valuation

Yake Wei , Ruoxuan Feng , Zihe Wang , Di Hu

Computer Vision and Pattern Recognition(CVPR)

---

Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Zequn Yang , Yake Wei , Ce Liang , Di Hu

The Twelfth International Conference on Learning Representations (ICLR)

---

SphereDiffusion: Spherical Geometry-aware Distortion Resilient Diffusion Model

Tao Wu , Xuewei Li , Zhongang Qi , Di Hu , Xintao Wang , Ying Shan , Xi Li

The 38th Annual AAAI Conference on Artificial Intelligence

---

Prompting Segmentation with Sound is Generalizable Audio-Visual Source Localizer

Yaoting Wang* , Weisong Liu* , Guangyao Li , Jian Ding , Di Hu , Xi Li

The 38th Annual AAAI Conference on Artificial Intelligence

---

Geometric-Inspired Graph-based Incomplete Multi-view Clustering

Zequn Yang , Han Zhang , Yake Wei , Zheng Wang , Feiping Nie , Di Hu

Pattern Recognition

---

Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs

Wenke Xia , Dong Wang , Xincheng Pang , Zhigang Wang , Bin Zhao , Di Hu , Xuelong Li

IEEE International Conference on Robotics and Automation (ICRA)

---

2023

TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World

Hongpeng Lin* , Ludan Ruan* , Wenke Xia* , Peiyu Liu , Jingyuan Wen , Yixin Xu , Di Hu , Ruihua Song , Wayne Xin Zhao , Qin Jin , Zhiwu Lu

ACM Multimedia(ACM MM)

---

Progressive Spatio-temporal Perception for Audio-Visual Question Answering

Guangyao Li , Wenxuan Hou , Di Hu

ACM Multimedia(ACM MM)

---

Towards Inadequately Pre-trained Models in Transfer Learning

Andong Deng , Xingjian Li , Di Hu , Tianyang Wang , Haoyi Xiong , Chengzhong Xu

International Conference on Computer Vision(ICCV)

---

Balanced Audiovisual Dataset for Imbalance Analysis

Wenke Xia* , Xu Zhao* , Xincheng Pang , Changqing Zhang , Di Hu

Computer Vision and Pattern Recognition(CVPR) Workshop

---

Multi-Scale Attention for Audio Question Answering

Guangyao Li , Yixin Xu , Di Hu

Interspeech, Oral Presentation

---

Supervised Knowledge May Hurt Novel Class Discovery Performance

ZiYun Li , Jona Otholt , Ben Dai , Di Hu , Christoph Meinel , Haojin Yang

Transactions on Machine Learning Research(TMLR)

---

Revisiting Pre-training in Audio-Visual Learning

Ruoxuan Feng , Wenke Xia , Di Hu

arXiv:2302.03533

---

MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

Ruize Xu , Ruoxuan Feng , Shi-xiong Zhang , Di Hu

ICASSP

---

2022

SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance

Xinchi Zhou, Dongzhan Zhou, Wanli Ouyang, Hang Zhou, Di Hu

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

---

Exploiting Visual Context Semantics for Sound Source Localization

Xinchi Zhou, Dongzhan Zhou, Di Hu, Hang Zhou, Wanli Ouyang

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

---

Self-supervised Learning for Heterogeneous Audiovisual Scene Analysis

Di Hu, Zheng Wang, Feiping Nie, Rong Wang, Xuelong Li

TMM

---

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Guangyao Li*, Yake Wei*, Yapeng Tian*, Chenliang Xu, Ji-Rong Wen, Di Hu

CVPR, Oral Presentation

---

Balanced Multimodal Learning via On-the-fly Gradient Modulation

Xiaokang Peng*, Yake Wei*, Andong Deng, Dong Wang, Di Hu

CVPR, Oral Presentation

---

SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation

Dongzhan Zhou, Xinchi Zhou, Di Hu, Hang Zhou, Lei Bai, Ziwei Liu, Wanli Ouyang

AAAI

下载：

---

Visual Sound Localization in-the-Wild by Cross-Modal Interference Erasing

Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, Xiaowei Zhou

AAAI

---

2021

Class-aware Sounding Objects Localization via Audiovisual Correspondence

Di Hu, Yake Wei, Rui Qian, Weiyao Lin, Ruihua Song, Ji-Rong Wen

TPAMI

---

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

Yapeng Tian, Di Hu, Chenliang Xu

CVPR

---

Unsupervised Multi-Source Domain Adaptation for Person Re-Identification

Zechen Bai, Zhigang Wang, Jian Wang, Di Hu, Errui Ding

CVPR

---

Temporal Relational Modeling with Self-Supervision for Action Segmentation

Dong Wang, Di Hu, Xingjian Li, Dejing Dou

AAAI

---

2020

Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching

Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, Dejing Dou

NeurIPS

---

A Two-Stage Framework for Multiple Sound-Source Localization

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, Weiyao Lin

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2020.

---

Co-Learn Sounding Object Visual Grounding and Visually Indicated Sound Separation in A Cycle

Yapeng Tian, Di Hu, Chenliang Xu

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2020.

---

Does Ambient Sound Help? - Audiovisual Crowd Counting

Di Hu, LichaoMou, Qingzhong Wang, Junyu Gao, Yuansheng Hua, Dejing Dou, and Xiaoxiang Zhu

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2020.

---

Heterogeneous Scene Analysis via Self-supervised Audiovisual Learning

Di Hu, Zheng Wang, HaoyiXiong, Dong Wang, FeipingNie, and Dejing Dou

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2020.

---

Multiple Sound Sources Localization from Coarse to Fine

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin

In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

---

Cross-Task Transfer for Multimodal Aerial Scene Recognition

Di Hu, Xuhong Li, LichaoMou, Pu Jin, Dong Chen, Liping Jing, Xiaoxiang Zhu, and Dejing Dou

In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

---

2019

Dense Multimodal Fusion for Hierarchically Joint Representation

Di Hu, Chengze Wang, FeipingNie, and Xuelong Li

In Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.

---

Listen to the Image

Di Hu, Dong Wang, FeipingNie, Qi Wang, and Xuelong Li

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. (CCF A)

---

Deep Multimodal Clustering for Unsupervised Audiovisual Learning

Di Hu, FeipingNie, and Xuelong Li

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. (CCF A)

---

Deep Linear Discriminant Analysis Hashing

Di Hu, FeipingNie, and Xuelong Li

Sci Sin Inform, 2019. (CCF A)

---

2018

Deep Binary Reconstruction for Cross-modal Hashing

Di Hu, FeipingNie, and Xuelong Li

IEEE Trans. Multimedia (TMM), 2018.

---

Discrete Spectral Hashing for Efficient Similarity Retrieval

Di Hu, FeipingNie, and Xuelong Li

IEEE Trans. Image Processing (TIP), 2018. (CCF A)

---

2017

Large Graph Hashing with Spectral Rotation

Xuelong Li, Di Hu, and FeipingNie

In Proceedings of the AAAIConferenceonArtificialIntelligence (AAAI), 2017. (CCF A)

---

Deep Binary Reconstruction for Cross-modal Hashing

Xuelong Li, Di Hu, and FeipingNie

In Proceedings of the ACM Conference on Multimedia (ACMMM), 2017. (CCF A)

---

Image2song: Song Retrieval via Bridging Image Content and Lyric Words

Xuelong Li, Di Hu, and Xiaoqiang Lu

In Proceedings of the IEEE Conference on Computer Vision (ICCV), 2017. (CCF A)

---

2016

Temporal Multimodal Learning in Audiovisual Speech Recognition

Di Hu, Xuelong Li, and Xiaoqiang Lu

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. (CCF A)

---

Multimodal Learning via Exploring Deep Semantic Similarity

Di Hu, Xiaoqiang Lu, and Xuelong Li

In Proceedings of the ACM Conference on Multimedia (ACMMM), 2016. (CCF A)

2025 智源学者

2025 世界人形机器人运动会场景赛亚军

2025 微软铸星学者

2023 2022年度吴文俊人工智能优秀青年奖

2021 中国科协青年人才托举工程项目

2021 荣获陕西省优秀博士论文奖

2020 荣获中国人工智能学会优秀博士论文奖

2019 入选百度『AIDU』全球顶尖人工智能人才计划

2019 荣获ACM XI’AN优秀博士论文奖（共2人）

2019 入选CVPR Doctoral Consortium博士生论坛（大陆共4人）

2018 荣获国家留学基金委赴卡内基梅隆大学联合培养学金

期刊审稿人: TPAMI, TNNLS, TIP, TKDE, TCSVT, TMM, etc.

会议高级程序委员: AAAI 2023-2026， IJCAI 2023-2026，ICML 2026，ICLR 2026

会议程序委员: NeurIPS 2020-2023, CVPR 2018 2020-2024, ICCV 2019-2023, ECCV2020, ICML 2021-2023, AAAI 2018 2020-2022, ICLR 2021-2024

联合组织者:

CVPR 2021 Tutorial on Audio-visual Scene Understanding

WACV 2021 Tutorial on Audio-visual Scene Understanding

ICDM 2019 Tutorial on Automated Deep Learning: Theory, Algorithms, Platforms, and Applications

邮箱：dihu[at]ruc.edu.cn

个人网页：https://gewu-lab.github.io/