About Me
I am a Research Scientist at NVIDIA, working on LLM post-training with a focus on agentic coding and terminal-use agents, and I contribute to the Nemotron model family.
I received my Ph.D. from the Department of Computer Science and Engineering at the Hong Kong University of Science and Technology (HKUST), co-supervised by Prof. Tong Zhang and Prof. Xiaofang Zhou. Before that, I obtained my B.Eng. in Computer Engineering from the University of Hong Kong (HKU) with first class honours. During my Ph.D., I worked on multimodal large language models (visual perception, preference alignment, hallucination and safety), data-centric AI and optimization, and was honored to receive the Apple Scholarship in AI/ML (2024).
I am always happy to discuss research and collaboration — feel free to reach out by email. Outside of work, I enjoy basketball, jumping rope and jogging. I am also a music lover — here is a clip of me performing on stage.
- LLM Post-Training
- Agentic Coding
- Reinforcement Learning
- Multimodal LLMs
- Data-Centric AI
News
- 2026.06Released Nemotron 3 Ultra, an open hybrid Mamba-Transformer MoE model for agentic reasoning.
- 2026.04Released Nemotron 3 Super.
- 2026.03Released Nemotron-Cascade 2: post-training LLMs with cascade RL and multi-domain on-policy distillation.
- 2026.02New paper: On Data Engineering for Scaling LLM Terminal Capabilities.
- 2024.12Image Textualization and LISA presented at NeurIPS 2024.
- 2024.09MLLM-Protector accepted to EMNLP 2024.
- 2024.07Bootstrapped Preference Optimization (BPO) accepted to ECCV 2024.
- 2024.03Received the Apple Scholarship in AI/ML.
Selected Publications
* equal contribution. Highlighted entries are (co-)first-authored. Full list on Google Scholar.
2026
On Data Engineering for Scaling LLM Terminal Capabilities
Renjie Pi*, Grace Lam*, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, Wei Ping
arXiv preprint, 2026
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation
Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, Renjie Pi, Grace Lam, Nayeon Lee, Alexander Bukharin, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
arXiv preprint, 2026
Nemotron 3 Super / Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic Reasoning
NVIDIA (Renjie Pi as contributor)
Technical reports, 2026
2025
Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
Renjie Pi*, Kehao Miao*, Li Peihang, Runtao Liu, Jiahui Gao, Jipeng Zhang, Xiaofang Zhou
arXiv preprint, 2025
MR. Judge: Multimodal Reasoner as a Judge
Renjie Pi, Felix Bai, Qibin Chen, Simon Wang, Jiulong Shan, Kieran Liu, Meng Cao
arXiv preprint, 2025
2024
Personalized Visual Instruction Tuning
Renjie Pi*, Jianshu Zhang*, Tianyang Han, Jipeng Zhang, Rui Pan, Tong Zhang
arXiv preprint, 2024
Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
Renjie Pi*, Jianshu Zhang*, Jipeng Zhang, Rui Pan, Zhekai Chen, Tong Zhang
NeurIPS 2024, Datasets and Benchmarks Track
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
Renjie Pi*, Tianyang Han*, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, Tong Zhang
ECCV 2024
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
Renjie Pi*, Tianyang Han*, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
EMNLP 2024
PerceptionGPT: Effectively Fusing Visual Perception into LLM
Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, Tong Zhang
CVPR 2024
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang
NeurIPS 2024
2023
DetGPT: Detect What You Need via Reasoning
Renjie Pi*, Jiahui Gao*, Shizhe Diao*, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang
EMNLP 2023
DynaFed: Tackling Client Data Heterogeneity with Global Dynamics
Renjie Pi*, Weizhong Zhang*, Yueqi Xie*, Jiahui Gao, Xiaoyu Wang, Sunghun Kim, Qifeng Chen
CVPR 2023
A Holistic View of Noise Transition Matrix in Deep Learning and Beyond
Yong Lin*, Renjie Pi*, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, Bo Han
ICLR 2023 Spotlight
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
Jiahui Gao*, Renjie Pi*, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong
ICLR 2023 Spotlight
Earlier and other publications
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, Tong Zhang
ACL 2025
GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis
Yueqi Xie, Minghong Fang, Renjie Pi, Neil Gong
ACL 2024
CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration
Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lanqing Hong, Lingpeng Kong, Xin Jiang, Zhenguo Li
COLM 2024
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, Dan Xu
CVPR 2024
Measure the Predictive Heterogeneity
Jiashuo Liu, Jiayun Wu, Renjie Pi, Renzhe Xu, Xingxuan Zhang, Bo Li, Peng Cui
ICLR 2023
Probabilistic Bilevel Coreset Selection
Xiao Zhou*, Renjie Pi*, Weizhong Zhang*, Yong Lin, Tong Zhang
ICML 2022
Model Agnostic Sample Reweighting for Out-of-Distribution Learning
Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, Tong Zhang
ICML 2022
G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation
Lewei Yao*, Renjie Pi*, Hang Xu, Wei Zhang, Zhenguo Li, Tong Zhang
ICCV 2021
Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
Lewei Yao*, Renjie Pi*, Hang Xu, Wei Zhang, Zhenguo Li, Tong Zhang
CVPR 2021
Education & Experience
- PresentNVIDIA — Research Scientist
- 2021 – 2025Hong Kong University of Science and Technology — Ph.D. in Computer Science and Engineering
- 2016 – 2021University of Hong Kong — B.Eng. in Computer Engineering, First Class Honours
- 2018.09 – 2019.01University of Maryland, College Park — Exchange Student
- 2017.06 – 2017.07Peking University — Visiting Student
Awards & Honors
- Apple Scholarship in AI/ML, 2024
- Postgraduate Studentship, HKUST
- S Y King Memorial Scholarship in Electrical and Electronic Engineering, HKU
- Entrance Scholarship in Electrical and Electronic Engineering, HKU
- HKU Foundation Entrance Scholarship