Who We Are

The Nanoscale Integrated Circuits and System Lab, Energy Efficient Computing Group (NICS-EFC) in the Department of Electronic Engineering at Tsinghua University is led by Professor Yu Wang. The Efficient Algorithm Team (EffAlg) in the NICS-EFC group is led by Research Assistant Professor Xuefei Ning. Our team has an in-depth academic collaboration with Infinigence-AI, and fellows from many institutions including SJTU, MSR, HKU, and so on.

Our current research primarily focuses on efficient deep learning, including model compression, architecture design, system co-optimization, and other techniques. Our work targets several application domains, including language generative models (i.e., LLMs), vision generative models, vision understanding models and so on. Most of our projects are open sourced at the thu-nics GitHub organization (most efficient DL projects) or the imagination-research GitHub organization (some efficient DL projects and projects for broader topics; These projects are co-advising work with Dr. Zinan Lin from MSR).

Our group welcomes all kinds of collaborations, and is continuously recruiting visiting students and engineers who are interested in efficient deep learning. If you're interested in collaborations or visiting student opportunities, email Xuefei or Prof. Yu Wang.

News

• 2024/07/01

  Our paper Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs is public on arXiv. We conduct expert pruning of MoE models and use weight merging to inherit the knowledge from the retrained and discarded experts. This is a novel and gradient-free way to inherit the knowledge during model compression. The code will soon be available at code.
• 2024/06/24

  Our paper Can LLMs Learn by Teaching? A Preliminary Study is public on arXiv. We make the first attempt in adapting the "learning by teach" strategy in education into LLMs and see if the contemporary LLMs are ready to learn by teach to improve the outcome or ability for mathematical reasoning and code synthesis. The results show some promise. We also provide a roadmap for future research. Welcome to check the code.
• 2024/06/24

  Our paper MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression is public on arXiv. MoA compresses attention in LLMs, so that they can compute short attention, but remember long context. It achieves 5.5-6.7x faster throughput than dense FlashAttention2, improving retrieval accuracy by 1.5-7.1x compared to uniform sparse attention. Welcome to check the code.
• 2024/06/14

  Our paper DiTFastAttn: Attention Compression for Diffusion Transformer Models is public on arXiv. We design three training-free techniques to compress the attention operation, which is the efficiency bottleneck when generating large-resolution images. For example, applying DiTFastAttn to PixArt-Sigma-XL can reduce the FLOPs and latency of attention computation by 88% and 37% when generating 2Kx2K images. Welcome to check the code.
• 2024/06/02

  Our two papers are public on arXiv. In the first paper, we propose MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization that successfully tackles the challenging few-step text-to-image diffusion model quantization. With negligible visual quality degradation and content change, MixDQ could achieve W4A8, equivalent to 3.4x memory compression and 1.5x latency speedup. Check our code and project page. The second paper is ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation, a quantization method specialized for the transformer-based video & image diffusion models. For popular large-scale models (e.g., open-sora, Latte, Pixart) for the video and image generation task, ViDiT-Q could achieve W8A8 quantization without metric degradation, and W4A8 without notable visual quality degradation. Check our code and project page. Update 07/01: MixDQ is accepted by ECCV'24!
• 2024/04/23

  Our survey A Survey on Efficient Inference for Large Language Models is public on arXiv. Any discussions and suggestions are welcome!
• 2024/04/05

  Our paper Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better is public on arXiv. This work proposes a method, Linear Combination of Saved Checkpoints (LCSC). LCSC uses gradient-free search-based checkpoint combination to obtain the final weights, achieving significant training speedups (23x on CIFAR-10 and 15x on ImageNet-64) compared to full gradient-based training. LCSC can be used to enhance pre-trained models with a small cost. Check our code.
• 2024/05/02

  1 paper, Evaluating Quantized Large Language Models, is accepted by ICML'24. This work evaluates 11 LLM families, different tasks (including emergent abilities, dialogue, long-context tasks, and so on), and different tensor types (Weight, Weight-Activation, Key-Value Cache). We provide quantative suggestions and qualitative insights on the quantization. Practitioners could benefit from this work with a full scope of quantization suggestions.
• 2024/02/27

  1 paper, FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models, is accepted by CVPR'24. This work is on selecting a compact data subset to evaluate text-to-image Diffusion models.
• 2024/02/09

  Our paper on long-context benchmark, LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K, is public on arXiv. Check the code.
• 2024/01/17

  2 papers are acccepted by ICLR'24. One is Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation, which accelerates LLM generation by letting the LLM itself to plan and generate segments in parallel, achieving ~2x speed-ups; Another is A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models, which summarizes the sampling strategies for diffusion and search for the best sampling strategy.

Efficient DL Projects

Technique

  Algorithm-level   Model-level (Quantization)   Model-level (Sparsification)   Model-level (Structure Optimization)   System-level

Target

  Efficient Inference   Efficient Training   Efficient Optimization Process

Domain

  Language   Vision Generation   Vision Recognition

ArXiv 2024

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
Model-level (Pruning) | Efficient Inference | Language Paper Code

ArXiv 2024

Can LLMs Learn by Teaching? A Preliminary Study
| | Language Paper Code

ArXiv 2024

MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression
Model-level (Sparsification) | Efficient Inference | Language Paper Code

ArXiv 2024

DiTFastAttn: Attention Compression for Diffusion Transformer Models
Model-level (Sparsification), Model-level (Structure Optimization) | Efficient Inference | Vision Generation Paper Code

ArXiv 2024

ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation
Model-level (Quantization) | Efficient Inference | Vision Generation Paper

ArXiv 2024

DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
Model-level (Structure Optimization) | Efficient Inference | Vision Generation Paper Code

ArXiv 2024

Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better
Algorithm-level | Efficient Training, Efficient Inference | Vision Generation Paper Code

ECCV 2024

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization
Model-level (Quantization) | Efficient Inference | Vision Generation Paper Code

ArXiv 2024

A Survey on Efficient Inference for Large Language Models Survey
| Efficient Inference | Language Paper

ArXiv 2024

LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K Benchmark, Evaluation
| | Paper Code

FPGA 2024

FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
System-level, Model-level (Quantization), Model-level (Sparsification) | Efficient Inference | Language Paper

DATE 2024

DyPIM: Dynamic-inference-enabled Processing-In-Memory Accelerator
System-level | Efficient Inference | Vision Recognition Paper

ICML 2024

Evaluating Quantized Large Language Models Evaluation
Model-level (Quantization) | Efficient Inference | Language Paper Code

CVPR 2024

FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models
Algorithm-level | Efficient Optimization Process | Vision Generation Paper Code

ICLR 2024

A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models
Algorithm-level | Efficient Inference | Vision Generation Paper

ICLR 2024

Skeleton-of-Thought: Prompting Large Language Models for Efficient Parallel Generation
Algorithm-level | Efficient Inference | Language Paper Code

WACV 2024

TCP: Triplet Contrastive-relationship Preserving for Class-Incremental Learning
| | Paper

NeurIPS Workshop 2023

LLM-MQ: Mixed-precision Quantization for Efficient LLM Deployment
Model-level (Quantization) | Efficient Inference | Language Paper

ICCV 2023

Ada3D: Exploiting the Spatial Redundancy with Adaptive Inference for Efficient 3D Object Detection
Model-level (Sparsification) | Efficient Inference | Vision Recognition Paper

ICML 2023

OMS-DPM: Deciding The Optimal Model Schedule for Diffusion Probabilistic Model
Algorithm-level | Efficient Inference | Vision Generation Paper Code

AAAI 2023 (Oral)

Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper Code

AAAI 2023

Memory-Oriented Structural Pruning for Efficient Image Restoration
Model-level (Structure Optimization) | Efficient Inference | Vision Generation Paper

AAAI 2023

Ensemble-in-One: Ensemble Learning within Random Gated Networks for Enhanced Adversarial Robustness
| | Vision Recognition Paper

TPAMI 2023

A Generic Graph-based Neural Architecture Encoding Scheme with Multifaceted Information
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper Code

DATE 2022 & TCAD 2023

Gibbon: Efficient Co-Exploration of NN Model and Processing-In-Memory Architecture
Model-level (Structure Optimization), System-level | Efficient Optimization Process, Efficient Inference | Vision Recognition Paper

TCAD 2022

Exploring the Potential of Low-bit Training of Convolutional Neural Networks
Model-level (Quantization) | Efficient Training | Vision Recognition Paper

CVPR 2022

CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance
Model-level (Structure Optimization) | Efficient Inference | Vision Recognition Paper

CVPR 2022

FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning
Algorithm-level | Efficient Training | Vision Recognition Paper

ECCV 2022

CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAS
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper

NeurIPS 2022 (Spotlight)

TA-GATES: An Encoding Scheme for Neural Network Architectures
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper

Low-Power CV 2022

Hardware Design and Software Practices for Efficient Neural Network Inference
Model-level (Structure Optimization), Model-level (Quantization), System-level | Efficient Inference | Vision Recognition Paper

NeurIPS 2021

Evaluating Efficient Performance Estimators of Neural Architectures Evaluation
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper Code

ASP-DAC 2020

Black Box Search Space Profiling for Accelerator-Aware Neural Architecture Search
Model-level (Structure Optimization) | Efficient Optimization Process, Efficient Inference | Vision Recognition Paper Code

ECCV 2020

A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
Model-level (Structure Optimization) | Efficient Optimization Process | Vision Recognition Paper Code

ECCV 2020 (Spotlight)

DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
Model-level (Structure Optimization) | Efficient Inference | Vision Recognition Paper

ArXiv 2020

aw_nas: A Modularized and Extensible NAS framework
Model-level (Structure Optimization) | Efficient Inference, Efficient Optimization Process | Vision Recognition, Language Paper Code