π A curated list of Awesome Parallel Decoding papers to accelerate the inference of (autoregressive) LLMs.
Autoregressive (AR) decoding generates one token per forward pass, leaving the strict token-by-token dependency as the dominant latency bottleneck of modern LLM serving. Parallel decoding refers to the broad family of techniques that shorten this critical path by introducing parallel candidates, parallel verification, parallel reasoning steps, or parallel task orchestration, while ideally preserving the quality (and, for lossless methods, the exact output distribution) of the original model. This repository organizes the literature along the granularity of parallelism β from (i) individual tokens, to (ii) reasoning sequences, to (iii) whole tasks β and additionally tracks the (iv) model architecture-level improvements and (v) hybrid advances. We also introduce (vi) system-level improvements and (vii) related benchmarks that make parallel decoding practical.
2026-10-18Our survey paper is now available! Please see paper/PD_Survey.pdf.
- πΊοΈ Overview: A Taxonomy of Parallel Decoding
- π 1. Token-Level Parallel Decoding
- π 2. Sequence-Level Parallel Decoding
- π 3. Task-Level Parallel Decoding
- π 4. Architecture-Level Parallel Decoding
- π 5. Hybrid Methods
- π 6. System and Infrastructure
- π 7. Benchmarks and Evaluation
Within each section, papers run from older to newer; click a paper to jump to its full row.
Token-level methods shorten autoregressive generation by proposing, organizing, or verifying multiple token candidates or future positions in parallel.
An independent draft model proposes candidate tokens that the target LLM verifies; this includes foundational speculative-decoding frameworks, drafter training and distillation, and cross-architecture draft adaptation.
The target model produces its own drafts through layer skipping, early exit, attached prediction heads, or internal feature prediction, removing the need for a separate full draft model.
Drafts come from retrieved text, n-gram or suffix caches, reusable context patterns, or parameter-free fixed-point/Jacobi procedures.
The main contribution concerns candidate-tree structure, multi-draft coupling and acceptance, adaptive draft length, verification order, or theoretical properties of speculative sampling.
Speculative decoding is integrated with another technique or application domain, including multimodal generation, quality/alignment/safety controls, watermarking, and accelerated reinforcement-learning rollouts.
Sequence-level methods expose parallelism over semantic units, reasoning segments, partial chains, or complete candidate trajectories.
Several reasoning continuations or complete answer sequences are generated in parallel and then verified, corrected, selected, or aggregated at the semantic level.
One output is decomposed into structurally independent sections, subproblems, or graph nodes that can be generated concurrently.
Parallel test-time compute is allocated across branching reasoning paths, with explicit control of exploration, pruning, early termination, and answer aggregation.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.02 | More Agents Is All You Need | Junyou Li et al. (@Tencent) | [code] |
TMLR'24 |
| 2024.10 | [Speculative Rejection] Fast Best-of-N Decoding via Speculative Rejection | Hanshi Sun et al. (@CMU & University of Virginia & UC Berkeley & Princeton & FDU & Google) | [code] |
NeurIPS'24 |
| 2025.02 | [DPTS] Dynamic Parallel Tree Search for Efficient LLM Reasoning | Yifu Ding et al. (@Beihang University & NTU & Wuhan University) | [code] |
ACL'25 |
| 2025.05 | [Group Think] Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity | Chan-Jan Hsu et al. (@MediaTek Research) | arXiv'25 | |
| 2025.07 | [SEAT] Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Framework | Zenan Xu et al. (@Tencent Hunyuan & CUHK) | arXiv'25 | |
| 2025.09 | [A2R] An Asymmetric Two-Stage Reasoning Framework for Parallel Reasoning | Ziqi Wang et al. (@USTC & Baidu & USYD) | arXiv'25 | |
| 2025.09 | [Parallel-R1] Towards Parallel Thinking via Reinforcement Learning | Tong Zheng et al. (@Tencent AI Lab & University of Maryland & UNC Chapel Hill & CityU & Washington University in St. Louis) | [code] |
ICLR'26 |
| 2025.09 | [ATTS] Asynchronous Test-Time Scaling via Conformal Prediction | Jing Xiong et al. (@HKU & Huawei & UCL & CUHK) | [code] |
ICLR'26 |
| 2025.10 | Parallel Test-Time Scaling for Latent Reasoning Models | Runyang You et al. (@PolyU & USTC & HIT Shenzhen & Shandong Jianzhu University) | [code] |
ACL'26 |
| 2025.10 | [DeepPrune] Parallel Scaling without Inter-trace Redundancy | Shangqing Tu et al. (@THU & ShanghaiTech University) | [code] |
Findings-ACL'26 |
| 2025.11 | [DTS] Enhancing Large Reasoning Models via Decoding Tree Sketching | Zicheng Xu et al. (@Johns Hopkins University & UNC Charlotte & Rice University & University of Minnesota) | [code] |
ICML'26 |
| 2026.01 | [SWE-Replay] SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents | Yifeng Ding, Lingming Zhang (@UIUC) | COLM'26 | |
| 2026.02 | [Parallel-Probe] Towards Efficient Parallel Thinking via 2D Probing | Tong Zheng et al. (@University of Maryland & Washington University in St. Louis & UNC Chapel Hill) | [code] |
ICML'26 |
| 2026.04 | Scaling Test-Time Compute for Agentic Coding | Joongwon Kim et al. (@Meta Superintelligence Labs & University of Washington & NYU & Google DeepMind & CMU & Princeton) | COLM'26 |
Task-level methods coordinate concurrent agents, plans, tool calls, functions, or environment actions.
Multiple LLM agents or teams solve and deliberate concurrently, then combine their results through voting, synthesis, or coordination.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2023.08 | [AutoGen] Enabling Next-Gen LLM Applications via Multi-Agent Conversations | Qingyun Wu et al. (@Microsoft & PSU & UW & Xidian University) | [code] |
COLM'24 |
| 2024.02 | More Agents Is All You Need | Junyou Li et al. (@Tencent) | [code] |
TMLR'24 |
| 2024.06 | [Mixture-of-Agents] Mixture-of-Agents Enhances Large Language Model Capabilities | Junlin Wang et al. (@Duke University & Together AI & University of Chicago & Stanford University) | [code] |
ICLR'25 |
| 2025.07 | [M1-Parallel] Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents | Enhao Zhang et al. (@UW & Microsoft) | ICMLW'25 | |
| 2026.04 | Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks | Yoonsang Lee et al. (@Princeton) | [code] |
COLM'26 |
Independent tools, functions, or predicted future actions are planned, dispatched, or speculatively executed in parallel.
Architecture-level methods redesign training or generation so strict left-to-right token causality is no longer the only decoding path.
The work designs, trains, converts, or scales architectures that support non-left-to-right generation, including masked diffusion, block diffusion, and semi-autoregressive models.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2017.11 | [NAT] Non-Autoregressive Neural Machine Translation | Jiatao Gu et al. (@Salesforce Research & HKU) | [code] |
ICLR'18 |
| 2018.02 | Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement | Jason Lee, Elman Mansimov, Kyunghyun Cho (@New York University & CIFAR) | [code] |
EMNLP'18 |
| 2018.03 | [Latent Transformer] Fast Decoding in Sequence Models Using Discrete Latent Variables | Εukasz Kaiser et al. (@Google Brain) | [code] |
ICML'18 |
| 2019.04 | [Mask-Predict] Parallel Decoding of Conditional Masked Language Models | Marjan Ghazvininejad et al. (@FAIR) | [code] |
EMNLP'19 |
| 2021.07 | [D3PM] Structured Denoising Diffusion Models in Discrete State-Spaces | Jacob Austin et al. (@Google Research) | [code] |
NeurIPS'21 |
| 2022.05 | [Diffusion-LM] Diffusion-LM Improves Controllable Text Generation | Xiang Lisa Li et al. (@Stanford University) | [code] |
NeurIPS'22 |
| 2023.10 | [SEDD] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution | Aaron Lou, Chenlin Meng, Stefano Ermon (@Stanford University & Pika Labs) | [code] |
ICML'24 |
| 2024.06 | [RADD] Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data | Jingyang Ou et al. (@RUC & Huawei) | [code] |
ICLR'25 |
| 2024.06 | [MD4] Simplified and Generalized Masked Diffusion for Discrete Data | Jiaxin Shi et al. (@Google DeepMind) | [code] |
NeurIPS'24 |
| 2024.06 | [MDLM] Simple and Effective Masked Diffusion Language Models | Subham Sekhar Sahoo et al. (@Cornell Tech) | [code] |
NeurIPS'24 |
| 2024.10 | [DiffuLLaMA] Scaling Diffusion Language Models via Adaptation from Autoregressive Models | Shansan Gong et al. (@HKU & UIUC & Apple & Tencent AI Lab) | [code] |
ICLR'25 |
| 2024.10 | [SDTT] Beyond Autoregression: Fast LLMs via Self-Distillation Through Time | Justin Deschenaux, Caglar Gulcehre (@EPFL / CLAIRE Lab) | [code] |
ICLR'25 |
| 2025.02 | [LLaDA] Large Language Diffusion Models | Shen Nie et al. (@Renmin University of China & Ant Group) | [code] |
NeurIPS'25 |
| 2025.03 | [Block Diffusion] Interpolating Between Autoregressive and Diffusion Language Models | Marianne Arriola et al. (@Cornell Tech & Stanford & Cohere) | [code] |
ICLR'25 |
| 2025.05 | [Dimple] Dimple: Discrete Diffusion Parallel Generation for Large Multimodal Modal | Runpeng Yu, Xinyin Ma, Xinchao Wang (@NUS) | EMNLP'26 | |
| 2025.06 | [Duo] The Diffusion Duality | Subham Sekhar Sahoo et al. (@Cornell Tech & EPFL & Cohere) | [code] |
ICML'25 |
| 2025.08 | [Dream 7B] Diffusion Large Language Models | Jiacheng Ye et al. (@HKU & Huawei) | [code] |
arXiv'25 |
| 2025.09 | [SDLM] Sequential Diffusion Language Models | Yangzhou Liu et al. (@Shanghai AI Lab & NJU & THU & FDU & CUHK & Soochow University & Donghua University) | [code] |
arXiv'25 |
| 2025.09 | [Fast-dLLM v2] Efficient Block-Diffusion LLM | Chengyue Wu et al. (@HKU & NVIDIA & MIT) | [code] |
ICLR'26 |
| 2025.09 | Training Optimal Large Diffusion Language Models | Jinjie Ni et al. (@NUS & Sea AI Lab & StepFun & Shanghai Qiji Zhifeng) | [code] |
NeurIPS'26 |
| 2025.10 | [RND1] Training Diffusion Language Models at Scale using Autoregressive Models | Radical Numerics Inc. (@Radical Numerics Inc.) | [code] |
misc'25 |
| 2025.10 | [E2D2] Encoder-Decoder Diffusion Language Models for Efficient Training and Inference | Marianne Arriola et al. (@Cornell University) | [code] |
NeurIPS'25 |
| 2025.10 | [SDAR] A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation | Shuang Cheng et al. (@Shanghai AI Lab & ZJU & UMD & SJTU & THU) | [code] |
Findings-ACL'26 |
| 2025.12 | [NBDiff] From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs | Yuchuan Tian et al. (@PKU & Huawei) | [code] |
arXiv'25 |
| 2025.12 | [Efficient-DLM] From Autoregressive to Diffusion Language Models, and Beyond in Speed | Yonggan Fu et al. (@NVIDIA & Georgia Tech & UChicago & HKU & MIT) | ICML'26 | |
| 2025.12 | [LLaDA2.0] Scaling Up Diffusion Language Models to 100B | Tiwei Bie et al. (@Ant Group & Renmin University of China & ZJU & Westlake University & HKUST) | [code] |
arXiv'25 |
| 2025.12 | Corrective Diffusion Language Models | Shuibai Zhang et al. (@UW-Madison & Duke) | [code] |
NeurIPS'26 |
| 2026.01 | [Diffusion In Diffusion] Reclaiming Global Coherence in Semi-Autoregressive Diffusion | Linrui Ma et al. (@Huawei) | arXiv'26 | |
| 2026.01 | [CD4LM] CD4LM: Consistency Distillation and aDaptive Decoding for Diffusion Language Models | Yihao Liang et al. (@Princeton & AMD) | [code] |
Findings-EMNLP'26 |
| 2026.02 | [Learn from Your Mistakes] Learn from Your Mistakes: Self-Correcting Masked Diffusion Models | Yair Schiff et al. (@Cornell & NVIDIA) | [code] |
NeurIPS'26 |
| 2026.02 | [Flow Map Language Models] Flow Map Language Models: One-step Language Modeling via Continuous Denoising | Chanhyuk Lee et al. (@KAIST & CMU) | [code] |
NeurIPS'26 |
| 2026.03 | [LLaDA-o] LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model | Zebin You et al. (@RUC & Ant Group) | [code] |
Findings-EMNLP'26 |
| 2026.03 | [CoDAR] CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think | Junzhe Shen et al. (@SJTU & Shanghai Innovation Institute & Shanghai AI Laboratory) | EMNLP'26 | |
| 2026.04 | [DMax] DMax: Aggressive Parallel Decoding for dLLMs | Zigeng Chen et al. (@NUS) | [code] |
NeurIPS'26 |
| 2026.04 | Introspective Diffusion Language Models | Yifan Yu et al. (@Together AI & UIUC & UT Austin & Princeton & Stanford) | [code] |
COLM'26 |
| 2026.05 | [CoBit] CoBit: Language Modeling with Bitstream Diffusion | Georgios Batzolis, Mark Girolami, Luca Ambrogioni (@Devotion AI Labs & University of Cambridge & Donders Institute & Radboud University) | NeurIPS'26 | |
| 2026.05 | How to Train Your Latent Diffusion Language Model Jointly With the Latent Space | Viacheslav Meshchaninov et al. (@Constructor University & HSE University & Applied AI Institute & Independent researcher) | [code] |
NeurIPS'26 |
| 2026.05 | Continuous Diffusion Scales Competitively with Discrete Diffusion for Language | Zhihan Yang et al. (@NVIDIA & Cornell & Georgia Tech & UW-Madison & MBZUAI-IFM) | NeurIPS'26 | |
| 2026.06 | [MBD-LMs] Multi-Block Diffusion Language Models | Yijie Jin et al. (@SJTU & XJTU & Huawei) | [code] |
arXiv'26 |
| 2026.07 | [Nemotron-Labs-Diffusion] Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding | Yonggan Fu et al. (@NVIDIA & Georgia Tech & HKU & University of Chicago & MIT) | EMNLP'26 |
Inference algorithms accelerate trained non-autoregressive or diffusion language models through unmasking policies, confidence schedules, trajectory methods, or error correction.
Hybrid methods make the deliberate combination of multiple decoding or generative paradigms their central contribution.
A non-autoregressive or semi-autoregressive language model drafts multiple tokens in parallel for verification by an autoregressive target.
The draft-then-verify principle is transferred to diffusion language models, model cascades, continuous diffusion models, and visual autoregressive generation.
Related hybrid resource. Speculative Cascades β A hybrid approach for smarter, faster LLM inference (Google Research blog, 2025) connects routing, selection, and speculation into one production-oriented framework.
System and infrastructure work turns algorithmic parallelism into reliable latency, throughput, memory, and SLO gains through serving, scheduling, and algorithmβruntime co-design.
Runtime and serving techniques optimize batching, GPU scheduling, pipelines, distributed execution, caching, or SLO-aware adaptation for parallel decoding.
The decoding algorithm is co-designed with deployment constraints or systems techniques such as quantization, KV-cache management, MoE execution, and hardware-aware verification.
These software projects support speculative or parallel decoding but are not paper records in the taxonomy, so they are not included in the classified-paper counts above.
| Name | Type | Documented Support | Code |
|---|---|---|---|
| vLLM | Serving engine | EAGLE / EAGLE-3, MTP, DFlash, PARD, MLP, draft-model SD, n-gram, suffix decoding | [code] |
| SGLang | Serving engine | EAGLE-2 / EAGLE-3, MTP, DFlash, draft-model SD, n-gram | [code] |
| TensorRT-LLM | Serving engine | Medusa, EAGLE, ReDrafter, n-gram, draft-model SD, lookahead decoding | [code] |
| TGI | Serving engine | Medusa, n-gram speculation | [code] |
| llama.cpp | Serving engine | Draft-model SD, EAGLE-3, DFlash, DSpark, MTP, n-gram cache/map | [code] |
| NVIDIA Triton | Serving engine | Medusa, ReDrafter, Lookahead, EAGLE / EAGLE-3, draft-model SD, n-gram | [code] |
| Speculators | Library | Unified abstraction & training for SD drafters (vLLM-native) | [code] |
| TorchSpec | Training framework | PyTorch-native large-scale SD drafter training | [code] |
This layer organizes and evaluates the field through related surveys, benchmark suites, and diagnostic studies.
Related surveys.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.01 | Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding | Heming Xia et al. (@PolyU & PKU & MSRA & Alibaba) | [code] |
Findings-ACL'24 |
| 2025.08 | A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models | Lingzhe Zhang, Liancheng Fang et al. (@PKU & UIC & THU & XPENG & Alibaba & HKUST(GZ)) | [list] |
arXiv'25 |
| 2025.10 | A Survey on Parallel Reasoning | Ziqi Wang, Boye Niu et al. (@USTC & Baidu & USYD) | [list] |
arXiv'25 |
Benchmark suites.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2024.01 | [Spec-Bench] A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding | Heming Xia et al. (@PolyU & PKU & MSRA & Alibaba) | [code] |
misc'24 |
| 2025.09 | Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling | Shengyin Sun, Yiming Li et al. (@CityU HK & Huawei) | [code] |
ICLR'26 |
Diagnostic studies.
| Date | Title | Authors | Code | Venue |
|---|---|---|---|---|
| 2025.10 | How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices | Han Peng, Peiyu Liu et al. (@RUC & UIBE & THU & CityU HK) | arXiv'25 | |
| 2025.10 | [ParallelBench] Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs | Wonjun Kang, Kevin Galim et al. (@FuriosaAI & UW-Madison & Microsoft & UC Berkeley & SNU & KRAFTON AI & Ludo Robotics) | [code] |
ICLR'26 |
- hemingkx/SpeculativeDecodingPapers β speculative decoding papers
- zhanglingzhe0820/Awesome-Parallel-Text-Generation β parallel text generation (AR-based & non-AR / dLLM)
- Geralt-Targaryen/Awesome-Speculative-Decoding β speculative decoding (with detailed reading notes)
- hemingkx/Awesome-Efficient-Reasoning β efficient reasoning (speculative reasoning & parallel thinking)
- Xiaohao-Liu/Awesome-Multi-Token-Prediction β multi-token prediction
- PPPP-kaqiu/Awesome-Parallel-Reasoning β parallel reasoning
- wang2226/Awesome-LLM-Decoding β general LLM decoding
- VILA-Lab/Awesome-DLMs β diffusion language models
- ML-GSAI/Diffusion-LLM-Papers - diffusion large language models
- MessiX77/Awesome-Efficient-dLLMs β efficient diffusion language models (dLLMs)
Contributions are very welcome! If you would like to add a paper, fix metadata (venue, affiliation, links), or propose a new category, please open a pull request or an issue. When adding a paper, please keep entries in chronological order within each section and follow the existing table format: Date | [Alias] Title (linked to paper) | Authors (@Affiliation) | Code (+ stars badge) | Venue. Where information is uncertain, please mark it as N/A rather than guessing.
π If you find this repository useful, please consider citing our survey paper: π
@misc{survey-parallel-decoding,
title = {Parallel Decoding of Language Models: A Survey},
author = {Yu-Yang Qian and Jia-Chen Liu and Hao-Cong Wu and Lanxiang Hu and Yibo Zhu and Hao Zhang and Peng Zhao},
year = {2026},
note = {\url{https://github.com/ZinYY/Awesome-Parallel-Decoding}}
}π Our survey paper is at this file.
