Skip to main content

Inference - Controlling Output Quality

"Inference transforms probability distributions into coherent text through strategic token selection."

Training builds the model, but inference determines its output. Understanding decoding strategies, sampling parameters, and optimization techniques is essential for controlling model behavior in production. This document covers autoregressive generation, decoding algorithms, sampling parameters, and performance optimization techniques for deploying LLMs at scale.

Autoregressive Generation​

The Generation Loop​

LLMs generate text autoregressively - one token at a time, with each new token conditioned on all previous tokens.

Basic Implementation​

import torch
import torch.nn.functional as F

def generate_autoregressive(
    model,
    input_ids: torch.Tensor,
    max_new_tokens: int = 100,
    temperature: float = 1.0,
    top_k: int = None,
    top_p: float = 1.0,
    eos_token_id: int = None
) -> torch.Tensor:
    """
    Generate tokens autoregressively.

    Args:
        model: Language model
        input_ids: Input token IDs (batch, seq_len)
        max_new_tokens: Maximum tokens to generate
        temperature: Sampling temperature
        top_k: Keep top k tokens
        top_p: Nucleus sampling threshold
        eos_token_id: End-of-sequence token ID
    """
    batch_size, seq_len = input_ids.shape
    current_ids = input_ids.clone()

    for step in range(max_new_tokens):
        # Forward pass
        with torch.no_grad():
            outputs = model(current_ids)
            logits = outputs.logits[:, -1, :]  # (batch, vocab_size)

        # Apply temperature
        logits = logits / temperature

        # Apply top-k filtering
        if top_k is not None:
            top_k_logits, top_k_indices = torch.topk(logits, top_k)
            logits = torch.full_like(logits, float('-inf'))
            logits.scatter_(1, top_k_indices, top_k_logits)

        # Apply top-p (nucleus) filtering
        if top_p < 1.0:
            sorted_logits, sorted_indices = torch.sort(logits, descending=True)
            cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

            # Remove tokens with cumulative probability above threshold
            sorted_indices_to_remove = cumulative_probs > top_p
            sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
            sorted_indices_to_remove[..., 0] = False

            indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)
            logits[indices_to_remove] = float('-inf')

        # Sample token
        probs = F.softmax(logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)  # (batch, 1)

        # Append to sequence
        current_ids = torch.cat([current_ids, next_token], dim=1)

        # Check for EOS
        if eos_token_id is not None and (next_token == eos_token_id).all():
            break

    return current_ids

# Usage
input_text = "The future of AI is"
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output_ids = generate_autoregressive(model, input_ids, max_new_tokens=50, temperature=0.8)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(output_text)

Decoding Strategies​

Strategy Comparison​

StrategyDescriptionSpeedQualityDiversityBest For
Greedy SearchAlways pick highest probabilityFastestGood for factsNoneFactual QA, code
Beam SearchKeep top k hypothesesSlowHigh qualityLowTranslation, summarization
SamplingSample from probability distributionFastVariableHighCreative writing
Nucleus (Top-p)Sample from smallest top massFastGoodHighGeneral chat, assistants
Top-kSample from top k tokensFastGoodMedium-HighBalanced generation
ContrastiveBalance prob + degeneration penaltyMediumVery highMediumLong-form content
MCTSTree search with lookaheadVery slowBestMediumComplex reasoning
def greedy_search(model, input_ids: torch.Tensor, max_new_tokens: int) -> torch.Tensor:
    """
    Greedy decoding: always select the most likely token.
    """
    current_ids = input_ids.clone()

    for _ in range(max_new_tokens):
        outputs = model(current_ids)
        next_token = outputs.logits[:, -1, :].argmax(dim=-1, keepdim=True)
        current_ids = torch.cat([current_ids, next_token], dim=1)

    return current_ids

# Example: Greedy might repeat "The the the the..." if it gets stuck
def beam_search(
    model,
    input_ids: torch.Tensor,
    max_new_tokens: int,
    num_beams: int = 4,
    length_penalty: float = 1.0
) -> torch.Tensor:
    """
    Beam search: keep top-k hypotheses at each step.

    Args:
        model: Language model
        input_ids: Input token IDs
        max_new_tokens: Maximum tokens to generate
        num_beams: Number of beams to track
        length_penalty: Penalize short sequences (1.0 = no penalty)
    """
    batch_size = input_ids.shape[0]
    input_ids = input_ids.repeat_interleave(num_beams, dim=0)

    # Initialize beams
    beam_scores = torch.zeros(batch_size * num_beams, device=input_ids.device)
    beam_scores[1::num_beams] = float('-inf')  # Only first beam is valid

    for step in range(max_new_tokens):
        outputs = model(input_ids)
        next_token_logits = outputs.logits[:, -1, :]
        next_token_scores = F.log_softmax(next_token_logits, dim=-1)

        # Add beam scores
        vocab_size = next_token_scores.shape[-1]
        next_scores = beam_scores.unsqueeze(-1) + next_token_scores

        # Reshape for top-k selection
        next_scores = next_scores.view(batch_size, num_beams * vocab_size)
        next_scores, next_tokens = torch.topk(next_scores, k=num_beams, dim=-1)

        # Convert flat indices to (beam, token) pairs
        beam_indices = next_tokens // vocab_size
        token_indices = next_tokens % vocab_size

        # Update beams
        input_ids = input_ids.view(batch_size, num_beams, -1)
        input_ids = input_ids[torch.arange(batch_size).unsqueeze(-1), beam_indices]
        input_ids = input_ids.reshape(batch_size * num_beams, -1)

        next_token_ids = token_indices.view(-1, 1)
        input_ids = torch.cat([input_ids, next_token_ids], dim=-1)

        beam_scores = next_scores.view(-1)

    # Select best beam
    input_ids = input_ids.view(batch_size, num_beams, -1)
    best_beam_indices = beam_scores.view(batch_size, num_beams).argmax(dim=-1)
    output_ids = input_ids[torch.arange(batch_size), best_beam_indices]

    return output_ids

# Example usage
output = beam_search(model, input_ids, max_new_tokens=50, num_beams=4)

Contrastive search combines probability modeling with degeneration penalty:

def contrastive_search(
    model,
    input_ids: torch.Tensor,
    max_new_tokens: int,
    top_k: int = 5,
    alpha: float = 0.6,
    momentum: float = 0.5
) -> torch.Tensor:
    """
    Contrastive search: balance probability and similarity to context.

    Args:
        model: Language model
        input_ids: Input tokens
        max_new_tokens: Maximum tokens to generate
        top_k: Number of candidates to consider
        alpha: Penalty for repetition (0 = pure sampling, 1 = pure degeneration penalty)
        momentum: Weight for updating cumulative probability

    Reference:
        Su, J., et al. (2022). "A Contrastive Framework for Neural Text Generation"
        https://arxiv.org/abs/2202.01855
    """
    current_ids = input_ids.clone()
    cumulative_probs = None

    for _ in range(max_new_tokens):
        # Forward pass
        with torch.no_grad():
            outputs = model(current_ids)
            logits = outputs.logits[:, -1, :]
            hidden_states = outputs.hidden_states[-1][:, -1, :]  # Last layer hidden state

        # Get top-k candidates
        top_k_logits, top_k_indices = torch.topk(logits, top_k)
        top_k_probs = F.softmax(top_k_logits, dim=-1)

        # Compute model confidence (probability mass)
        model_confidence = top_k_probs.max(dim=-1)[0]

        # Compute degeneration penalty (similarity to previous tokens)
        # Use cosine similarity between current hidden state and previous states
        prev_hidden = outputs.hidden_states[-1][:, :-1, :]  # All previous hidden states

        # Compute similarity with most recent tokens
        if prev_hidden.size(1) > 0:
            # Similarity with last few tokens (local context)
            recent_hidden = prev_hidden[:, -min(5, prev_hidden.size(1)):, :]
            similarities = F.cosine_similarity(
                hidden_states.unsqueeze(1),
                recent_hidden,
                dim=-1
            ).max(dim=1)[0]
            degeneration_penalty = similarities.max()
        else:
            degeneration_penalty = torch.tensor(0.0)

        # Update cumulative probability
        if cumulative_probs is None:
            cumulative_probs = model_confidence
        else:
            cumulative_probs = momentum * cumulative_probs + (1 - momentum) * model_confidence

        # Select token that maximizes: (1-alpha) * probability - alpha * degeneration
        # For each top-k token, compute the score
        scores = (1 - alpha) * top_k_probs - alpha * degeneration_penalty

        # Select best token
        best_idx = scores.argmax(dim=-1)
        next_token = top_k_indices[range(len(best_idx)), best_idx].unsqueeze(-1)

        current_ids = torch.cat([current_ids, next_token], dim=1)

    return current_ids

MCTS (Monte Carlo Tree Search) Decoding​

For tasks requiring deeper lookahead:

class MCTSNode:
    """Node for Monte Carlo Tree Search decoding."""
    def __init__(self, token_id: int, parent=None):
        self.token_id = token_id
        self.parent = parent
        self.children = []
        self.visits = 0
        self.total_value = 0.0
        self.prior_prob = 0.0

    def ucb_score(self, c_puct: float = 1.0) -> float:
        """Upper Confidence Bound for selection."""
        if self.visits == 0:
            return float('inf')

        exploitation = self.total_value / self.visits
        exploration = c_puct * math.sqrt(math.log(self.parent.visits) / self.visits) if self.parent else 0

        return exploitation + exploration

def mcts_decode(
    model,
    input_ids: torch.Tensor,
    max_new_tokens: int,
    num_simulations: int = 50,
    c_puct: float = 1.0,
    temperature: float = 1.0
) -> torch.Tensor:
    """
    Monte Carlo Tree Search for lookahead decoding.

    More expensive but can find better sequences for complex tasks.
    """
    current_ids = input_ids.clone()

    for step in range(max_new_tokens):
        # Build search tree
        root = MCTSNode(token_id=None)

        # Run simulations
        for _ in range(num_simulations):
            # Selection: traverse tree using UCB
            node = root
            path = []

            while node.children:
                # Select child with highest UCB score
                node = max(node.children, key=lambda n: n.ucb_score(c_puct))
                path.append(node)

            # Expansion: add new child if not terminal
            if len(path) < 5:  # Limit depth
                # Get model predictions
                with torch.no_grad():
                    outputs = model(current_ids)
                    logits = outputs.logits[:, -1, :] / temperature
                    probs = F.softmax(logits, dim=-1)

                # Sample candidate token
                token = torch.multinomial(probs, num_samples=1).item()
                child = MCTSNode(token, parent=node)
                node.children.append(child)
                node.prior_prob = probs[0, token].item()
                path.append(child)

            # Evaluation: estimate value using model
            with torch.no_grad():
                # Construct full sequence for evaluation
                eval_ids = current_ids.clone()
                for node in path[1:]:
                    eval_ids = torch.cat([eval_ids, torch.tensor([[node.token_id]])], dim=1)

                outputs = model(eval_ids)
                # Value = negative loss (higher is better)
                value = -outputs.loss.item() if hasattr(outputs, 'loss') else 0.0

            # Backpropagation: update values
            for node in path:
                node.visits += 1
                node.total_value += value

        # Select best child after simulations
        if root.children:
            best_child = max(root.children, key=lambda n: n.visits)
            next_token = torch.tensor([[best_child.token_id]])
            current_ids = torch.cat([current_ids, next_token], dim=1)
        else:
            break

    return current_ids

Sampling with Temperature​

def sample_with_temperature(
    model,
    input_ids: torch.Tensor,
    max_new_tokens: int,
    temperature: float = 1.0
) -> torch.Tensor:
    """
    Sample from the probability distribution with temperature scaling.
    """
    current_ids = input_ids.clone()

    for _ in range(max_new_tokens):
        outputs = model(current_ids)
        logits = outputs.logits[:, -1, :] / temperature

        # Sample from softmax distribution
        probs = F.softmax(logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)

        current_ids = torch.cat([current_ids, next_token], dim=1)

    return current_ids

# Temperature effects
# T=0.1: Very deterministic, almost greedy
# T=0.5: Focused, mostly high-probability tokens
# T=1.0: Standard sampling
# T=1.5: More creative, includes lower-probability tokens
# T=2.0+: Very random, often incoherent

Sampling Parameters​

Temperature​

Controls randomness in sampling:

def apply_temperature(logits: torch.Tensor, temperature: float) -> torch.Tensor:
    """
    Apply temperature scaling to logits.

    Lower temperature -> sharper distribution
    Higher temperature -> flatter distribution
    """
    return logits / temperature

# Temperature effects on probability distribution
logits = torch.tensor([2.0, 1.0, 0.0, -1.0, -2.0])

print("Temperature effects:")
for temp in [0.1, 0.5, 1.0, 2.0]:
    scaled = apply_temperature(logits, temp)
    probs = F.softmax(scaled, dim=0)
    print(f"  T={temp:.1f}: {probs.tolist()}")

Output:

Temperature effects:
  T=0.1: [0.97, 0.03, 0.00, 0.00, 0.00]  # Very sharp
  T=0.5: [0.67, 0.24, 0.07, 0.01, 0.00]  # Focused
  T=1.0: [0.50, 0.27, 0.12, 0.07, 0.04]  # Balanced
  T=2.0: [0.39, 0.32, 0.16, 0.09, 0.05]  # Flat

Top-K vs Top-P (Nucleus)​

def apply_top_k(logits: torch.Tensor, top_k: int) -> torch.Tensor:
    """
    Filter to keep only top k tokens.
    """
    top_k_logits, top_k_indices = torch.topk(logits, top_k)
    filtered = torch.full_like(logits, float('-inf'))
    filtered.scatter_(0, top_k_indices, top_k_logits)
    return filtered

def apply_top_p(logits: torch.Tensor, top_p: float) -> torch.Tensor:
    """
    Nucleus sampling: keep smallest top set with cumulative mass >= top_p.
    """
    sorted_logits, sorted_indices = torch.sort(logits, descending=True)
    cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

    # Find indices to remove
    sorted_indices_to_remove = cumulative_probs > top_p
    sorted_indices_to_remove[1:] = sorted_indices_to_remove[:-1].clone()
    sorted_indices_to_remove[0] = False

    # Scatter back to original order
    indices_to_remove = sorted_indices_to_remove.scatter(0, sorted_indices, sorted_indices_to_remove)
    logits[indices_to_remove] = float('-inf')

    return logits

# Example: Top-k vs Top-p
logits = torch.randn(50000)  # Large vocabulary

# Top-k: Always keep exactly k tokens
top_k_filtered = apply_top_k(logits, top_k=50)

# Top-p: Keep variable number of tokens
top_p_filtered = apply_top_p(logits, top_p=0.9)
num_kept = (top_p_filtered != float('-inf')).sum()
print(f"Top-p=0.9 kept {num_kept} tokens")

Frequency and Presence Penalties​

def apply_frequency_penalty(
    logits: torch.Tensor,
    token_ids: torch.Tensor,
    frequency_penalty: float = 0.0,
    presence_penalty: float = 0.0
) -> torch.Tensor:
    """
    Apply frequency and presence penalties.

    Args:
        logits: Model logits (vocab_size,)
        token_ids: Previously generated token IDs
        frequency_penalty: Penalize based on frequency (higher = less repetition)
        presence_penalty: Penalize based on presence (binary)
    """
    # Count token frequencies
    unique_tokens, counts = torch.unique(token_ids, return_counts=True)

    # Apply penalties
    for token, count in zip(unique_tokens, counts):
        # Frequency penalty: scales with count
        logits[token] -= frequency_penalty * count

        # Presence penalty: binary (present or not)
        logits[token] -= presence_penalty

    return logits

# Example
generated_tokens = torch.tensor([10, 20, 10, 30, 10, 20])  # 10: 3x, 20: 2x, 30: 1x
logits = torch.randn(50000)

# With penalties, tokens 10 and 20 get penalized more heavily
logits_penalized = apply_frequency_penalty(
    logits.clone(),
    generated_tokens,
    frequency_penalty=0.5,
    presence_penalty=0.1
)

Parameter Reference Table​

ParameterRangeEffectUse Case
temperature0.0 - 2.0Randomness0.2: coding, facts / 0.8: chat / 1.2: creative
top_k1 - 100Diversity1: greedy / 40-50: balanced / 100: very diverse
top_p0.1 - 1.0Mass filtering0.5: focused / 0.9: standard / 1.0: no filtering
frequency_penalty0.0 - 2.0Reduce repetition0.0: none / 0.5: some / 1.0: strong
presence_penalty0.0 - 2.0Encourage variety0.0: none / 0.5: moderate / 1.0: strong

Performance Optimizations​

KV Cache​

Key-Value cache stores attention keys and values from previous tokens to avoid recomputation.

Complexity comparison:

  • Without cache: O(N^2) per step
  • With cache: O(N) per step
class KVCache:
    """
    Key-Value cache for efficient autoregressive generation.
    """
    def __init__(self, batch_size: int, num_layers: int, num_heads: int, head_dim: int, max_len: int = 2048):
        self.batch_size = batch_size
        self.num_layers = num_layers
        self.num_heads = num_heads
        self.head_dim = head_dim
        self.max_len = max_len

        # Pre-allocate cache
        # Shape: (num_layers, batch_size, num_heads, max_len, head_dim)
        self.keys = torch.zeros(num_layers, batch_size, num_heads, max_len, head_dim)
        self.values = torch.zeros(num_layers, batch_size, num_heads, max_len, head_dim)
        self.current_len = 0

    def update(self, layer_idx: int, keys: torch.Tensor, values: torch.Tensor):
        """
        Update cache for a specific layer.

        Args:
            layer_idx: Layer index
            keys: New keys (batch, num_heads, seq_len, head_dim)
            values: New values (batch, num_heads, seq_len, head_dim)
        """
        seq_len = keys.size(2)
        if self.current_len + seq_len > self.max_len:
            raise ValueError(f"Cache overflow: {self.current_len + seq_len} > {self.max_len}")

        # Store keys and values
        self.keys[layer_idx, :, :, self.current_len:self.current_len+seq_len, :] = keys
        self.values[layer_idx, :, :, self.current_len:self.current_len+seq_len, :] = values

    def get(self, layer_idx: int) -> tuple[torch.Tensor, torch.Tensor]:
        """
        Get cached keys and values for a layer.

        Returns:
            keys: (batch, num_heads, current_len, head_dim)
            values: (batch, num_heads, current_len, head_dim)
        """
        return (
            self.keys[layer_idx, :, :, :self.current_len, :],
            self.values[layer_idx, :, :, :self.current_len, :]
        )

    def increment(self, n: int):
        """Increment current length."""
        self.current_len += n

# Modified attention with KV cache
def attention_with_cache(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    cache: KVCache,
    layer_idx: int,
    use_cache: bool = True
) -> tuple[torch.Tensor, tuple[torch.Tensor, torch.Tensor]]:
    """
    Compute attention with KV caching.

    Args:
        q: Query (batch, num_heads, seq_len, head_dim)
        k: Key (batch, num_heads, seq_len, head_dim)
        v: Value (batch, num_heads, seq_len, head_dim)
        cache: KV cache
        layer_idx: Current layer index
        use_cache: Whether to use/update cache

    Returns:
        output: Attention output (batch, num_heads, seq_len, head_dim)
        (k, v): Keys and values for caching
    """
    if use_cache:
        # Update cache with new keys and values
        cache.update(layer_idx, k, v)

        # Get all keys and values (cached + new)
        k_all, v_all = cache.get(layer_idx)
    else:
        k_all, v_all = k, v

    # Compute attention scores
    scores = torch.matmul(q, k_all.transpose(-2, -1)) / math.sqrt(q.size(-1))
    attn_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attn_weights, v_all)

    return output, (k, v)

Speculative Decoding​

Use a smaller draft model to predict tokens, then verify with the larger model.

def speculative_decode(
    draft_model,
    target_model,
    input_ids: torch.Tensor,
    max_new_tokens: int,
    spec_len: int = 4
) -> torch.Tensor:
    """
    Speculative decoding with a draft model.

    Args:
        draft_model: Smaller, faster model
        target_model: Larger, better model
        input_ids: Input tokens
        max_new_tokens: Maximum tokens to generate
        spec_len: Number of tokens to speculate ahead
    """
    current_ids = input_ids.clone()

    while current_ids.size(1) < input_ids.size(1) + max_new_tokens:
        # Step 1: Draft model predicts spec_len tokens
        draft_tokens = []
        draft_probs = []

        draft_input = current_ids
        for _ in range(spec_len):
            with torch.no_grad():
                draft_outputs = draft_model(draft_input)
                draft_logits = draft_outputs.logits[:, -1, :]
                draft_prob = F.softmax(draft_logits, dim=-1)

            # Sample token
            draft_token = torch.multinomial(draft_prob, num_samples=1)
            draft_tokens.append(draft_token)
            draft_probs.append(draft_prob)

            draft_input = torch.cat([draft_input, draft_token], dim=1)

        draft_tokens = torch.cat(draft_tokens, dim=1)

        # Step 2: Target model verifies draft tokens
        with torch.no_grad():
            target_outputs = target_model(current_ids)
            target_logits = target_outputs.logits[:, -1:, :]  # Single token
            target_prob = F.softmax(target_logits, dim=-1)

        # Step 3: Accept or reject each draft token
        accepted_tokens = []

        for i, (token, draft_p) in enumerate(zip(draft_tokens.T, draft_probs)):
            # Get target probability for this token
            target_p = target_prob[0, :, token]

            # Accept with probability target_p / draft_p
            accept_prob = target_p / draft_p[0, token]

            if torch.rand(1) < accept_prob:
                accepted_tokens.append(token)
                current_ids = torch.cat([current_ids, token.unsqueeze(0)], dim=1)

                # Update target model for next verification
                with torch.no_grad():
                    target_outputs = target_model(current_ids)
                    target_logits = target_outputs.logits[:, -1:, :]
                    target_prob = F.softmax(target_logits, dim=-1)
            else:
                # Reject: resample from target distribution
                resampled = torch.multinomial(target_prob[0], num_samples=1)
                accepted_tokens.append(resampled)
                current_ids = torch.cat([current_ids, resampled.unsqueeze(0)], dim=1)
                break  # Stop speculation, resync

    return current_ids

# Speedup: 2-3x for 10x model size ratio

2025: Medusa - Multi-Head Speculative Decoding​

Medusa (2024) eliminates the need for a separate draft model by adding multiple decoding heads to the base model.

How it works:

  1. Training: Add lightweight prediction heads to the base model
  2. Inference: All heads predict tokens in parallel
  3. Verification: Base model verifies all predictions simultaneously
  4. Speedup: 2.2-3.6x faster with minimal quality loss

Advantages over draft model approach:

  • No separate draft model needed
  • Better quality (heads are trained on same model)
  • Easier deployment (single model)
  • Scales with model size

Performance comparison:

MethodSpeedupDraft ModelQualitySetup
Traditional Speculative2-3xYes~99%Complex (2 models)
Medusa-12.2xNo~99%Simple (1 model + heads)
Medusa-23.6xNo~98.5%Simple (1 model + heads)
class MedusaModel:
    """
    Medusa: Single-model speculative decoding with multiple heads.

    Reference: https://github.com/FasterDecoding/Medusa
    """

    def __init__(self, base_model, num_heads=5):
        self.base_model = base_model
        self.num_heads = num_heads

        # Medusa heads are lightweight linear layers
        # Each head predicts tokens at different lookahead distances
        self.medusa_heads = nn.ModuleList([
            nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size)
            for _ in range(num_heads)
        ])

    def forward(self, input_ids):
        """
        Forward pass with parallel multi-head prediction.
        """
        # Get base model output
        outputs = self.base_model(input_ids, output_hidden_states=True)
        hidden_states = outputs.hidden_states[-1]  # Last layer hidden state

        # Base model prediction (next token)
        base_logits = outputs.logits[:, -1, :]

        # Parallel predictions from all Medusa heads
        medusa_logits = []
        for head in self.medusa_heads:
            medusa_logits.append(head(hidden_states[:, -1, :]))

        return base_logits, medusa_logits

    def generate_medusa(
        self,
        input_ids,
        max_new_tokens=100,
        temperature=1.0
    ):
        """
        Medusa generation with parallel verification.
        """
        current_ids = input_ids.clone()

        for step in range(max_new_tokens):
            # Get predictions from base model and all heads
            with torch.no_grad():
                base_logits, medusa_logits = self.forward(current_ids)

                # Apply temperature
                base_logits = base_logits / temperature
                medusa_logits = [logits / temperature for logits in medusa_logits]

                # Sample from base model
                base_probs = F.softmax(base_logits, dim=-1)
                base_token = torch.multinomial(base_probs, num_samples=1)

                # Get predictions from all Medusa heads
                candidate_tokens = []
                candidate_probs = []

                for head_idx, logits in enumerate(medusa_logits):
                    probs = F.softmax(logits, dim=-1)
                    token = torch.multinomial(probs, num_samples=1)
                    candidate_tokens.append(token)
                    candidate_probs.append(probs)

            # Verify candidates in parallel
            accepted_count = 0
            for head_idx, (token, draft_p) in enumerate(zip(candidate_tokens, candidate_probs)):
                # Position to verify: 1 + head_idx tokens ahead
                verify_pos = current_ids.size(1) + head_idx

                if verify_pos >= input_ids.size(1) + max_new_tokens:
                    break

                # Get base model probability for this candidate
                with torch.no_grad():
                    verify_outputs = self.base_model(current_ids)
                    verify_logits = verify_outputs.logits[:, -1, :]
                    verify_probs = F.softmax(verify_logits / temperature, dim=-1)

                target_p = verify_probs[0, :, token]

                # Acceptance criterion
                accept_prob = target_p / (draft_p[0, token] + 1e-8)
                if torch.rand(1).item() < accept_prob:
                    current_ids = torch.cat([current_ids, token], dim=1)
                    accepted_count += 1
                else:
                    # Reject and resample from base model
                    resampled = torch.multinomial(verify_probs[0], num_samples=1)
                    current_ids = torch.cat([current_ids, resampled], dim=1)
                    break  # Stop accepting after first rejection

        return current_ids

2025: QuantSpec - Quantized KV Speculative Decoding​

QuantSpec (2024) combines speculative decoding with quantized KV cache for self-speculation.

Key innovations:

  1. Hierarchical KV quantization: Different precision levels for different tokens
  2. Self-speculative: No separate draft model needed
  3. 4-bit KV cache: Reduces memory by 75%
  4. 2.5x speedup: With minimal quality degradation
def quantspec_inference(
    model,
    input_ids,
    max_new_tokens=100,
    kv_bits=4
):
    """
    QuantSpec: Self-speculative decoding with quantized KV cache.

    Args:
        model: Language model
        input_ids: Input tokens
        max_new_tokens: Maximum tokens to generate
        kv_bits: Bits for KV quantization (4, 8)

    Reference: https://arxiv.org/abs/2402.10568
    """
    # Hierarchical KV cache with different quantization levels
    # Recent tokens: FP16 (high quality)
    # Medium tokens: INT8 (balanced)
    # Old tokens: INT4 (compressed)

    cache_structure = {
        'recent': {'tokens': 64, 'bits': 16},  # Most recent: FP16
        'medium': {'tokens': 256, 'bits': 8},   # Medium: INT8
        'old': {'tokens': -1, 'bits': 4}        # Old: INT4
    }

    current_ids = input_ids.clone()
    kv_cache = {}

    for step in range(max_new_tokens):
        # Quantize KV cache based on recency
        for layer_idx in range(len(model.model.layers)):
            if layer_idx in kv_cache:
                keys, values = kv_cache[layer_idx]

                # Apply hierarchical quantization
                if keys.size(2) > cache_structure['recent']['tokens']:
                    # Old tokens: aggressive quantization
                    recent_k = keys[:, :, :cache_structure['recent']['tokens'], :]
                    recent_v = values[:, :, :cache_structure['recent']['tokens'], :]

                    old_k = keys[:, :, cache_structure['recent']['tokens']:, :]
                    old_v = values[:, :, cache_structure['recent']['tokens']:, :]

                    # Quantize old tokens to INT4
                    old_k_q, k_scale = quantize_vector(old_k, bits=4)
                    old_v_q, v_scale = quantize_vector(old_v, bits=4)

                    # Reconstruct for attention
                    old_k_r = dequantize_vector(old_k_q, k_scale)
                    old_v_r = dequantize_vector(old_v_q, v_scale)

                    keys = torch.cat([recent_k, old_k_r], dim=2)
                    values = torch.cat([recent_v, old_v_r], dim=2)

        # Standard forward pass with quantized cache
        with torch.no_grad():
            outputs = model(current_ids, past_key_values=kv_cache)
            logits = outputs.logits[:, -1, :]

        # Sample token
        probs = F.softmax(logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        current_ids = torch.cat([current_ids, next_token], dim=1)

    return current_ids

def quantize_vector(x: torch.Tensor, bits: int) -> tuple[torch.Tensor, torch.Tensor]:
    """Quantize vector to N-bit representation."""
    qmin = -(2 ** (bits - 1))
    qmax = 2 ** (bits - 1) - 1

    # Compute scale
    x_max = x.abs().max(dim=-1, keepdim=True)[0]
    scale = x_max / (qmax / 2)

    # Quantize
    x_q = (x / scale).round().clamp(qmin, qmax)

    return x_q, scale

def dequantize_vector(x_q: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:
    """Dequantize vector from N-bit representation."""
    return x_q * scale

2025: EAGLE - Ngram Speculative Decoding​

EAGLE (2024) uses ngram-based drafting without training a separate draft model.

Key approach:

  • Ngram patterns: Leverage frequent token sequences
  • No training: Works with any base model
  • 2-3x speedup: For repetitive text

2026: Gemma 4 Multi-Token Prediction (MTP)​

Gemma 4 (Google, 2026) 采用 Multi-Token Prediction (MTP) 作为推理加速的核心技术，实现最高 3x 推理加速。与传统 speculative decoding 使用独立 draft model 不同，Gemma 4 的 MTP drafter 直接集成在模型架构中。

核心原理：

技术传统 Speculative DecodingGemma 4 MTP
Draft 模型独立的小模型模型内置 MTP head
额外参数完整 draft model轻量级线性层
推理开销需要两次前向传播单次前向传播 + 多头预测
加速比2-3x最高 3x

MTP Drafter 工作流程：

  1. 训练阶段：主模型额外训练多个 prediction head，每个 head 预测未来第 N 个 token
  2. 推理阶段：主模型生成当前 token 的同时，MTP heads 并行预测后续 token
  3. 验证阶段：使用 tree attention 一次性验证所有候选 token
  4. 接受/拒绝：通过 acceptance criterion 决定接受多少 token

与 Medusa 的关键区别：

  • Medusa 的 heads 是事后添加的线性层，MTP heads 与主模型联合训练
  • MTP 利用模型内部隐状态的更深层次信息
  • 在长序列场景下优势更明显

来源：Google Blog - Multi-token-prediction in Gemma 4

2026: DSpark（ds4）- 隐状态感知的 Draft 模型投机解码​

antirez 的 DwarfStar/ds4 本地推理引擎集成了 DeepSeek 为 V4 Flash 发布的辅助 draft 模型 DSpark，展示了投机解码的一个新变体：

与传统 draft 模型的区别：

对比维度传统 Speculative DecodingDSpark
Draft 输入仅已生成的 token 序列直接读取主模型隐状态
每步提议通常 1 个 token最多 5 个未来 token
验证方式主模型逐 token 验证主模型校验，只提交接受的前缀
回退策略全部丢弃重来低置信后缀回退普通解码，主模型始终权威

工程要点：draft 模型"看得见"主模型的内部表示而非仅仅输出序列，使提议质量显著高于 ngram / 独立小模型方案；同时拒绝路径的设计保证生成分布与主模型单独解码完全一致（无损）。配合 2-bit 专家量化 + KV cache 磁盘持久化，M5 Max 128GB 上实现 ~500 t/s prefill、35-40 t/s 解码。

来源：ds4 GitHub（2026-10）

2026: 异步 Continuous Batching​

Hugging Face 于 2026 年 5 月发布 异步 Continuous Batching 技术，将 LLM 推理的 GPU 利用率从 ~75% 提升至接近 100%。

核心原理：

传统 Continuous Batching 中，CPU 准备下一个 batch（tokenization、采样）与 GPU 推理是串行的，约 25% 的 GPU 时间浪费在空闲间隙上。异步 batching 将 CPU batch 准备与 GPU 计算解耦并行：

对比维度同步 Batching异步 Batching
CPU/GPU 关系串行（CPU 准备 → GPU 计算）并行（CPU 准备 ∥ GPU 计算）
GPU 利用率~75%（存在空闲间隙）接近 100%
延迟较高降低
吞吐量基准提升 ~30%

技术实现：

  • CPU 端独立线程处理 prefill 请求的 tokenization 和 KV cache 分配
  • GPU 端持续执行 decode 步骤，不被 CPU 操作阻塞
  • 双缓冲区机制确保数据一致性和无锁交换

来源：Hugging Face Blog - Unlocking Asynchronicity in Continuous Batching（May 14, 2026）

2026-10：SSM 状态管理与全 4-bit 推理​

两条新进展分别解决混合架构的两个 I/O 瓶颈：

  • ReplaySSM——缓存 SSM 输入而非状态（arXiv 2610.13778）：状态空间模型（SSM）每步解码都要读写固定尺寸的循环状态，在混合架构中成为状态 I/O 瓶颈（算术强度低）。ReplaySSM 反转思路：不缓存状态本身，而是缓存喂给状态的输入 token，需要时重放重建——把"每步读写大状态"变成"顺序重放廉价的输入"，显著提高解码阶段的算术强度。
  • DualQuant——免校准的权重+激活双量化（arXiv 2610.13767）：原生 4-bit 加速器要兑现吞吐收益，必须把权重和激活一起压到 4-bit。现有后训练权重量化成熟，但激活量化通常依赖校准数据。DualQuant 用双尺度（dual-scale）方案做到校准无关的权重+激活同时低比特量化，免去逐模型校准流程。

与 DwarfStar 的 2-bit 专家量化对照：那条路线压权重换内存，DualQuant 压激活换计算吞吐，两者正交可叠加——低比特推理的"最后几个瓶颈"正在被逐个拆掉。

来源：arXiv:2610.13778、arXiv:2610.13767（2026-10-09）

Quantization​

Reduce precision to speed up computation and reduce memory.

def quantize_int8(weight: torch.Tensor) -> tuple[torch.Tensor, float, float]:
    """
    Quantize weights to 8-bit integers.

    Args:
        weight: FP32 weight tensor

    Returns:
        int8_weight: Quantized weights
        scale: Scaling factor
        zero_point: Zero point
    """
    # Compute scale and zero point
    qmin, qmax = -128, 127
    fp_min = weight.min().item()
    fp_max = weight.max().item()

    scale = (fp_max - fp_min) / (qmax - qmin)
    zero_point = qmin - fp_min / scale

    # Quantize
    int8_weight = torch.round(weight / scale + zero_point).clamp(qmin, qmax).to(torch.int8)

    return int8_weight, scale, zero_point

def dequantize_int8(int8_weight: torch.Tensor, scale: float, zero_point: float) -> torch.Tensor:
    """Dequantize INT8 weights back to FP32."""
    return scale * (int8_weight.float() - zero_point)

# Quantization formats
quantization_formats = {
    "FP32": {"bits": 32, "range": "Full precision", "speed": "1x"},
    "FP16": {"bits": 16, "range": "Half precision", "speed": "2x"},
    "BF16": {"bits": 16, "range": "Brain float", "speed": "2x"},
    "INT8": {"bits": 8, "range": "Integer quantization", "speed": "4x"},
    "INT4": {"bits": 4, "range": "Aggressive quantization", "speed": "6x"},
}

# Trade-offs: INT8/INT4 lose quality but gain speed
# Typical degradation: INT8 (~1-2%), INT4 (~5-10%)

2025: KV Cache Quantization​

KV cache is the memory bottleneck for long-context inference. Quantizing the cache dramatically reduces memory.

State-of-the-art techniques:

TechniqueStrategyMemory ReductionQuality Impact
CommVQCommutative vector quantization8-16x~1-2%
ShadowKVShadow representation with selective quantization4-8x<1%
KV QuantPer-channel quantization4x~2%
QuantSpecHierarchical quantization4-16x~2%
def commvq_quantize(
    keys: torch.Tensor,
    values: torch.Tensor,
    bits: int = 4
) -> tuple[torch.Tensor, torch.Tensor, dict]:
    """
    CommVQ: Commutative Vector Quantization for KV cache.

    Commutative property allows order-independent quantization,
    enabling better cache compression.

    Reference: https://arxiv.org/abs/2402.10568
    """
    batch_size, num_heads, seq_len, head_dim = keys.shape

    # Split into codebooks
    num_codebooks = 256 // bits  # e.g., 64 codebooks for 2-bit
    codebook_dim = head_dim // num_codebooks

    # Reshape for codebook assignment
    keys_reshaped = keys.view(batch_size, num_heads, seq_len, num_codebooks, codebook_dim)
    values_reshaped = values.view(batch_size, num_heads, seq_len, num_codebooks, codebook_dim)

    # Learn codebooks (during training) or use pre-defined (inference)
    # For inference, we quantize using pre-learned codebooks
    keys_q = []
    values_q = []

    for i in range(num_codebooks):
        k_slice = keys_reshaped[:, :, :, i, :]
        v_slice = values_reshaped[:, :, :, i, :]

        # Find nearest codebook entry
        k_indices = vector_quantization(k_slice, num_codebooks=2**bits)
        v_indices = vector_quantization(v_slice, num_codebooks=2**bits)

        keys_q.append(k_indices)
        values_q.append(v_indices)

    keys_q = torch.stack(keys_q, dim=-1)  # (batch, heads, seq_len, codebooks)
    values_q = torch.stack(values_q, dim=-1)

    metadata = {
        'num_codebooks': num_codebooks,
        'codebook_dim': codebook_dim,
        'bits': bits
    }

    return keys_q, values_q, metadata

def shadowkv_quantize(
    keys: torch.Tensor,
    values: torch.Tensor,
    important_ratio: float = 0.1
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    ShadowKV: Selective quantization based on attention importance.

    Keep important tokens in high precision, quantize the rest.

    Reference: https://arxiv.org/abs/2310.11982
    """
    batch_size, num_heads, seq_len, head_dim = keys.shape

    # Compute attention scores to determine importance
    # (Use last token query to estimate importance)
    last_query = keys[:, :, -1:, :]  # Query from last token
    attn_scores = torch.matmul(last_query, keys.transpose(-2, -1))
    attn_scores = attn_scores.squeeze(-2)  # (batch, heads, seq_len)

    # Determine top-k important tokens
    num_important = int(seq_len * important_ratio)
    _, important_indices = torch.topk(attn_scores, k=num_important, dim=-1)

    # Create importance mask
    importance_mask = torch.zeros_like(attn_scores, dtype=torch.bool)
    importance_mask.scatter_(-1, important_indices, True)

    # Expand mask for all dimensions
    importance_mask = importance_mask.unsqueeze(-1).expand(-1, -1, -1, head_dim)

    # Quantize: keep important tokens in FP16, others in INT4
    keys_q = torch.where(importance_mask, keys, quantize_to_int4(keys))
    values_q = torch.where(importance_mask, values, quantize_to_int4(values))

    return keys_q, values_q

def quantize_to_int4(x: torch.Tensor) -> torch.Tensor:
    """Quantize tensor to INT4."""
    # Scale to [-8, 7] range (INT4)
    x_max = x.abs().max()
    scale = x_max / 8.0
    x_q = (x / scale).round().clamp(-8, 7)
    return x_q * scale  # Dequantize for attention computation

2025: Tree-Based Speculation​

STree, SpecInfer, and TreeAttention use tree structures for parallel speculation.

Key concept: Instead of sequential speculation, speculate multiple branches in parallel and verify them together.

Performance comparison:

MethodParallelismSpeedupUse Case
Sequential Speculative1 branch2-3xGeneral text
Tree-based4-8 branches3-5xDiverse outputs
STreeAdaptive4-6xHybrid SSM-Transformer
def tree_based_speculation(
    model,
    input_ids,
    max_new_tokens=100,
    num_branches=4,
    spec_depth=3
):
    """
    Tree-based speculative decoding with parallel branch verification.

    Speculate multiple continuation paths in parallel, select the best.

    Reference: SpecInfer (https://arxiv.org/abs/2305.09181)
    """
    current_ids = input_ids.clone()

    for step in range(max_new_tokens):
        # Generate multiple candidate branches in parallel
        branches = []

        for branch_idx in range(num_branches):
            # Sample branch with temperature
            branch_tokens = []
            branch_input = current_ids

            for depth in range(spec_depth):
                with torch.no_grad():
                    outputs = model(branch_input)
                    logits = outputs.logits[:, -1, :] / 0.8  # Temperature
                    probs = F.softmax(logits, dim=-1)

                # Sample token
                token = torch.multinomial(probs, num_samples=1)
                branch_tokens.append(token)
                branch_input = torch.cat([branch_input, token], dim=1)

            branches.append(torch.cat(branch_tokens, dim=1))

        # Verify all branches in parallel
        branch_scores = []

        for branch_tokens in branches:
            # Compute full sequence score
            full_sequence = torch.cat([current_ids, branch_tokens], dim=1)

            with torch.no_grad():
                outputs = model(full_sequence)

                # Compute log-likelihood
                logits = outputs.logits[:, :-1, :]  # All but last
                targets = full_sequence[:, 1:]  # All but first

                log_probs = F.log_softmax(logits, dim=-1)
                token_log_probs = torch.gather(log_probs, 2, targets.unsqueeze(-1)).squeeze(-1)

                branch_score = token_log_probs.sum().item()

            branch_scores.append(branch_score)

        # Select best branch
        best_branch_idx = max(range(num_branches), key=lambda i: branch_scores[i])
        best_branch = branches[best_branch_idx]

        # Accept all tokens from best branch
        current_ids = torch.cat([current_ids, best_branch], dim=1)

    return current_ids

2025: SpecPV - Partial Verification for Long Context​

SpecPV (2024) optimizes speculative decoding for long-context generation by partially verifying draft tokens.

Problem: Full verification becomes expensive for long contexts (100k+ tokens).

Solution:

  1. Selectively verify high-impact tokens
  2. Skip verification for low-probability tokens
  3. Batch verification for token groups

Benefits:

  • 1.8-2.2x speedup for 100k+ context
  • Reduces verification compute by 60%
  • Maintains quality with smart token selection

Interactive Parameter Demo​

def compare_decoding_strategies(
    model,
    tokenizer,
    prompt: str,
    strategies: list[dict]
) -> dict[str, str]:
    """
    Compare different decoding strategies on the same prompt.

    Args:
        model: Language model
        tokenizer: Tokenizer
        prompt: Input prompt
        strategies: List of strategy configurations

    Returns:
        Dictionary mapping strategy names to outputs
    """
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    results = {}

    for config in strategies:
        name = config.pop('name', 'unknown')

        output_ids = generate_autoregressive(
            model,
            input_ids,
            max_new_tokens=100,
            **config
        )

        output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
        results[name] = output_text

        # Reset config for next iteration
        config['name'] = name

    return results

# Example comparison
prompt = "Once upon a time"
strategies = [
    {"name": "greedy", "temperature": 0.0},
    {"name": "focused", "temperature": 0.3, "top_p": 0.9},
    {"name": "balanced", "temperature": 0.8, "top_p": 0.9},
    {"name": "creative", "temperature": 1.2, "top_k": 50},
    {"name": "wild", "temperature": 1.5, "top_k": 100},
]

results = compare_decoding_strategies(model, tokenizer, prompt, strategies)

for name, output in results.items():
    print(f"\n{name.upper()}:")
    print(output)

Key Takeaways​

  1. Decoding strategy determines output quality and variety
  2. Temperature controls randomness vs determinism
  3. Top-k and Top-p balance diversity and coherence
  4. KV cache is essential for efficient generation
  5. Quantization trades quality for speed and memory
  6. 2025 inference optimization:
    • Medusa: Multi-head speculation (2.2-3.6x speedup, no draft model)
    • QuantSpec: Quantized KV + self-speculation (2.5x speedup, 75% memory reduction)
    • KV cache quantization: CommVQ/ShadowKV (4-16x memory reduction)
    • Tree-based speculation: Parallel branch verification (3-5x speedup)
    • SpecPV: Partial verification for long context (1.8-2.2x speedup)

2026 补充：Token 效率成为推理经济学核心​

2026 年的推理优化重心从"解码加速"扩展到"少想废话"：

  • Reasoning token 占比失控：推理模型的内部思考常占生成 token 的 90% 以上；多轮 agent 场景下每轮都要回放之前的全部推理，成本随轮数近似二次增长。Fireworks 的 Ember-1（基于 Kimi K3 训练）证明：通过 50+ 训练实验让模型学会"高效思考"，可在保持质量的前提下削减 35–50% 的 token（SWE-bench Verified 92.2% vs K3-max 93.2%，成本降 15–51%）。
  • 调低 reasoning effort 不是等价解：简单调低思考档位会显著掉分；"训练出的高效推理"与"限制推理预算"是两条不同的帕累托曲线。
  • KV cache 内存决定 agent 单价：DeepSeek V4.1-Flash 通过架构手段把长会话 KV cache 压到上一代的 25%，agent 长程任务成本随之骤降——缓存优化已从系统层话题变成模型架构话题。
  • 选型启示：评估模型时除了看 pass rate，还应看 cost/task 与 token/task——同一基准上"同质量、半价"的专门化模型（如 Ember-1、各家 Flash 变体）正成为生产环境的主流选择。

2025 Inference Optimization FAQ​

Q: When should I use Medusa vs traditional speculative decoding?

A: Use Medusa when:

  • You want single-model deployment (no draft model management)
  • Training budget is available (need to train Medusa heads)
  • Want consistent 2.2-3.6x speedup across tasks
  • Easier deployment is prioritized

Use traditional speculative decoding when:

  • You already have a good draft model available
  • No training budget (use pre-trained models)
  • Need maximum compatibility with existing infrastructure
  • Can manage two-model deployment

2025 Verdict: Medusa is becoming the default for new deployments due to simplicity and competitive performance.

Q: What causes KV cache to become a bottleneck?

A: KV cache memory grows linearly with sequence length:

  • Memory = 2 × num_layers × num_heads × seq_len × head_dim × bytes_per_value
  • For Llama 3 70B (80 layers, 64 heads, 128 dim, FP16):
    • 8k tokens: ~2 GB per batch
    • 128k tokens: ~32 GB per batch
    • 1M tokens: ~256 GB per batch (impossible on single GPU)

2025 Solutions:

  1. GQA (Grouped-Query Attention): 4-8x reduction (already in Llama 3)
  2. KV quantization: 4-16x additional reduction (CommVQ, ShadowKV)
  3. SpecPV: Partial verification for long context
  4. Ring Attention: Distribute across multiple GPUs

Rule of thumb: For 32k+ context, always use KV quantization.

Q: How do I choose the right quantization strategy?

A: Decision framework:

Use FP16/BF16 when:

  • Quality is paramount (benchmarking, research)
  • Memory is not constrained (A100/H100)
  • Short context (< 8k tokens)

Use INT8 when:

  • Production deployment with quality tolerance of 1-2%
  • Consumer GPUs (RTX 4090, etc.)
  • Standard context (8k-32k tokens)

Use INT4 when:

  • Maximum throughput required
  • Edge deployment or mobile
  • Can accept 5-10% quality degradation
  • Long context (32k+ tokens)

Use hierarchical quantization (QuantSpec) when:

  • Very long context (100k+ tokens)
  • Need to balance memory and quality
  • Recent tokens need high precision, old tokens can be compressed

2025 Best Practice: Start with INT8, use INT4 only if memory constrained. Consider QuantSpec for 100k+ context.

Q: What is the difference between Medusa-1 and Medusa-2?

A: Both use the same multi-head architecture, but differ in training:

Medusa-1:

  • 5 decoding heads
  • Trained with standard fine-tuning
  • 2.2x speedup
  • ~99% quality retention
  • Faster to train (2-3 hours on 8x A100)

Medusa-2:

  • More heads (typically 8-10)
  • Trained with advanced curriculum learning
  • 3.6x speedup
  • ~98.5% quality retention
  • Longer training (6-8 hours on 8x A100)

When to use which:

  • Medusa-1: Quick deployment, resource-constrained training
  • Medusa-2: Maximum performance, can afford longer training

Production tip: Most deployments use Medusa-1 as the quality-speed trade-off is better for general use.

Q: How does tree-based speculation compare to sequential speculation?

A: Key differences:

Sequential Speculative (traditional):

  • Single branch of speculation
  • Verify tokens one at a time
  • 2-3x speedup
  • Works well for predictable text

Tree-Based (STree, SpecInfer):

  • Multiple parallel branches (4-8)
  • Verify all branches, select best
  • 3-5x speedup
  • Better for diverse/creative outputs
  • Higher compute cost during verification

2025 Trade-off:

  • Sequential: Lower latency, better for real-time chat
  • Tree-based: Higher throughput, better for batch generation

Rule of thumb: Use sequential for interactive chat, tree-based for offline generation (content creation, report writing).

Q: What is CommVQ and why is it important?

A: CommVQ (Commutative Vector Quantization) is a breakthrough in KV cache quantization.

Problem with standard quantization:

  • Order-dependent: quantizing [A, B, C] ≠ [B, A, C]
  • Hard to cache and reuse quantized KV
  • Accumulates errors

Commutative property:

  • Order-independent: quantize([A, B, C]) = quantize([B, A, C])
  • Enables efficient caching and reuse
  • Better error distribution

Benefits:

  • 8-16x memory reduction (4-bit KV)
  • ~1-2% quality loss
  • Faster than naive quantization (cache-friendly)
  • Enables 1M+ token context on single GPU

Adoption: Used in vLLM, TensorRT-LLM for production long-context inference.

Q: How do I implement speculative decoding in production?

A: Implementation checklist:

1. Choose approach:

# For new models
if training_budget and want_single_model:
    use_medusa()  # Best for new deployments
elif have_draft_model:
    use_traditional_speculative()  # Leverage existing models
else:
    use_no_speculation()  # Baseline

2. Configure speculation length:

  • Spec length 4-8: Balanced for most tasks
  • Spec length 2-4: For low-latency (chat)
  • Spec length 8-16: For high-throughput (batch)

3. Monitor acceptance rate:

# Track acceptance rate during inference
acceptance_rate = accepted_tokens / total_draft_tokens

# Target: 60-80% acceptance rate
if acceptance_rate < 0.5:
    # Reduce spec length or improve draft model
    spec_len = max(2, spec_len - 2)
elif acceptance_rate > 0.9:
    # Can increase spec length for more speedup
    spec_len = min(16, spec_len + 2)

4. Production libraries:

  • vLLM: Best for production (Medusa, PagedAttention, KV quantization)
  • TensorRT-LLM: NVIDIA optimized (INT8/INT4, speculative)
  • llama.cpp: CPU/Metal inference (quantization focused)

2025 recommendation: Start with vLLM + Medusa for most GPU deployments.

Q: What is the impact of sampling parameters on inference speed?

A: Sampling parameters affect speed through:

Temperature:

  • Low (0.1-0.3): Faster (greedy-like), less diverse
  • High (1.0-1.5): Slower (more computation), more diverse
  • Impact: ~10-20% speed difference

Top-k:

  • Small (1-10): Faster to compute softmax
  • Large (50-100): More tokens to consider
  • Impact: ~5-10% speed difference

Speculative compatibility:

  • Greedy/temp=0: Perfect for speculation (80-90% acceptance)
  • High temp: Lower acceptance rate (50-60%)

2025 optimization tip: For maximum speed with speculation, use temperature=0.3-0.5, top_k=40. This balances speed and quality.

Q: How do I debug inference performance issues?

A: Debugging checklist:

1. Profile first:

import torch.profiler as profiler

with profiler.profile activities=[profiler.ProfilerActivity.CPU, profiler.ProfilerActivity.CUDA]) as prof:
    output = model.generate(input_ids, max_new_tokens=100)

print(prof.key_averages().table(sort_by="cuda_time_total"))

2. Check common bottlenecks:

  • Memory bound: KV cache too large → Use quantization
  • Compute bound: Model too large → Use quantization/smaller model
  • IO bound: Data loading → Prefetch inputs
  • CPU bottleneck: Tokenization → Use GPU tokenization (experimental)

3. Enable optimizations:

# Flash Attention 2 (automatic in most frameworks)
model.config.use_flash_attention_2 = True

# Compile model (PyTorch 2.0+)
model = torch.compile(model)

# Use KV cache
model.config.use_cache = True

# Enable speculation if available
model.config.speculative_decoding = True

4. Monitor metrics:

  • Tokens/second: Should be > 50 for 7B model on A100
  • Memory usage: Should be < 80% GPU memory
  • Cache hit rate: > 95% for KV cache
  • Speculation acceptance: > 60% for good speedup

2025 tools:

  • vLLM profiler: Built-in performance analysis
  • TensorBoard: Track metrics over time
  • Nvprof: NVIDIA GPU profiler

Spring AI Streaming Implementation​

Spring AI provides streaming support for real-time LLM responses using Server-Sent Events (SSE) and reactive programming.

Server-Sent Events (SSE) Streaming​

// SSE Streaming with Spring AI
@RestController
@RequestMapping("/api/chat")
public class ChatController {
    private final ChatClient chatClient;

    @GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> streamChat(@RequestParam String message) {
        return chatClient.prompt()
            .user(message)
            .stream()
            .content();  // Returns Flux<String> for streaming
    }
}

Reactive Streaming Service​

// Service for streaming LLM responses
@Service
public class StreamingChatService {
    private final ChatClient chatClient;

    // Stream responses to frontend
    public Flux<String> streamResponse(String userMessage) {
        return chatClient.prompt()
            .user(userMessage)
            .stream()
            .content()
            .doOnNext(token -> log.debug("Received token: {}", token))
            .doOnComplete(() -> log.info("Stream completed"))
            .doOnError(error -> log.error("Stream error", error));
    }

    // Stream with progress tracking
    public Flux<StreamingResponse> streamWithProgress(String userMessage) {
        AtomicInteger tokenCount = new AtomicInteger(0);

        return chatClient.prompt()
            .user(userMessage)
            .stream()
            .content()
            .map(token -> {
                int count = tokenCount.incrementAndGet();
                return new StreamingResponse(token, count, false);
            })
            .concatWithValues(StreamingResponse.done());
    }

    // Record for streaming response
    public record StreamingResponse(
        String token,
        int tokenCount,
        boolean isDone
    ) {
        public static StreamingResponse done() {
            return new StreamingResponse("", 0, true);
        }
    }
}

Parameter Comparison Examples​

// Effect of sampling parameters on output
@Service
public class ParameterComparisonService {
    private final ChatClient chatClient;

    // Code generation: Low temperature for consistency
    public String generateCode(String description) {
        return chatClient.prompt()
            .user("Write code to: " + description)
            .options(OpenAiChatOptions.builder()
                .temperature(0.2)  // Low = more deterministic
                .topP(0.95)
                .maxTokens(1500)
                .build())
            .call()
            .content();
    }

    // Creative writing: Higher temperature
    public String generateStory(String prompt) {
        return chatClient.prompt()
            .user("Write a story about: " + prompt)
            .options(OpenAiChatOptions.builder()
                .temperature(0.9)  // High = more creative
                .topP(0.9)
                .maxTokens(2000)
                .presencePenalty(0.5)  // Encourage new ideas
                .build())
            .call()
            .content();
    }

    // Summarization: Low temperature, focused
    public String summarize(String text) {
        return chatClient.prompt()
            .user("Summarize: " + text)
            .options(OpenAiChatOptions.builder()
                .temperature(0.3)  // Focused
                .topP(1.0)  // Consider all tokens
                .maxTokens(500)
                .build())
            .call()
            .content();
    }
}

Streaming with Backpressure​

// Handle backpressure for slow clients
@Service
public class BackpressureStreamingService {
    private final ChatClient chatClient;

    public Flux<String> streamWithBackpressure(String message, int bufferSize) {
        return chatClient.prompt()
            .user(message)
            .stream()
            .content()
            .onBackpressureBuffer(BufferOverflowStrategy.DROP_OLDEST)
            .doOnNext(token -> log.debug("Emitted token: {}", token));
    }

    // Rate-limited streaming
    public Flux<String> streamRateLimited(String message, int tokensPerSecond) {
        Duration delay = Duration.ofMillis(1000 / tokensPerSecond);

        return chatClient.prompt()
            .user(message)
            .stream()
            .content()
            .delayElements(delay);
    }
}

WebSocket Streaming​

// WebSocket streaming for bidirectional communication
@Controller
public class WebSocketChatController {
    private final StreamingChatService chatService;

    @MessageMapping("/chat")
    public Flux<String> handleChatMessage(String message) {
        return chatService.streamResponse(message)
            .doOnNext(token -> log.debug("Sending token: {}", token));
    }
}

推理效率新进展（2026-10）​

解码阶段的延迟与显存本质上是访存带宽问题，arXiv 上三条新工作从不同侧面压缩解码成本：

  • VFold：对称性感知的跨层 value cache 压缩（arXiv 2610.12338）——KV cache 在长上下文下主导显存，VFold 利用层间 cache 相似性做 value 合并，无需改动模型架构、开销远低于既有跨层压缩方法，长上下文 agent 会话的显存账单直接受益
  • SparseDecoding：解码感知的剪枝（arXiv 2610.12327）——传统 Hessian 引导的免训练剪枝用预收集校准集估计重要性，与真实解码分布脱节；SparseDecoding 用解码期统计指导逐层剪枝，剪枝收益与延迟降低对齐得更好
  • Latent Core Tokenizer：压缩但要公平（arXiv 2610.12376）——tokenizer 只追求压缩率会把词表容量倾斜向高资源语言；LCT 用 MDL + 熵边界信号 + 形态句法约束把"结构发现"与"词表构建"分离，多语言场景下同等词表更均匀的能力分布

共同主题：推理经济学从"换更快的硬件"转向"在算法与数据结构层面重新设计解码路径"——与决策模型（单次前向、有界输出）的兴起同属一个趋势：每 token 的成本正在成为架构决策，而非采购决策。

2026-10-15：KV 潜在压缩、混合线性注意力投机解码与"弹性"Transformer​

新一轮 arXiv 工作把解码成本压缩推进到注意力算子内部：

  • Latent Compaction with Test-Time Training（arXiv 2610.20582）——长上下文 KV cache 压缩必须在"没见到问题"时就决定保留什么。该方法为每篇文档学习一个紧凑的潜在 cache，测试时训练（TTT）让压缩保留"未来可能被问到"的信息，而非通用摘要——agent 长会话的显存账单与检索精度同时受益
  • ReaDy：混合线性注意力的投机解码（arXiv 2610.20503）——特征式 drafter 依赖目标模型隐藏状态，但线性注意力模型的 recurrent state 强调的上下文未必贴合后续 draft 位置的需要；ReaDy 用循环状态作为动态上下文为每个 draft 位置重组条件，混合架构（hybrid linear attention）上投机解码的有效性显著改善
  • Your Transformer Is Secretly Elastic（arXiv 2610.20588）——模型家族只给 S/B/L 几个固定档位，逼用户在硬件与负载之间提前"押注"。该工作指出单个训练好的 transformer 内部天然存在可弹性伸缩的子结构，无需为每个尺寸各训一遍——推理侧"按需取用模型大小"从量化/剪枝走向架构原生
  • Continuum Attention（arXiv 2610.20586）——把注意力的离散 key/value 查找替换为对连续分段线性记忆曲线的积分，每个 query 沿曲线诱导指数相关性密度——注意力算子本身的连续化重写，为超长上下文提供新的数学形态

共同方向：KV cache、draft 模型、模型尺寸、注意力算子——推理栈的每一层都在从"离散固定"走向"连续可调"；当 Muse Spark Open 这类模型能在桌面统一内存上运行（见 Frontier 趋势 10-15 条目），推理效率的竞争主战场正从数据中心向端侧蔓延。

2026-10-16：2-bit 专家权重进 24GB 与"按步路由"的成本分层​

  • Nemotron 3.5 Bolt 74B-A9B：Nvidia 用 2-bit 专家权重把 74B 总参 / 9B 激活的 MoE 塞进 24GB 显存，RTX Spark 上约 84 tok/s（低于 Qwen3.6 与 Gemma 4 同表数字）——激进量化 + MoE 稀疏激活的组合让"70B 级模型上工作站"成立，代价是解码速度让位；同期 Nemotron 3.5 Super VL（120B/12B 激活）开放权重主打多模态 agent
  • NVIDIA EdgeWise：按步路由：训练本地 Qwen3.5-9B 判断哪些步骤"关键"、只把这些步骤委托给 GPT-5.5 / Opus 4.8——SWE-ReBench 云成本 -62%、Terminal-Bench 2.1 -34%，Pass@3 基本不变。与 Bolt 的"压缩单模型"不同，这条路线是在模型间路由：推理成本优化从"让一个大模型更便宜"扩展为"每步用恰当大小的模型"
  • Anthropic RL 训练对齐审计员：用成对奖励 RL 训练 Haiku 4.5 审计 agent，审计质量 72.7 对 Opus 4.6 的 68.0（简单奖励先被 reward-hack）——小模型 + RL 专化在"评估类"工作负载上替代大模型的又一证据，与腾讯 Hy-Memory-30B-A3B（专职写 agent 记忆，见 Frontier 10-16 条目）同属专职小模型替代通用大模型的趋势

共同主题：推理经济学从"单模型压缩"（量化/KV cache）走向两个新自由度——垂直方向压权重（2-bit 专家）、水平方向路由步骤（本地小模型 + 按需上云）。"每一步该花多少钱"正在成为 agent 架构里的显式设计参数。

2026-10-17：DeepSeek-V4.1-Flash-1017 开源权重刷榜 agent 基准，端侧推理下探到手机 Neural Engine​

  • DeepSeek-V4.1-Flash-1017（MIT 权重，HN 500 分）：V4.1-Flash 的"minor update"新快照——552B backbone + 196B Engram 共约 763B 总参，prefill 仅激活 8B、decode 激活 16B，1M 上下文 / 384K 输出。自报 11 项基准全线上扬 2.0-11.1 分：DeepSWE v1.1 77.0（上一版 74.2，对比表里 Opus 5 为 74.0、GPT-5.6 Sol 73.0），Automation-Bench 60.5、Agents' Last Exam 35.4、Apex Bench 72.1 均居对比表首位；但 HLE（无工具）42.6 仍明显落后 Opus 5 的 56.3，Terminal-Bench 4.0 42.3 也低于 51.8——**"agentic 任务平价、深度推理仍有差距"**的分层格局。成本侧据 The Decoder 引用的官方图表：达到 77.0 DeepSWE 分数的每任务成本约 **0.20∗∗，上一版约0.20**，上一版约 0.50；Vercel AI Gateway 定价 0.15/M输入、0.15/M 输入、0.60/M 输出
  • 集成成本被低估：1017 不带 Jinja chat template——prompt 格式化由 Python 参考实现（encoding.py）+ Rust 库 deepseek-recipe（Python 绑定）承担，serving 栈必须先适配这套协议；约 510GB 的 FP8/FP4 混合分片也让"开源"与"可自托管"之间隔着不小的工程距离。所有分数均为厂商自报（Terminal-Bench 用 Claude Code harness、DeepSWE 用 Mini-SWE-V2），尚无独立复现
  • Qwen3.8-27B 跑进 iPhone 18 Neural Engine（HN）：1.5 bpw 极限量化下 15–70 tok/s——端侧推理的主战场从"Mac 统一内存"进一步下探到手机 NPU，与桌面级 510GB 的 DeepSeek 形成"云端巨兽 vs 口袋小模型"的两极

共同主题：开源权重阵营在 agent 基准上已能对闭源旗舰打"局部平价 + 十分之一成本"的性价比战争，但协议/工程集成成本（无 Jinja 模板、Engram 分片重建）正在成为开源模型落地的新摩擦面。

2026-10-21：扩散/流式语言模型冲进实用区间，投机解码与 MTP 逼近理论边界​

  • Inception Mercury 3（扩散 LM，1,068 tok/s + 1M 上下文）：整块 token 并行精炼而非逐 token 自回归，官方宣称比 GPT-6 Luna / Claude Haiku 5.5 快 4-8 倍且基准反超（自报、未独立复现）。若成立，"循环里跑的模型"（路由器、分类器、摘要器）的最优解第一次不是自回归模型——速度/成本轴对纯智能轴的替代在低价位段成真
  • Large Flow Language Models（arXiv 2610.28559）：连续流式语言模型为并行生成提供离散扩散所没有的控制与加速工具箱——并行解码的另一条技术路线从论文走向规模化配方
  • Adaptive Multi-Token Prediction（arXiv 2610.28619）：MTP 必须在起草前固定投机长度，而最优长度随上下文变化；该工作给出自适应 horizon 的决策框架——投机解码的收益上限取决于"何时停"，与 DeepInfer 的置信度调度验证（见 10-20 前沿条目）同题
  • BitMTP（arXiv 2610.28500）：MTP 与超低位（2-bit 级）骨干结合——投机解码与极限量化两条省钱路线首次正交叠加
  • Qwen3.6 27B（Alibaba，10-21 发布）：开源权重阵营本周第三个新模型（继 MiniMax M3.1 Flash、DeepSeek V4.1 Flash 1017），27B 档位继续下探端侧可跑的旗舰能力

共同主题：推理竞争的坐标系正在换轴——从"谁更聪明"到"每美元、每秒多少可用智能"；并行生成（扩散/流式）+ 投机解码自适应化 + 低位量化三者叠加，可能把循环型 agent 工作负载的成本再压一个数量级。

2026-10-22：循环化扩散 LM、零数据长上下文 RL 与训练栈多硬件原生化​

  • Looped Uniform Diffusion Language Models（arXiv 2610.30066）：均匀扩散语言模型（UDLM）把训练目标改为循环化（looped）形式后生成质量显著提升——并行生成的扩散路线继续补训练侧拼图，紧接 10-21 的 Mercury 3 与流式 LM 配方
  • LongZero（arXiv 2610.30132）：长上下文推理的 RL 训练受限于"长、多样、难、可验证"语料稀缺——该工作从零合成符合全部四个条件的训练数据，绕开数据瓶颈。长上下文能力的竞争从"谁有数据"转向"谁会合成数据"
  • AWS TorchNeuron / Google TorchTPU：原生 PyTorch 直达 Trainium 与 TPU，训练栈正在把 CUDA 中间层挤薄——与 OpenAI 模型登陆 Google Cloud Model Garden（GPT-6 Astra/Sol/Luna 全系上架）同周：模型与算力的排他绑定继续解体，推理侧的硬件议价格局随之变化

共同主题：并行生成补训练目标课、长上下文 RL 数据合成化、训练栈多硬件原生化——推理成本曲线的三个输入端同时被压。

2026-10-23：思想剪枝、投机解码检索化、扩散 LM 提交策略与低位 n-gram 嵌入​

  • Direct Pass@k Optimization for Thought Pruning（arXiv 2610.31667）：test-time scaling 把全部候选解跑完代价高昂——直接对 pass@k 做思想剪枝优化，在质量与多样性之间平衡，砍掉不值得跑完的推理链。推理预算分配从"全部跑完再选"走向生成中途的在线剪枝
  • ValuSpec（arXiv 2610.31338）：树状投机解码中，相似请求反复诱导重叠的候选分支——在目标验证前先做候选估值（plug-and-play），实践服务场景下重复分支的浪费被检索式复用消掉
  • Learned Unmasking in Diffusion LMs（arXiv 2610.31576）：掩码扩散语言模型的效率取决于"提交哪些预测"——对学习型 unmasking 策略做受控研究，厘清监督信号、输入、容量各自贡献多少速度-精度收益；扩散 LM 实用化的下一块拼图（接 10-21/10-22 的 Mercury 3、循环化 UDLM）
  • Low-Bit N-Gram Embeddings（arXiv 2610.31646）：开源权重模型流行的 n-gram 嵌入表占内存不小——低位化后保住性能，嵌入表也能省一个数量级
  • Kimi K3.1 Pro Highspeed（产品侧）：同代模型的 2-3 倍推理速度变体——"高速版"作为独立 SKU 出现，说明推理成本已直接进入产品定价结构（详见 frontier 章动态）

共同主题：推理经济学从"更大 batch、更快 kernel"深入到生成过程内部的预算分配（哪些思想值得跑完、哪些 token 值得提交、哪些嵌入值得存满精度）。

2026-10-24：循环架构 KV 共享、因果 CoT 隔离与自蒸馏指纹​

  • BeNice: KV-Sharing for Looped Architectures（arXiv 2610.31633）：循环化架构（同一组层反复过）推理时每次循环生成独立 KV cache，显存随循环数线性膨胀——通过跨循环 KV 共享闭合"质量-显存"缺口。循环化正是本周扩散/循环 LM 路线（Mercury 3、循环化 UDLM）的通用骨架，它的推理显存问题现在有了第一个系统性答案
  • Causal Chain of Thought（arXiv 2610.31639）：标准自回归 CoT 里稠密注意力允许每个推理步"偷看"完整上下文，推理链表面合理、实际并未因果驱动最终答案——通过上下文隔离（限制每步可见范围）迫使推理步真实承担因果作用。对"CoT 到底是不是真推理"的可解释性审计给出一个工程抓手
  • OP-Print（arXiv 2610.31616）：on-policy 自蒸馏会在模型内部留下训练成员指纹（membership imprint）——自蒸馏不再是"无风险的免费午餐"，数据溯源与模型指纹的攻防延伸到蒸馏管线
  • Cost-Aware Abstention（arXiv 2610.31657）：以 IT 工单路由为例做成本感知弃权——在标定集上按显式代价选好弃权策略再冻结部署。弃权（abstain）从模型属性变成按业务代价离线调优的决策策略

共同主题：推理研究的对象从"单次前向"扩展到整个生成与训练管线的副作用——循环的显存、CoT 的因果性、蒸馏的指纹、弃权的代价，都是"多想/多练"的隐藏账单。

2026-10-25：晶圆级部署万亿参数模型与未来 token 泄漏审计​

  • Cerebras 晶圆级部署 3T 参数 Kimi K3（HN 321 分）：单块 WSE-3T 晶圆只有 44GB 片上内存，而万亿参数模型仅权重就需要 TB 级——Cerebras 公开完整部署数学：4-bit 权重 + 8-bit KV 下每层约 16GB；线性注意力层无需 KV cache（每请求只存固定大小状态），一块晶圆放 2 层；稠密注意力层要留 KV cache 空间，一块晶圆放 1 层——K3 的 69 层线性 + 24 层稠密 = 35 + 24 共 59 块晶圆流水线，晶圆间只传激活值（RDMA over Ethernet，约 2μs/跳），实现 1000-2000+ tok/s。这是线性注意力第一次在专用硬件部署经济学上给出完整算例：混合架构里"哪种注意力层放多少"直接决定硬件账单
  • Beyond Causal Masks（arXiv 2610.31522）：系统审计自适应序列模型中隐藏的未来 token 泄漏——泄漏会让模型在 teacher-forced 评估中虚高、free-running 生成时退化，制造虚假的架构收益；给出定位泄漏层的方法。接昨天的因果 CoT：评估与生成的间隙正在成为推理研究的独立对象
  • Advantage-Based On-Policy Distillation（arXiv 2610.31563）：在线策略蒸馏（OPD）优于 SFT 的同时引入新噪声——标准 OPD 让学生"每步都听教师"，本文改为按优势值决定何时跟教师：教师信号只在高优势轨迹段生效。蒸馏从"全盘模仿"进化为带信用分配的选择性学习
  • Bilingual LM 阅读预测（arXiv 2610.31620）：单语 LM surprisal 无法刻画二语阅读中的跨语言迁移，双语模型给出更好拟合—— surprisal 作为"计算困惑度"的行为预测工具，正从单语走向多语心理语言学

共同主题：部署经济学的粒度从"集群级"下探到**"层类型级"（线性 vs 稠密注意力各占多少晶圆），而评估的严谨性从"分数真假"延伸到"架构收益真假"**——泄漏审计与选择性蒸馏都在回答同一个问题：收益到底来自哪里。

2026-10-26：投机解码进入 RL 训练与"舍入位置"保 on-policy​

  • NgramRL（arXiv 2610.32881）：RL 后训练的成本被自回归 rollout 生成支配（常超 70% 训练时间）——用 rollout 感知的 n-gram 投机解码加速采样而不破坏训练信号。投机解码从"推理加速"正式进入"训练加速"：RL rollout 是它第二个主场
  • Where Rounding Happens Matters（arXiv 2610.32951）：推理与训练通常跑在两套系统上——即使权重相同、同为 BF16，舍入位置的实现差异也会让"名义 on-policy"的 RL 更新悄悄变成 off-policy；对齐舍入位置即可稳定训练。接 10-20 DeepInfer 的 batch-invariant 推理：浮点细节的可复现性正在成为训练与推理两侧的正式工程对象
  • Ubicloud 修复 vLLM 工具调用准确性（HN 125 分）：生产 agent 的第一线问题正在从"生成质量"移到**"工具调用 fidelity"**——解析器/模板/采样参数的组合让同一模型在不同栈上工具调用成功率差异巨大，推理引擎的工程细节直接决定 agent 可靠性
  • Green Code or Costly Code?（arXiv 2610.32891）：LLM 迭代式代码优化的能耗审计——执行变快不等于能耗变低，多轮优化本身的能耗常吃掉优化收益。算力承诺动辄千亿的当下，"优化"本身需要能耗核算

共同主题：今天四条都在讲基础设施的诚实时刻——rollout 到底花了多少时间（NgramRL）、更新到底是不是 on-policy（舍入位置）、工具调用到底成没成功（vLLM）、优化到底省没省电（能耗审计）。收益的粒度越细，越需要把每一层的账算清。

2026-10-27：KV 池容量的租户隔离代价、MXFP8 端到端 RL 与扩散 LM 的"何时提交"​

  • Cloudflare GLM-5.3 跨请求 KV cache 泄露（HN 175 分）：PD 分离架构下，**KV 池（681,280 token）小于上下文窗口（1,048,576 token）**的配置错配触发了 SGLang 的错误路径 bug：放不下的超长请求被替换成单 token 占位符，prefill 只传了占位符的 KV，decode 误以为完整覆盖，读着前一请求的残留缓存页生成了含其他用户数据的回答（155 条响应受影响）。修复双管齐下：池扩到上下文 2 倍（204 万 token）+ 拒绝路径在 disaggregation 准入队列提前退役（任何 KV 传输发生之前），并计划对 RDMA/Mooncake KV 传输协议做形式化验证。工程启示：PD 分离把 KV cache 从"引擎内部状态"变成"跨节点传输的分布式数据"，一致性 bug 的爆炸半径从崩溃升级为跨租户数据泄露；"池 ≥ 窗口"必须是不变式而非配置约定
  • 端到端 MXFP8 强化学习（arXiv 2610.34915）：在 verl（Megatron/TE 训练 + vLLM/SGLang rollout）上实现 Blackwell B200 的全栈 MXFP8 RL——低精度不只是便宜，RL 还要求 sampler 与 learner 数值一致，matching 何时有效、何时失效给出实测。接昨天"舍入位置"与 DeepInfer 的 batch-invariant：训练-推理一致性的精度粒度从 BF16 下探到 FP8 块格式
  • Learning When to Commit（arXiv 2610.35804）：扩散 LM 可并行生成多 token 但需要多轮解码——高效生成取决于提交哪些 token 才最能帮助预测剩余位置（高置信 ≠ 高信息量），提出 schedule distillation 学习"何时提交"。扩散 LM 的效率优化从"并行多少"进化为"提交什么"
  • Constrained Decoding: Harmful or Beneficial?（arXiv 2610.35328）：把约束解码对 LLM 性能的影响按三类约束（格式/有效性/域约束）重新分类测量——"约束解码天然损害推理"这个默认假设并不总成立，影响取决于约束类型与任务的结构关系。对把 structured output 当默认配置的 agent 栈是一份必要的校正

共同主题：四条都在给推理栈的"默认假设"做体检——池装得下（Cloudflare）、低精度不改变策略（MXFP8）、置信度高就该提交（dLLM）、约束解码伤推理（taxonomy）。默认假设每被显式检验一次，生产就安全一分。

References​

Radford, A., Wu, J., Child, R., et al. (2019). "Language Models are Unsupervised Multitask Learners." OpenAI technical report.

Link: https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf

GPT-2 paper introducing nucleus sampling and decoding strategies.

Holtzman, A., Buys, J., Du, L., et al. (2020). "The Curious Case of Neural Text Degeneration." ACL 2020.

Link: arXiv:1904.09751

Analysis of repetition and degeneration in neural text generation.

Levi, D., Khashabi, D., & Roth, D. (2023). "Speculative Decoding: Accelerating LLM Inference." arXiv preprint.

Link: arXiv:2211.17192

Introduction of speculative decoding for faster inference.

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2024). "QLoRA: Efficient Finetuning of Quantized LLMs." ICLR 2024.

Link: arXiv:2305.14314

Quantization-aware training for large language models.