vllm-omni/docs/design/feature/expert_parallel.md

# Expert Parallel

This section describes how to add Expert Parallel (EP) to a diffusion transformer that uses Mixture-of-Experts (MoE) layers.
We use **HunyuanImage3.0** as the reference implementation.

---

## Table of Contents

- [Overview](#overview)
- [Step-by-Step Implementation](#step-by-step-implementation)
- [Testing](#testing)
- [Reference Implementations](#reference-implementations)
- [Summary](#summary)

---

## Overview

### What is Expert Parallel?

**Expert Parallel** is a parallelism strategy in Mixture-of-Experts (MoE) models that distributes different expert networks across distinct computational devices. Each device holds and computes only a subset of experts (local experts), with tokens dispatched to and gathered from remote devices via collective communication operations (e.g., All-to-All, All-Gather).

| Backend | Description |
|---------|-------------|
| `allgather_reducescatter` | Default backend based on allgather/reducescatter primitives, suitable for general EP+DP deployments.|

## Configuration

Enable EP by setting the `--enable-expert-parallel` flag. The EP size is automatically calculated as:

```text
EP_SIZE = TP_SIZE × SP_SIZE × CFG_SIZE × DP_SIZE
```


Where:

- `TP_SIZE`: Tensor parallel size
- `SP_SIZE`: Sequence parallel size
- `CFG_SIZE`: Classifier-free guidance parallel size
- `DP_SIZE`: Data parallel size
- `EP_SIZE`: Expert parallel size (computed automatically)

Note:
- Expert parallelism is only applicable to Mixture-of-Experts (MoE) models.
- The EP group is created **per pipeline stage**, meaning it includes all ranks that participate in model parallelism except pipeline parallelism.
- The underlying communication pattern for expert parallelism is **All-to-All** among the ranks in the EP group.

For example, consider a configuration with `TP=2`, `SP=1`, `CFG=2`, and `DP=4` (total 2×1×2×4 = 16 GPUs).

- Expert layers are handled by an EP group of size 16.

- Attention layers use tensor parallelism of size 2 within each of the 8 DP groups (because `DP×CFG×SP = 4×2×1 = 8` groups, each containing the 2 TP ranks). Inside each such group, the attention weights are sharded across the 2 GPUs.


## Step-by-Step Implementation

### Step 1: Configure Expert Parallelism Settings

Calculate local experts per rank:

```
ep_size = 8  # Expert Parallel size (typically equals TP size)
num_experts = 64
num_local_experts = num_experts // ep_size  # 8 experts per card

# Check divisibility
assert num_experts % ep_size == 0, "Experts must be divisible by EP size"
```

### Step 2: Use Sparse MoE Block to enable EP routing.

Example:
```
from vllm.model_executor.layers.linear import ReplicatedLinear
class HunYuanSparseMoeBlock(nn.Module):
    def __init__(
        self,
        config: PretrainedConfig,
        layer_id: int = -1,
        prefix: str = "",
    ):
        super().__init__()
        self.tp_size = get_tensor_model_parallel_world_size()
        self.n_routed_experts = config.num_experts  # 64

        # Calculate local experts per rank (key for EP)
        if self.tp_size > self.n_routed_experts:
            raise ValueError(f"TP size {self.tp_size} > experts {self.n_routed_experts}")

        # Routing gate (replicated on all ranks, computes scores for all tokens to all experts)
        self.gate = ReplicatedLinear(
            config.hidden_size,
            config.num_experts,
            bias=False,
            quant_config=None,
            prefix=f"{prefix}.gate",
        )

        # EP expert layer (factory loads platform-specific implementation)
        self.experts = HunyuanFusedMoE(...)
```
**Key Points:**
- gate is **ReplicatedLinear** (replicated on all ranks)
- experts is created via **HunyuanFusedMoE factory**, which automatically handles EP dispatch

### Step 3: Initialize EP Runtime

Initialize the EP communication context before model loading.
```
from vllm.utils.import_utils import resolve_obj_by_qualname
# Call during __init__ or model loading
op_name = "hunyuan_fused_moe"

# Prepare EP runtime: establish communication groups, assign local expert indices, init _expert_map
current_omni_platform.prepare_diffusion_op_runtime(op_name)

# Factory automatically resolves platform implementation (GPU: FusedMoE / NPU: AscendFusedMoE)
impl = resolve_obj_by_qualname(
    current_omni_platform.get_diffusion_model_impl_qualname(op_name)
)
```

### Step 4: Expert Weight Mapping & Loading

Each rank loads only the expert weights assigned to its local allocation.
```
# Get expert parameter mapping (different per rank)
expert_mapping = HunyuanFusedMoE.make_expert_params_mapping(
    model=self,
    ckpt_gate_proj_name="gate_proj",
    ckpt_down_proj_name="down_proj",
    ckpt_up_proj_name="up_proj",
    num_experts=64,
    num_redundant_experts=0,
)
# Returns: [(param_name, weight_name, expert_id, shard_id), ...]
# Note: Each rank only contains mappings for its local expert_ids

# Filter non-local experts during loading
for name, loaded_weight in weights:
    if "mlp.experts" in name:
        # Parse expert_id from weight name (implementation needed)
        expert_id = parse_expert_id_from_name(name)
        local_expert_start = (ep_rank) * num_local_experts
        local_expert_end = (ep_rank + 1) * num_local_experts

        if not (local_expert_start <= expert_id < local_expert_end):
            continue  # Skip non-local expert weights
```
### Step 5: Forward Pass with EP

Example (MoE Forward):
```
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
    orig_shape = hidden_states.shape
    hidden_states = hidden_states.view(-1, hidden_states.shape[-1])

    # 1. Global routing computation (all tokens, all expert scores)
    # hidden_states: [num_tokens, hidden_dim] (full tensor)
    router_logits, _ = self.gate(hidden_states)  # [num_tokens, num_experts]

    # 2. EP dispatch and compute (HunyuanFusedMoE handles all_to_all internally)
    # - Dispatch: Send tokens to target ranks based on router_logits
    # - Local Compute: Each rank processes only its num_local_experts
    # - Combine: Results returned to original token positions
    final_hidden_states = self.experts(
        hidden_states=hidden_states,
        router_logits=router_logits,
    )

    # 3. Add shared expert output (not EP, computed on all ranks)
    if self.shared_mlp is not None:
        shared_out = self.shared_mlp(hidden_states)
        final_hidden_states = final_hidden_states + shared_out

    # 4. Tensor Parallel All-Reduce (synchronize across TP group)
    if self.tp_size > 1:
        final_hidden_states = self.experts.maybe_all_reduce_tensor_model_parallel(
            final_hidden_states
        )

    return final_hidden_states.view(orig_shape)
```

## Testing
After adding Expert Parallel support, test via command line:
```bash
cd examples/offline_inference/text_to_image
python text_to_image.py \
    --model Your-org/your-model \
    --prompt "a cup of coffee on the table" \
    --output "ep_enabled.png" \
    --num-inference-steps 50 \
    --guidance-scale 5.0 \
    --tensor-parallel-size 8 \
    --seed 1234 \
    --enable-expert-parallel
```

vLLM‑Omni currently focuses on core diffusion model inference acceleration, so the Expert Parallel implementation includes only the basic multi‑GPU expert sharding functionality (enabled via --enable-expert-parallel). Advanced features such as communication backend selection (--all2all-backend), load balancing (--enable-eplb and its configuration), and multi‑node deployment belong to the extended capabilities of the main vLLM project and have not yet been integrated into Omni.

## Reference Implementations

Complete examples in the codebase:

| Model | Path | Pattern | Notes |
|-------|------|---------|-------|
| **HunyuanImage3.0** | `vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py` | Standard EP | Full implementation with validation |
| **EP Tests** | `vllm-omni/tests/e2e/offline_inference/test_expert_parallel.py` | E2E testing | EP correctness and performance |
| **Constraint Tests** | `vllm-omni/tests/diffusion/models/hunyuan_image3/test_hunyuan_fused_moe.py` | Unit testing | Validation logic |

---
## Summary

Adding Expert Parallel support to diffusion model:

1. **Identify MoE layers** - Locate the router and expert networks in each transformer block.
2. **Validate EP constraints** – Ensure num_experts is divisible by expert_parallel_size.
3. **Test** - Run with enable-expert-parallel, check memory reduction, speedup, and output quality against single‑GPU baseline.