[edge-infer] WIP snapshot: uncommitted M0/W1/W4/W5 edge work as of 2026-09-18
Verbatim copy of the edge/omni-edge working tree (tracked diffs + untracked edge/local, tests/edge/local, docs/edge, pipeline files). Taken so that vLLM-Omni v0.29.0rc1 can be merged on a branch without touching the user's working tree. logs_edge/ and benchmark .npy outputs excluded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
3834de3105
commit
4e3e8a1ebc
62 files changed
+11043
-56
No files matched your search
@@ -0,0 +1,43 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Spark-X2.5 as an Omni pipeline: one autoregressive text stage.
|
||||
|
||||
Spark's model definition has been in ``vllm_omni.model_executor.models`` for a
|
||||
while, but only as a vLLM architecture -- there was no pipeline, so
|
||||
``Omni(model=<a Spark checkpoint>)`` had nothing to resolve and the model was
|
||||
reachable only through plain ``vllm.LLM``. That is the gap M0 closes: the text
|
||||
mode has to run *through* the Omni stage runtime, or none of the stage
|
||||
lifecycle, admission or cancellation work applies to it.
|
||||
|
||||
There is exactly one stage, and that is the design, not a simplification to be
|
||||
undone later. The proposal's first decision is that an autoregressive session
|
||||
stays on one backend by default: prefill and decode share the KV cache, and a
|
||||
per-token boundary between devices buys nothing and costs a synchronization.
|
||||
Spark's tokenizer, chat template and sampling are model-adapter concerns that
|
||||
run in-process around this stage; they are not stages.
|
||||
"""
|
||||
|
||||
from vllm_omni.config.stage_config import (
|
||||
PipelineConfig,
|
||||
StageExecutionType,
|
||||
StagePipelineConfig,
|
||||
)
|
||||
|
||||
SPARK2_5_PIPELINE = PipelineConfig(
|
||||
model_type="spark2_5",
|
||||
model_arch="Spark2_5ForCausalLM",
|
||||
hf_architectures=("Spark2_5ForCausalLM",),
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="spark2_5",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="text",
|
||||
owns_tokenizer=True,
|
||||
engine_output_type="text",
|
||||
model_arch="Spark2_5ForCausalLM",
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -494,9 +494,12 @@ class Spark2_5ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
|
||||
inputs_embeds: torch.Tensor | None = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor | IntermediateTensors:
|
||||
return self.model(
|
||||
input_ids, positions, intermediate_tensors, inputs_embeds, **kwargs
|
||||
)
|
||||
# Omni's AR model runner hands every stage model ``sampling_metadata``,
|
||||
# ``logits_index`` and ``sampler``; the omni-native models take them in
|
||||
# ``**kwargs``. Spark samples through vLLM's own sampler, and the
|
||||
# decoder stack reads none of them, so they stop here rather than
|
||||
# widening ``Spark2_5Model.forward`` to a signature it ignores.
|
||||
return self.model(input_ids, positions, intermediate_tensors, inputs_embeds)
|
||||
|
||||
def compute_logits(self, hidden_states: torch.Tensor) -> torch.Tensor | None:
|
||||
return self.logits_processor(self.lm_head, hidden_states)
|
||||
|
||||
Reference in new issue
Block a user