Revert "[Config Refactor] Derive deploy override fields from stage config (#3162)"

This reverts commit 01ebc0cdf5.

Signed-off-by: GitHub <noreply@github.com>

Co-authored-by: Gaohan123 <20148503+Gaohan123@users.noreply.github.com>

.github/ISSUE_TEMPLATE/100-documentation.yml

@@ -1,29 +0,0 @@
-name: 📚 Documentation
-description: Report an issue related to documentation
-title: "[Doc]: "
-labels: ["documentation"]
-
-body:
-- type: textarea
-  attributes:
-    label: 📚 The doc issue
-    description: >
-      A clear and concise description of what content in the documentation is an issue.
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: Suggest a potential alternative/fix
-    description: >
-      Tell us how we could improve the documentation in this regard.
-- type: markdown
-  attributes:
-    value: >
-      Thanks for contributing 🎉!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/200-installation.yml

@@ -1,47 +0,0 @@
-name: 🛠️ Installation
-description: Report an issue here when you hit errors during installation.
-title: "[Installation]: "
-labels: ["installation"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/hsliuustc0106/vllm-omni/issues?q=is%3Aissue+sort%3Acreated-desc+).
-- type: textarea
-  attributes:
-    label: Your current environment
-    description: |
-      Please run the following and paste the output below.
-      ```sh
-      wget https://raw.githubusercontent.com/hsliuustc0106/vllm-omni/main/vllm_omni/collect_env.py
-      # For security purposes, please feel free to check the contents of collect_env.py before running it.
-      python collect_env.py
-      ```
-      It is suggested to download and execute the latest script, as vllm-omni might frequently update the diagnosis information needed for accurately and quickly responding to issues.
-    value: |
-      ```text
-      The output of `python collect_env.py`
-      ```
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: How you are installing vllm-omni
-    description: |
-      Paste the full command you are trying to execute.
-    value: |
-      ```sh
-      pip install -vvv vllm-omni
-      ```
-- type: markdown
-  attributes:
-    value: >
-      Thanks for contributing 🎉!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/400-bug-report.yml

@@ -1,111 +0,0 @@
-name: 🐛 Bug report
-description: Raise an issue here if you find a bug.
-title: "[Bug]: "
-labels: ["bug"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/hsliuustc0106/vllm-omni/issues?q=is%3Aissue+sort%3Acreated-desc+).
-- type: markdown
-  attributes:
-    value: |
-      ⚠️ **SECURITY WARNING:** Please review any text you paste to ensure it does not contain sensitive information such as:
-      - API tokens or keys (e.g., Hugging Face tokens, OpenAI API keys)
-      - Passwords or authentication credentials
-      - Private URLs or endpoints
-      - Personal or confidential data
-      
-      Consider redacting or replacing sensitive values with placeholders like `<YOUR_TOKEN_HERE>` when sharing configuration or code examples.
-- type: textarea
-  attributes:
-    label: Your current environment
-    description: |
-      Please run the following and paste the output below.
-      ```sh
-      wget https://raw.githubusercontent.com/hsliuustc0106/vllm-omni/main/vllm_omni/collect_env.py
-      # For security purposes, please feel free to check the contents of collect_env.py before running it.
-      python collect_env.py
-      ```
-      It is suggested to download and execute the latest script, as vllm-omni might frequently update the diagnosis information needed for accurately and quickly responding to issues.
-    value: |
-      <details>
-      <summary>The output of <code>python collect_env.py</code></summary>
-
-      ```text
-      Your output of `python collect_env.py` here
-      ```
-
-      </details>
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: 🐛 Describe the bug
-    description: |
-      Please provide a clear and concise description of what the bug is.
-
-      If relevant, add a minimal example so that we can reproduce the error by running the code. It is very important for the snippet to be as succinct (minimal) as possible, so please take time to trim down any irrelevant code to help us debug efficiently. We are going to copy-paste your code and we expect to get the same result as you did: avoid any external data, and include the relevant imports, etc. For example:
-
-      ```python
-      from vllm_omni import OmniLLM, create_ar_stage_config, create_dit_stage_config
-
-      # Create stage configurations
-      ar_config = create_ar_stage_config(
-          stage_id=0,
-          model_path="Qwen/Qwen3-0.6B",
-          input_modalities=["text"],
-          output_modalities=["text"]
-      )
-      
-      dit_config = create_dit_stage_config(
-          stage_id=1,
-          model_path="stabilityai/stable-diffusion-2-1",
-          input_modalities=["text"],
-          output_modalities=["image"]
-      )
-
-      # Initialize OmniLLM
-      omni = OmniLLM([ar_config, dit_config])
-
-      # Generate
-      outputs = omni.generate(prompt="A scenic watercolor painting of a lighthouse at sunset")
-      ```
-
-      If the code is too long (hopefully, it isn't), feel free to put it in a public gist and link it in the issue: https://gist.github.com.
-
-      Please also paste or describe the results you observe instead of the expected results. If you observe an error, please paste the error message including the **full** traceback of the exception. It may be relevant to wrap error messages in ```` ```triple quotes blocks``` ````.
-
-      Please set the environment variable `export VLLM_OMNI_LOGGING_LEVEL=DEBUG` to turn on more logging to help debugging potential issues.
-
-      If you experienced crashes or hangs, it would be helpful to run vllm-omni with `export VLLM_OMNI_TRACE_FUNCTION=1` . All the function calls in vllm-omni will be recorded. Inspect these log files, and tell which function crashes or hangs.
-    placeholder: |
-      A clear and concise description of what the bug is.
-
-      ```python
-      # Sample code to reproduce the problem
-      ```
-
-      ```
-      The error message you got, with the full traceback and the error logs with [dump_input.py:##] if present.
-      ```
-  validations:
-    required: true
-- type: markdown
-  attributes:
-    value: |
-      ⚠️ Please separate bugs of `transformers` implementation or usage from bugs of `vllm-omni`. If you think anything is wrong with the model's output:
-
-      - Try the counterpart of `transformers` first. If the error appears, please go to [their issues](https://github.com/huggingface/transformers/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc).
-
-      - If the error only appears in vllm-omni, please provide the detailed script of how you run `transformers` and `vllm-omni`, also highlight the difference and what you expect.
-
-      Thanks for reporting 🙏!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/500-feature-request.yml

@@ -1,38 +0,0 @@
-name: 🚀 Feature request
-description: Submit a proposal/request for a new vllm-omni feature
-title: "[Feature]: "
-labels: ["feature request"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/hsliuustc0106/vllm-omni/issues?q=is%3Aissue+sort%3Acreated-desc+).
-- type: textarea
-  attributes:
-    label: 🚀 The feature, motivation and pitch
-    description: >
-      A clear and concise description of the feature proposal. Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., *"I'm working on X and would like Y to be possible"*. If this is related to another GitHub issue, please link here too.
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: Alternatives
-    description: >
-      A description of any alternative solutions or features you've considered, if any.
-- type: textarea
-  attributes:
-    label: Additional context
-    description: >
-      Add any other context or screenshots about the feature request.
-- type: markdown
-  attributes:
-    value: >
-      Thanks for contributing 🎉!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/600-new-model.yml

@@ -1,45 +0,0 @@
-name: 🤗 Support request for a new model
-description: Submit a proposal/request for a new model support
-title: "[New Model]: "
-labels: ["new model"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/hsliuustc0106/vllm-omni/issues?q=is%3Aissue+sort%3Acreated-desc+).
-
-      #### We also highly recommend you read the contributing guidelines first to understand how to add a new model.
-- type: textarea
-  attributes:
-    label: The model to consider.
-    description: >
-      A huggingface url, pointing to the model, e.g. https://huggingface.co/openai-community/gpt2 .
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: The closest model vllm-omni already supports.
-    description: >
-      Here is the list of models already supported by vllm-omni. Which model is the most similar to the model you want to add support for?
-- type: textarea
-  attributes:
-    label: What's your difficulty of supporting the model you want?
-    description: >
-      For example, any new operators or new architecture? Is it a multimodal model? Does it require special preprocessing?
-- type: textarea
-  attributes:
-    label: Use case and motivation
-    description: >
-      What is the primary use case for this model? Why is it important to support in vllm-omni?
-- type: markdown
-  attributes:
-    value: >
-      Thanks for contributing 🎉!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/700-performance-discussion.yml

@@ -1,59 +0,0 @@
-name: ⚡ Discussion on the performance of vllm-omni
-description: Submit a proposal/discussion about the performance of vllm-omni
-title: "[Performance]: "
-labels: ["performance"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/hsliuustc0106/vllm-omni/issues?q=is%3Aissue+sort%3Acreated-desc+).
-- type: textarea
-  attributes:
-    label: Proposal to improve performance
-    description: >
-      How do you plan to improve vllm-omni's performance?
-  validations:
-    required: false
-- type: textarea
-  attributes:
-    label: Report of performance regression
-    description: >
-      Please provide detailed description of performance comparison to confirm the regression. You may want to run the benchmark script at https://github.com/hsliuustc0106/vllm-omni/tree/main/tests/benchmarks .
-  validations:
-    required: false
-- type: textarea
-  attributes:
-    label: Misc discussion on performance
-    description: >
-      Anything about the performance.
-  validations:
-    required: false
-- type: textarea
-  attributes:
-    label: Your current environment (if you think it is necessary)
-    description: |
-      Please run the following and paste the output below.
-      ```sh
-      wget https://raw.githubusercontent.com/hsliuustc0106/vllm-omni/main/vllm_omni/collect_env.py
-      # For security purposes, please feel free to check the contents of collect_env.py before running it.
-      python collect_env.py
-      ```
-      It is suggested to download and execute the latest script, as vllm-omni might frequently update the diagnosis information needed for accurately and quickly responding to issues.
-    value: |
-      ```text
-      The output of `python collect_env.py`
-      ```
-  validations:
-    required: false
-- type: markdown
-  attributes:
-    value: >
-      Thanks for contributing 🎉!
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/750-RFC.yml

@@ -1,52 +0,0 @@
-name: 💬 Request for comments (RFC).
-description: Ask for feedback on major architectural changes or design choices.
-title: "[RFC]: "
-labels: ["RFC"]
-
-body:
-- type: markdown
-  attributes:
-    value: >
-      #### Please take a look at previous [RFCs](https://github.com/hsliuustc0106/vllm-omni/issues?q=label%3ARFC+sort%3Aupdated-desc) for reference.
-- type: textarea
-  attributes:
-    label: Motivation.
-    description: >
-      The motivation of the RFC.
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: Proposed Change.
-    description: >
-      The proposed change of the RFC.
-  validations:
-    required: true
-- type: textarea
-  attributes:
-    label: Feedback Period.
-    description: >
-      The feedback period of the RFC. Usually at least one week.
-  validations:
-    required: false
-- type: textarea
-  attributes:
-    label: CC List.
-    description: >
-      The list of people you want to CC.
-  validations:
-    required: false
-- type: textarea
-  attributes:
-    label: Any Other Things.
-    description: >
-      Any other things you would like to mention.
-  validations:
-    required: false
-- type: checkboxes
-  id: askllm
-  attributes:
-    label: Before submitting a new issue...
-    options:
-      - label: Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://vllm-omni.readthedocs.io), which can answer lots of frequently asked questions.
-        required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +0,0 @@
-blank_issues_enabled: false
-contact_links:
-  - name: Questions
-    url: https://github.com/hsliuustc0106/vllm-omni/discussions
-    about: Ask questions and discuss with other vLLM-omni community members

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,21 +0,0 @@
-<!-- markdownlint-disable -->
-PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
-
-## Purpose
-
-## Test Plan
-
-## Test Result
-
----
-<details>
-<summary> Essential Elements of an Effective PR Description Checklist </summary>
-
-- [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
-- [ ] The test plan, such as providing test command.
-- [ ] The test results, such as pasting the results comparison before and after, or e2e results
-- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
-- [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft.
-</details>
-
-**BEFORE SUBMITTING, PLEASE READ <https://github.com/hsliuustc0106/vllm-omni/blob/main/CONTRIBUTING.md>** (anything written below this line will be removed by GitHub Actions)

.gitignore

@@ -1,237 +0,0 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-cover/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-.pybuilder/
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# pdm
-#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
-#pdm.lock
-#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
-#   in version control.
-#   https://pdm.fming.dev/#use-with-ide
-.pdm.toml
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# pytype static type analyzer
-.pytype/
-
-# Cython debug symbols
-cython_debug/
-
-# cursor
-.cursor/
-
-# docker
-docker/
-
-# scripts
-scripts/
-
-# tests
-tests/
-
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be added to the global gitignore or merged into this project gitignore.  For a PyCharm
-#  project, it is recommended to include the following files:
-#  .idea/
-#  *.iml
-#  *.ipr
-#  *.iws
-#  .idea/
-
-# vLLM-omni specific
-# Model files and checkpoints
-models/
-checkpoints/
-*.bin
-*.safetensors
-*.pt
-*.pth
-
-# Cache directories
-cache/
-.cache/
-diffusion_cache/
-kv_cache/
-
-# Logs
-logs/
-*.log
-
-# Temporary files
-tmp/
-temp/
-.tmp/
-
-# IDE files
-.vscode/
-.idea/
-*.swp
-*.swo
-*~
-
-# OS files
-.DS_Store
-.DS_Store?
-._*
-.Spotlight-V100
-.Trashes
-ehthumbs.db
-Thumbs.db
-
-# Data files
-data/
-datasets/
-*.csv
-*.json
-*.jsonl
-*.parquet
-
-# Output files
-outputs/
-results/
-generated/
-
-# Configuration overrides
-configs/local.yaml
-configs/production.yaml
-configs/development.yaml
-
-# Docker
-.dockerignore
-Dockerfile.dev

LICENSE

@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

README.md

@@ -1,75 +0,0 @@
-# vLLM-omni: Multi-modal Extension for vLLM
-
-vLLM-omni is designed to extend vLLM capabilities to support multi-modality model inference and serving, particularly focusing on non-autoregressive structures and non-textual outputs.
-
-## 🎯 Overview
-
-Traditional vLLM systems are limited to text-based, autoregressive generation. vLLM-omni addresses this limitation by enabling support for:
-
-- **Multi-modal Models**: Text, image, video, audio, and sensor data processing
-- **Non-autoregressive Architectures**: Diffusion Transformers (DiT) and other parallel generation models
-- **Heterogeneous Outputs**: Beyond traditional text generation to multimodal outputs
-
-## 🏗️ Architecture
-
-vLLM-omni is built on a modular architecture that extends vLLM's core functionality:
-
-
-## 🚀 Key Features
-
-### Multi-Engine Support
-
-- **Autoregressive Engine**: Traditional text generation with enhanced KV-caching
-- **Diffusion Engine**: Support for DiT models and iterative generation
-- **Hybrid Engine**: Combined AR+DiT processing pipelines
-
-### Modality Processing
-
-- **Text**: Advanced tokenization and embedding generation
-- **Image**: Vision encoder integration (CLIP, etc.)
-- **Audio**: Speech processing and audio embedding
-
-## 📋 Supported Models
-
-### AR + Diffusion Transformer (DiT) Models
-- Qwen-omni (Thinker-Talker-Codec structure)
-- HunyunaImage 3.0 (Ongoing)
-- Qwen-Image (Ongoing)
-
-## 🛠️ Installation
-
-Set up basic environments
-```bash
-uv venv --python 3.12 --seed
-source .venv/bin/activate
-```
-Install certain version of vllm with commitid: 808a7b69df479b6b3a16181711cac7ca28a9b941
-
-```bash
-git clone https://github.com/vllm-project/vllm.git
-cd vllm
-git checkout 808a7b69df479b6b3a16181711cac7ca28a9b941
-VLLM_USE_PRECOMPILED=1 uv pip install --editable .
-```
-
-## Run examples (Qwen2.5-omni)
-
-Get into the example folder
-```bash
-cd vllm_omni
-cd examples/offline_inference/qwen2_5_omni
-```
-Modify PYTHONPATH in run.sh as your path of vllm_omni. Then run.
-```bash
-bash run.sh
-```
-The output audio is saved in ./output_audio
-
-## To-do list
-- [x] Offline inference example for Qwen2.5-omni with single request
-- [ ] Adaptation from current vllm branch to stable vllm v0.11.0
-- [ ] Offline inference example for Qwen2.5-omni with streaming multiple requests
-- [ ] Online inference support
-- [ ] Support for other models
-
-For detailed model management, see [vllm_omni_design.md](docs/architecture/vllm_omni_design.md) and [high_level_arch_design.md](docs/architecture/high_level_arch_design.md).

docs/api/API_DESIGN_TEMPLATE.md

@@ -1,336 +0,0 @@
-# API Design Template for vLLM-omni Modules
-
-This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-omni. Use this template to ensure consistency and completeness across all modules.
-
-## Template Structure
-
-### 1. Module Overview
-- **Purpose**: What this module does
-- **Responsibilities**: Key responsibilities of the module
-- **Dependencies**: What other modules this depends on
-- **Integration Points**: How it integrates with other modules
-
-### 2. Core Classes/Interfaces
-- **Base Classes**: Abstract base classes and interfaces
-- **Implementation Classes**: Concrete implementations
-- **Data Structures**: Key data structures and models
-
-### 3. Public API Methods
-- **Initialization**: Constructor and setup methods
-- **Core Operations**: Main functionality methods
-- **Configuration**: Configuration and parameter methods
-- **Lifecycle Management**: Start, stop, cleanup methods
-- **Monitoring**: Status, metrics, and debugging methods
-
-### 4. Configuration
-- **Configuration Classes**: Dataclasses or config objects
-- **Required Parameters**: Must-have configuration
-- **Optional Parameters**: Optional configuration with defaults
-- **Validation**: Parameter validation rules
-
-### 5. Error Handling
-- **Custom Exceptions**: Module-specific exceptions
-- **Error Codes**: Standardized error codes
-- **Recovery Strategies**: How to handle and recover from errors
-
-### 6. Examples
-- **Basic Usage**: Simple usage examples
-- **Advanced Usage**: Complex scenarios
-- **Integration Examples**: How to use with other modules
-
----
-
-## Example: Core Module Template
-
-### 1. Module Overview
-
-**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-omni.
-
-**Responsibilities**:
-- Request scheduling and prioritization
-- DiT cache management
-- Resource allocation and coordination
-- Inter-module communication
-
-**Dependencies**:
-- `vllm_omni.request` - Request handling
-- `vllm_omni.config` - Configuration management
-- `vllm_omni.utils` - Utility functions
-
-**Integration Points**:
-- Receives requests from entrypoints
-- Coordinates with engine modules
-- Manages worker allocation
-- Provides status to monitoring systems
-
-### 2. Core Classes/Interfaces
-
-```python
-from abc import ABC, abstractmethod
-from typing import List, Optional, Dict, Any
-from dataclasses import dataclass
-from enum import Enum
-
-class DiTCacheManager:
-    """Manages DiT cache for diffusion models."""
-    
-    def __init__(self, config: DiTCacheConfig):
-        self.config = config
-        self.cache = {}
-        self.cache_stats = {}
-    
-    async def get_cache(self, cache_key: str) -> Optional[Any]:
-        """Retrieve cached data."""
-        pass
-    
-    async def set_cache(self, cache_key: str, data: Any) -> None:
-        """Store data in cache."""
-        pass
-    
-    async def invalidate_cache(self, cache_key: str) -> None:
-        """Invalidate cached data."""
-        pass
-```
-
-### 3. Public API Methods
-
-#### Initialization
-```python
-class OmniScheduler():
-    def __init__(self, config: SchedulerConfig):
-        """Initialize the core scheduler."""
-        self.config = config
-        self.scheduler = self._create_scheduler()
-        self.cache_manager = DiTCacheManager(config.dit_cache_config)
-        self._running = False
-    
-    def _create_scheduler(self) -> BaseScheduler:
-        """Factory method to create appropriate scheduler."""
-        if self.config.scheduler_type == SchedulerType.FIFO:
-            return FIFOScheduler(self.config)
-        elif self.config.scheduler_type == SchedulerType.PRIORITY:
-            return PriorityScheduler(self.config)
-        # ... other scheduler types
-```
-
-#### Core Operations
-```python
-    async def schedule(self, request: OmniRequest) -> bool:
-        """
-        Schedule a request for processing.
-        
-        Args:
-            request: The request to schedule
-            
-        Returns:
-            bool: True if successfully scheduled, False otherwise
-            
-        Raises:
-            SchedulerError: If scheduling fails
-            QueueFullError: If queue is at capacity
-        """
-        pass
-```
-
-#### Configuration
-```python
-    def update_config(self, new_config: SchedulerConfig) -> None:
-        """Update scheduler configuration."""
-        pass
-    
-    def get_config(self) -> SchedulerConfig:
-        """Get current configuration."""
-        pass
-```
-
-#### Lifecycle Management
-```python
-    async def start(self) -> None:
-        """Start the scheduler."""
-        self._running = True
-        # Start background tasks
-    
-    async def stop(self) -> None:
-        """Stop the scheduler gracefully."""
-        self._running = False
-        # Cleanup resources
-    
-    async def shutdown(self) -> None:
-        """Force shutdown the scheduler."""
-        # Immediate cleanup
-```
-
-#### Monitoring
-```python
-    def get_status(self) -> Dict[str, Any]:
-        """Get current scheduler status."""
-        return {
-            "running": self._running,
-            "queue_size": self.scheduler.queue_size(),
-            "processed_requests": self.scheduler.processed_count(),
-            "cache_hit_rate": self.cache_manager.get_hit_rate()
-        }
-    
-    def get_metrics(self) -> Dict[str, float]:
-        """Get performance metrics."""
-        pass
-```
-
-### 4. Configuration
-
-```python
-@dataclass
-class OmniConfig:
-    """Configuration for the core module."""
-    
-    # Scheduler configuration
-    scheduler: SchedulerConfig = field(default_factory=SchedulerConfig)
-    
-    # Cache configuration
-    dit_cache: DiTCacheConfig = field(default_factory=DiTCacheConfig)
-    
-    # Resource limits
-    max_memory_gb: float = 16.0
-    max_gpu_utilization: float = 0.8
-    
-    # Timeouts
-    request_timeout: int = 300
-    worker_timeout: int = 60
-    
-    def validate(self) -> None:
-        """Validate configuration parameters."""
-        if self.max_memory_gb <= 0:
-            raise ValueError("max_memory_gb must be positive")
-        if not 0 < self.max_gpu_utilization <= 1:
-            raise ValueError("max_gpu_utilization must be between 0 and 1")
-```
-
-### 5. Error Handling
-
-```python
-class CoreModuleError(Exception):
-    """Base exception for core module errors."""
-    pass
-
-class SchedulerError(CoreModuleError):
-    """Scheduler-related errors."""
-    pass
-
-class QueueFullError(SchedulerError):
-    """Queue is at capacity."""
-    pass
-
-class CacheError(CoreModuleError):
-    """Cache-related errors."""
-    pass
-
-class ResourceError(CoreModuleError):
-    """Resource allocation errors."""
-    pass
-```
-
-### 6. Examples
-
-#### Basic Usage
-```python
-from vllm_omni.core import CoreScheduler, CoreModuleConfig
-from vllm_omni.request import create_text_request
-
-# Create configuration
-config = CoreModuleConfig(
-    scheduler=SchedulerConfig(scheduler_type=SchedulerType.FIFO),
-    max_memory_gb=8.0
-)
-
-# Initialize scheduler
-scheduler = CoreScheduler(config)
-await scheduler.start()
-
-# Create and schedule a request
-request = create_text_request(
-    request_id="req_001",
-    prompt="Hello, world!",
-    sampling_params=sampling_params
-)
-
-success = await scheduler.schedule_request(request)
-if success:
-    result = await scheduler.process_request(request)
-    print(f"Result: {result}")
-```
-
-#### Advanced Usage
-```python
-# Custom scheduler with priority handling
-config = CoreModuleConfig(
-    scheduler=SchedulerConfig(
-        scheduler_type=SchedulerType.PRIORITY,
-        priority_weights={"high": 2.0, "normal": 1.0, "low": 0.5}
-    )
-)
-
-scheduler = CoreScheduler(config)
-await scheduler.start()
-
-# Monitor scheduler status
-status = scheduler.get_status()
-print(f"Queue size: {status['queue_size']}")
-print(f"Cache hit rate: {status['cache_hit_rate']:.2%}")
-```
-
----
-
-## Module-Specific Guidelines
-
-### Core Module
-- Focus on scheduling, caching, and resource management
-- Provide clear interfaces for other modules
-- Handle concurrency and thread safety
-- Implement comprehensive monitoring
-
-### Engine Module
-- Handle model loading and inference
-- Support both AR and diffusion models
-- Provide unified interface for different model types
-- Implement efficient memory management
-
-### Executor Module
-- Coordinate between different engines
-- Handle request routing and load balancing
-- Manage execution pipelines
-- Provide error recovery mechanisms
-
-### Worker Module
-- Handle actual model execution
-- Manage GPU resources
-- Implement batching strategies
-- Provide performance optimization
-
----
-
-## Checklist for API Design
-
-- [ ] **Clear Purpose**: Module purpose is well-defined
-- [ ] **Complete Interface**: All public methods are documented
-- [ ] **Error Handling**: Comprehensive error handling strategy
-- [ ] **Configuration**: Flexible configuration system
-- [ ] **Examples**: Basic and advanced usage examples
-- [ ] **Type Hints**: All methods have proper type hints
-- [ ] **Documentation**: Comprehensive docstrings
-- [ ] **Testing**: Unit tests for all public methods
-- [ ] **Integration**: Clear integration points with other modules
-- [ ] **Performance**: Performance considerations documented
-
----
-
-## Submission Guidelines
-
-1. Create a new file: `docs/api/[module_name]_api.md`
-2. Follow the template structure exactly
-3. Include all required sections
-4. Provide working code examples
-5. Ensure all methods have proper docstrings
-6. Include error handling strategies
-7. Submit for review before implementation
-
-This template ensures consistency and completeness across all vLLM-omni modules.

docs/api/README.md

@@ -1,235 +0,0 @@
-# vLLM-omni API Documentation
-
-This directory contains comprehensive API design documentation for all core modules in vLLM-omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules.
-
-## 📋 API Design Templates
-
-### 1. [API Design Template](API_DESIGN_TEMPLATE.md)
-**Master template** for creating API documentation for any module in vLLM-omni. Use this as a starting point for new modules.
-
-### 2. [Core Module API](core_module_api.md)
-**Core module** provides fundamental scheduling, caching, and resource management functionality.
-
-**Key Components:**
-- Request scheduling and prioritization
-- DiT cache management for diffusion models
-- Resource allocation and coordination
-- Inter-module communication
-
-### 3. [Engine Module API](engine_module_api.md)
-**Engine module** handles model loading, inference execution, and output processing.
-
-**Key Components:**
-- Model loading and initialization
-- Inference execution for AR and diffusion models
-- Input preprocessing for various modalities
-- Output postprocessing and formatting
-
-### 4. [Executor Module API](executor_module_api.md)
-**Executor module** coordinates and manages request execution across different engines and workers.
-
-**Key Components:**
-- Request routing and load balancing
-- Execution pipeline coordination
-- Worker management and task distribution
-- Error handling and recovery
-
-### 5. [Worker Module API](worker_module_api.md)
-**Worker module** provides the actual execution environment for model inference.
-
-**Key Components:**
-- Model execution and inference
-- GPU resource management
-- Request batching and processing
-- Performance optimization
-
-## 🎯 How to Use These Templates
-
-### For New Module Development
-
-1. **Start with the Master Template**
-   - Copy `API_DESIGN_TEMPLATE.md`
-   - Rename to `[module_name]_api.md`
-   - Follow the structure exactly
-
-2. **Fill in Module-Specific Details**
-   - Update the module overview
-   - Define core classes and interfaces
-   - Specify public API methods
-   - Add configuration options
-   - Define error handling strategy
-   - Provide usage examples
-
-3. **Review and Validate**
-   - Ensure all sections are complete
-   - Verify code examples work
-   - Check for consistency with other modules
-   - Validate configuration options
-
-### For Existing Module Updates
-
-1. **Update API Documentation**
-   - Modify existing API files
-   - Add new methods and classes
-   - Update examples and configuration
-   - Maintain backward compatibility notes
-
-2. **Version Control**
-   - Track changes in git
-   - Use clear commit messages
-   - Tag major API changes
-
-## 📚 Template Structure
-
-Each API template follows this standardized structure:
-
-### 1. Module Overview
-- **Purpose**: What the module does
-- **Responsibilities**: Key responsibilities
-- **Dependencies**: Required modules
-- **Integration Points**: How it connects with other modules
-
-### 2. Core Classes/Interfaces
-- **Base Classes**: Abstract base classes
-- **Implementation Classes**: Concrete implementations
-- **Data Structures**: Key data models
-
-### 3. Public API Methods
-- **Initialization**: Constructor and setup
-- **Core Operations**: Main functionality
-- **Configuration**: Settings management
-- **Lifecycle Management**: Start/stop/cleanup
-- **Monitoring**: Status and metrics
-
-### 4. Configuration
-- **Configuration Classes**: Dataclasses for settings
-- **Required Parameters**: Must-have settings
-- **Optional Parameters**: Optional settings with defaults
-- **Validation**: Parameter validation rules
-
-### 5. Error Handling
-- **Custom Exceptions**: Module-specific errors
-- **Error Codes**: Standardized error codes
-- **Recovery Strategies**: Error recovery approaches
-
-### 6. Examples
-- **Basic Usage**: Simple usage examples
-- **Advanced Usage**: Complex scenarios
-- **Integration Examples**: Multi-module usage
-
-## 🔧 Implementation Guidelines
-
-### Code Standards
-- Use type hints for all methods
-- Include comprehensive docstrings
-- Follow PEP 8 style guidelines
-- Use async/await for I/O operations
-
-### Error Handling
-- Define specific exception types
-- Provide meaningful error messages
-- Include error recovery strategies
-- Log errors appropriately
-
-### Configuration
-- Use dataclasses for configuration
-- Provide sensible defaults
-- Include validation methods
-- Support environment variables
-
-### Testing
-- Write unit tests for all public methods
-- Include integration tests
-- Test error conditions
-- Validate configuration options
-
-## 🚀 Getting Started
-
-### For Developers
-
-1. **Choose a Module**
-   - Pick the module you want to work on
-   - Read the corresponding API documentation
-   - Understand the responsibilities and interfaces
-
-2. **Set Up Development Environment**
-   ```bash
-   # Clone the repository
-   git clone https://github.com/hsliuustc0106/vllm-omni.git
-   cd vllm-omni
-   
-   # Install dependencies
-   pip install -r requirements-dev.txt
-   
-   # Set up pre-commit hooks
-   pre-commit install
-   ```
-
-3. **Start Implementation**
-   - Create the module directory
-   - Implement base classes first
-   - Add concrete implementations
-   - Write tests as you go
-
-### For Contributors
-
-1. **Review API Documentation**
-   - Read through all module APIs
-   - Understand the overall architecture
-   - Identify areas for improvement
-
-2. **Propose Changes**
-   - Create issues for API changes
-   - Discuss in pull requests
-   - Update documentation accordingly
-
-3. **Submit Contributions**
-   - Follow the coding standards
-   - Include tests for new features
-   - Update documentation
-   - Submit pull requests
-
-## 📖 Additional Resources
-
-### Architecture Overview
-- [vLLM-omni Architecture Design](../architecture/vLLM-omni%20arch%20design%20doc.md)
-- [Component Categorization](../../ARCHITECTURE_CATEGORIZATION.md)
-
-### Development Guides
-- [Development Setup](../../README.md#development)
-- [Testing Guidelines](../../tests/README.md)
-- [Contributing Guidelines](../../CONTRIBUTING.md)
-
-### Examples
-- [Basic Examples](../../examples/basic/)
-- [Advanced Examples](../../examples/advanced/)
-- [Multimodal Examples](../../examples/multimodal/)
-
-## 🤝 Contributing
-
-We welcome contributions to improve the API documentation and implementation. Please:
-
-1. **Follow the Template Structure**
-   - Use the master template as a guide
-   - Maintain consistency across modules
-   - Include all required sections
-
-2. **Provide Working Examples**
-   - Test all code examples
-   - Include both basic and advanced usage
-   - Show integration patterns
-
-3. **Keep Documentation Updated**
-   - Update docs when APIs change
-   - Version control documentation changes
-   - Maintain backward compatibility notes
-
-## 📝 Notes
-
-- All API documentation should be kept up-to-date with implementation
-- Code examples should be tested and working
-- Configuration options should be validated
-- Error handling should be comprehensive
-- Performance considerations should be documented
-
-For questions or suggestions about the API documentation, please open an issue or start a discussion in the repository.

docs/architecture/high level arch design.md

@@ -1,139 +0,0 @@
-# vLLM-omni Architecture Design Document
-
-# Introduction
-
-This document outlines the architectural design for vLLM-omni, a new repository developed under the vLLM project. vLLM-omni supports multi-modality models with non-autoregressive structures and non-textual outputs, extending vLLM beyond traditional text-based, autoregressive generation.
-
-# Goals
-
-The primary goals for vLLM-omni are:
-
-* **Support for Non-textual Output:** Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, and video, alongside text.  
-* **Non-Autoregressive Structure:** Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation.  
-* **Integration with vLLM Core:** Maintain compatibility and leverage existing vLLM key modules and optimizations where applicable.  
-* **Extensibility:** Design a modular and flexible architecture that can easily accommodate new modalities, model architectures, and output formats.
-
-# Expected Gains
-
-* It breaks through the limitation of vLLM that only supports autoregressive text generation models, and provides architectural support for popular model structures, especially multimodal generation.  
-* By supporting new output formats and new model architectures, developers can easily connect to mature ecosystems such as Higress Gateway and Ray, improving the ease of deployment and efficiency of corresponding models.  
-* Integrating efficient cache management, multiple parallel strategies, disaggregation from vLLM, to improve inference efficiency of corresponding models.
-
-# Non-Autoregressive Structure
-
-* **Diffusion Transformer:** initialized with a random latent noise, the transformer model iteratively predicts latent output for multiple steps, under control of sampling timestamps. Finally, it is decoded by the decoder of Variational AutoEncoder (VAE). It is not supported by vLLM now.  
-* **Autoregressive:** dominant generation paradigm for text, now supported by vLLM. It generates tokens conditioned on previous tokens one by one. The efficient KV Cache management provided by vLLM can accelerate inference effectively.  
-  ![][image1]                       ![][image2]
-
-
-|  | Autoregressive | DiT |
-| :---- | ----- | ----- |
-| Scenarios | Text generation | Multimodal generation |
-| Generation | Token-by-token, left to right | Continuous vector, iteratively refine |
-| Key advantages | Smooth, good local consistency | Global consistency, diversity and controlability |
-| Efficiency | More efficient with KV cache | Need generation repeatedly, slow |
-| Error accumulation | Easy to accumulate | Can fix possibly |
-| Key challenges | Limited global planning, long context generation | Large global computational cost |
-| Sequence length | Varied length | Fixed length |
-| Attention mask | Lower triangular | Full bi-directional matrix |
-| Parallel strategy | TP, DP, PP, .etc | DP, CP, .etc |
-| vLLM support | Supported | Unsupported |
-
-# Targeted models to support
-
-According to analysis for current popular open-source models, most of them have the combination of AR+DiT. Specifically, they can be further categorized into 3 types below:
-
-* **DiT as a main structure, with AR as text encoder (e.g.: Qwen-Image)**  
-  A popular model for image generation and editing. Main visual generation models are similar to the structure, such as Flux.  
-  ![][image3]  
-* **AR as a main structure, with DiT as multi-modal generator (e.g. BAGEL)**  
-  A unified multimodal comprehension and generation model. Visual generation can utilize CoT text generation output. Main unified multimodal models are similar to the structure.  
-      ![][image4]  
-* **Extension, multiple AR+DiT (e.g. Qwen-omni)**  
-  A new model for multimodal input and output. It designs the Thinker-Talker-Codec structure, which is a double-AR+DiT format.  
-  ![][image5]  
-  
-
-
-# Architectural Overview
-
-vLLM-omni will import core modules from existing vLLM framework and introduce several new components and modifications to achieve its goals. The core architecture will be based on a modular design, allowing for independent development and integration of different modality handlers and output generators.  
-![][image6]
-
-## General ideas
-
-* It divides models in vLLM-omni into multiple stages, which are processed by different Engine Cores. Each stage can choose AR Engine Core for AR model part, and Diffusion Engine Core for diffusion model part.  
-* Designed a specialized diffusion engine core for DiT or dLLM.  
-* Plug-in with external DiT acceleration modules such as xDiT, Cache-DiT.  
-* Output linked with Higress API, Gradio for online demo, ComfyUI for less-code process.
-
-## Key Components
-
-* **Processor:** Input processing, tokenization or encoding  
-* **Scheduler:** for order and queue of incoming requests  
-* **Executor:**  Prepare model input and run  
-* **Multimodal output processor:** support multimodal outputs  
-* **Hidden State Output processor:** support intermediate hidden vectors as request output  
-* **Engine Core:** accelerate AR modules, imported from vLLM main repo  
-* **Diffusion Engine Core:** Wrap and accelerate DiT modules.  
-* **Diffusion model:** Wrap up diffusion models with vLLM format
-
-# Examples data flow of specific models
-
-1. Qwen-omni  
-   ![][image7]![][image8]  
-2. Qwen-Image  
-   ![][image9]      ![][image10]
-
-# Overview of function call dependency
-
-![][image11]
-
-# Roadmap
-
-* AsyncLLM:  
-- [ ] Initializer: resource allocation for different stages (2025 H2)  
-- [ ] Hidden state Output Processor (2025 H2)  
-- [ ] Multimodal Output Processor (2025 H2)  
-- [ ] Interface for API, Gradio, ComfyUI  
-* OmniEngine:  
-- [ ] Basic Diffusion Engine core: Diffusion step manager, Cache manager (2025 H2)  
-- [ ] Pipeline of different Engine Cores (2025 H2)  
-- [ ] Communication between Engine Cores via AsyncLLM (2025 H2)  
-* Model support (Included but not limited to):  
-- [ ] DiT: Qwen-Image (2025 H2), Flux-Kontext  
-- [ ] Unified-MLLM: BAGEL (2025 H2), OmniGen2, Janus-Pro  
-- [ ] Extensions of AR+DiT: Qwen-omni (2025 H2)  
-* Optimization  
-- [ ] Integration of other DiT acceleration modules, such as xDiT, Cache-DiT  
-- [ ] Disaggregation for AR and DiT  
-- [ ] Integration with vLLM's existing KV cache for potential future hybrid models.  
-- [ ] Support for streaming multi-modal inputs and outputs
-
-# Contact
-
-For any questions or feedback regarding this design document, please contact   
-Han Gao: [hgaoaf@connect.ust.hk](mailto:hgaoaf@connect.ust.hk),   
-Hongsheng Liu: [hsliuustc@gmail.com](mailto:hsliuustc@gmail.com)
-
-[image1]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAALYAAAFwCAYAAAAVNoEZAABxw0lEQVR4XuydBVRcWb6ve90709OStnTScel4x93dIUaEuBJ3dzciRPDgEtzdCRpCBCIEEkIgQHCCBHe+t6uSvNvNnZ55r2/fng5zvrX+69TZp4qkan3nz2/vKup88sm/OWqXL3HxwnnOnj7JubOnuaB6jqtXLqJ28TxXRWmrX8Ha0gIzE0OuX1bFQFsTcxMDLMyMMDXS44bWVRxsLLG5aYKuxlVsLMxwc3bAWF8bNzdnHGwt8fb2wNXZHlcnW3HMDjsbC4Ju3cLO2hwvT3ecnexwsLcmIMAHHy83+X18PVxxtLMUt+1xcbTBw8URRwdr+f1tLM1wd3PB08NZlAs24ue4uTrKf0bD5yfxb8qWDWvZtnE9W0Vt37yBvTu3cWDPTlE7OHviCBfPnsTEwECIfg7Vk0e5fkkVoxtamBneQF1NlSsXTmMrZJaN6elcF7LexFaIZ6SngbOjvfy2l5DUSwhoa2mK9U1jfL09ibwTKaS3IPx2BA4ONnh7uuHr4yUX1d/Pi0B/X1xd7PHz8cTT3UUc85SfIMHBQUJ4a3EfH/mJ4OsjThoXB/njPNydJbEl3rFisTKrli5i9fLFrF21jC0bVdi2aR07tqzn+KF9nD91HF1NDU4eO8CZIwe5fO4UBjoacpEvnDnG5fMnsDQ1Rk/rOje0rwmxLeQd3chAC0d7W6zMjfBwc8JHiGlmpIu5oS6hISHcibgtOrE99+4+wE50dR8hraeHm1zOW4G+BAb4yjtyYICf/FiAED0wMICw0DDxODvR8QNEBeLv7yO6txPeotN7e7lLYku8Y8akccycOhGlGVNRVprJsoXzWLlEmbUrl7Jv5xbOHD8sBD/ApnWrOLZvN2ePH0FDRJVronMf3rdDiH1aiK4p9s+JE0AWS6xE59bA2EAHO6ub3DTRw1UI7Ovjg/oVEWV01Llz567ouH6i+wbz8FEMtiKaBPr7yzuvn6+nED9I3rH9/bwJDgqUSy3r1OHh4YSEBOPu7io6/W1CQ0PFSeAvxHeTd3B/X19JbIl3TB4zjOnjRzJz8jiUFCazYPZ0li1QYrXo4nt3bObE4QPs3LIRlRWLObh7h9jfj5rqGdGtj7N/11Yh9hk0r14W+yfQ0bgiZLZC8/pljPR1ROwwwdxYF2d7GyGfN1cunkJfiB0h4oe3lweRkfeIjn4iF9vf1wcnkbX9fL0ICb4ll1kutrgtkzdEyB4ZGUlQ0C35Y+/cuSPv3oHimOxkkHX2W6KjN3x+Ev+mjBzQmzGD+zJxxCAmjxrC9AmjUBZyL1dWEkJvEF17G6tEB1++aB47Nq0Xsm+Vy71PbHdv2yTP4LL9Q3t3onVdTYhsyCXV02LyqCuyuS762texlk0oXV05e/IQetrqBInuK5v8RUc/JiLiDvYiisgmf7Y2N0UssZJv1a9fFQJ7yiUPkXXrsBAePHiAn+jMsghy//590dlDRRTxk8cW2VhoWLAktsQ7Bv7UleH9egi5+8kFl3Xw+TOnsmTBLDGpXMsOMaFcunCuiChKbF27ml1CbtnEcvt6FbnY504eE517uxB+CxpXLwlxNbhwXjbhvCGXWFOMmZsZibztwMmj+7khsniA6Kwurs4iSoi87OKCuuj4urqa3DQ3xdHBVt6pXZwd0NJQJ1AmbGiw6NC3xYkQLbK6lxA6WC55yP8VWxZrbhEWHiKJLfGOPl06MKhnVwb06Cy2XZg0aqiIJJNYNHcG61cvl2dtZXF78byZrF22iHUrl7B53WqRwxcI6dfLO/WW9Wvk3f3i2VOonj7BeRFT9HW0ULtwRoh/FIMb2hgZ6otMLuvql9klos3COYri3xrOiEH96dujK907d+Snrp3o37c3Gzdu4MKFCxjqG7Bn9y7uRt6Vd+gHUQ9xFsKHiMmnbEy29ReRRbaaEhYWJo8qDZ+fxL8pfbt2pF+3TqJ+ZHCvLiKODGX2tAly8VRWLJFPJBcoKcpFX6E8j9WL300slyrPkXd0WdfeIE4A2UqKbHlQtpJy7tQxtNWvckZILYspOprX0dFSZ8+2DezcqMKsCaOZOGwQvTp1oHPbVnRo2YL2rZrz/TdN+O6rJrRp8YMQvA+q5y9w9NARbgUFEf3wEffuR2FrZyUXOiIiQnTyEPmkUSa2LNLcu/dAElviHb06taF357bIBB/cqzNjh/YXOXs0cxWnsWjebPlKyVwRTebNmILyLAWWzJ3NMuW5LJijIO/o61YvY/WShWxUWcmB3dvZLgQ/emAvZ04cZc/2zULmTaiKuLJv22ZmjxvJqH496dK6pagW9Gnfhp7tWtH5h+9p//23tPruG1qI+v6br/n2qy/47K9/odn337FGRQUHJ2fCbt/GyFhPxI4Q+eRRJrivr7d8AvngQZTo6tGS2BLv6P6jELtLO7nY/bv/yPCBPZkiBJylMIV5QuS5M6czR3EKStMnMU8m+5yZLFGazdwZU1mzfAmrly2UTyzXr1omjyMb1qxgn8jch/fvYdsGFXZsXMMusV0i7j+6V1eGdm3PGBF5pvXtztxBPZjetyujxL8/VPw/erRuzo9C8pZNheBNv+b7r7/iy88/o2PHjixcspTbtyO4IbK4bMnvzp13KySyzB0cIptYRktiS/wXnzdpzpdNmtGtQyt+6tiK/j06MXbEYCaPG8WMKROYOWUc0yaOY6ro4orittKMacxWmMysaeNZtmAei+fPEZ19FquWLmSdkHv5ovnyzL1dxJQ1Qvr1ohaOG8rxJTO4ukYJ/a1LODprDEsGdmdkux8Y2KopHb/+knZffk6LJp/xvRD5e3G7+Zdf0PKbr2gtJG8jZP/uqy9Zu24tqmdOEn77jrxjy96wkb1BExERyd2794m4c1cSW+Idn33ZlL99IaJA6x/o1KaZ6N7tGTmkP+NGDpXLPG38KLnkk8eNYPqkMXKpZ0ydIKQfK2LJHHkHV1aaIYReIM/jMtFlgss691KRzeeKn6G7YwVup3fgr3YAw83KLO/VkbX9OrNhYBfmd23JrC6tmNT+B7p89zVdRafu1uwbOjf9ivbffUUHsS/r4h3F/69Fs2YsWbhAZOn78owdGBgol1vWyWWTyYg7EZLYEu9o/cVXrO71E6aL53Nx1mT6iLw9sE9Phg/qx/jhQ+Q1doTYjhryXu6xTJNtx4scrjCVOSKiKImoIsvispJ19MULlEQGn8GsYf05vWI2+aGOZHqb4LJvNY+v7iP2+hEeXdpL2NF1uG9V5ti4nuwb1pUlPVoz88cWTGzblHGihrVuypA23zOsfXOGCPE7t/qBZt9+g7qmFmFh4XKxZXlb9kbPbdG1ZZ87afj8JP5NGd2mFfuHD8BCeQ43F8/hzMxxzB3al5H9fmJo/z4MH9CXEYMHMEpMKscNFxFl9AjGjxjERNHRFSbIJB/N9MljmDV9MjOnTUJh8nhmT5/CxNHDMNu/luI77pRGB5Dhb0t1zB2qnoRREuxEnpsx989sI3T/apzXKXBz6WRMl01Bf9F4VKcP5uj4Pmwd1p21g7ui3Lsj83p1oEerZrQV3fv7pt9y4uRxbgUHERIWRlh4BMFC9FvBoZLYEu/YObgfp0YO4ZLowtemjhViKXB1wVT2KIxBccRAxorOPVgIPnRQX8YMHSjv4GPeSz5p7EgmjhkhjywyoadPkmXxMfJMPmHoAOJstKhLjKLiaTiVCQ+pSY6jJj6K0jA3CjxNSdI/S4L2MR5c2sW9CzsIOqKC156lGC2bivr8MRye0I+dI3uyelBXlvXvxMA2zWnb7Ds+//QvjBkziiAxaQwJDRWTx3D5kqC/9Ja6xAcU27Rkcgvx67/59yzo2JrDo/thvVYZj51rCT20leCDm7DfvALN5XNRHj2UcQP7MKRPV9HJe4suLuvkQvZRw+SCTxg9XH57pDgZji6bSXVcJHWZidS+SaMuN43alGdysasfBlER6UNpsCNv/azJctQh1eoacdrHuX9hF67bFmK+SpFz04dwZHw/to3sxfqh3ZgssniXZl/LlwT/+p//ia+vn5g8+uPt7SX/YJSrq4sktsQ7un39Fd2/+ZoB337NpJZNWdezI/pKY3FSmU+o2mVCjc3xd/LCw9kHbSNrDl/QQFFpAaNHDGNov3dZfIzo3rLJ5tiR7/L4zJGDCBQ5ui4tntr8TOpF1WWnUJsUQ+2z+1RHBVF1L4DSUFeKAuzJdjEkzUaT+BtniLq8F+ftypitVOCsEPvoxH7sHt2XzUN7MLtHGwa0aEpLIfYXf/sMS0sLPDw8cHR0xN7ORl4Nn5/Evymd2rXgxzY/0EF0wj5dO3NE2xTj5CLMC+rQL6zDtKQe+/I6rEvrMS6owfBNFQY5leimlnLI0pOl2/aLDN6X0UPede9TG1dxS/sMiO5cHO5DcYgHed4OZNkZURbqSfWDYKrvBlN+25/iIDcK/Rx542JEpp0uTzWPEXl+Jw6b52O8fDrnFYZxcupg9o7pK7p2Dxb3aYdil5a0+eYrvvnicw4e3I/lTQsM9G+gr6eL3g1dSWyJd3T44Rt+bN2C+bsOc9w7ApMiuFkMN/JrsSyuwbm8FuuiWkzF/s3Caizf1mCZX4VpbgV6yW+5/iCRpVv3MkFMJBdMn0BehDf50SGQLaLH6xfUy+JHVCiVYZ6UeltRGexO9T2xHxFI0S2RtX0cyHMxI9P2Bo9Flw8/vQ3bjfMwWDqVc0LsE1MGsXd0H7aP6M6SPu2Z3bUVbb9pwjdffsnKlcvQ1rjO1SuXuHZVVpclsf8gNomSvdgfKuaXh//1qD3O4fLrKjRya9HJrUE3uxqr4mrcy2vwLKvDMKsKSyG0s9i3Lxad+20txqJj3xRlJeQ2yyhBNy6Dq0GPWLN8Kbn3g6jNEdEjL5uHphZcO3QRNyNHEnyCqY2NpfbFC2rjnlBxL0x0clkMuUmSmQZPdS9wR3U/t4Line truncated
-
-[image2]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARUAAAEOCAMAAACQIXOIAAADAFBMVEUAAAAAAAD/AAAAAAAAAADfACAAAAAAAADvADAAAAAAAAAAAADqACsAAADgACkAAAAAAAAAAAAAAAAAAAAAAAAtAAbmAC0AAAAAAADqADAAAAAAAAAAAAAAAAAAAADnADAAAAAAAAAAAADqAC4AAAAAAAAAAAAAAADpADAAAAAAAADoAC4AAADnADAAAAAAAAAAAACxACQAAAAAAADoADAQAALkAC8AAAAAAAAAAADoAC8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADqAC4AAAAAAAB/ABoAAADpAC8AAAAAAAAAAADqAC8AAAAAAAAAAAAAAADpAC8AAAAAAADpAC8AAAAAAAAAAAAAAAAAAAAAAAAAAADpAC8AAAA0AAvhAC0AAABXABLqAC8AAAAAAADpAC8AAAAAAADqAC+rACLpAC/IACgAAADGACjqADDpAC8AAADVACsAAAAdHRoAAAAdHRojHBowGhw2GR08GB1JAA9XFSBdFCFmABVpEiKQDSWWDCenACKq4+q8Biu9ACbCBSvTACvcAi7jAS/pAC/4Y4L9gpv+h6D/kKcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAn++oGAAAAdHRSTlMAAQEECAgKCxATFRYYGRkaHSAhJCgoKCsvMDY3Ojw+QEJFR0hJTE1QUFRYWF1gYmRoaWttcHJyc3R3eHyAgYSGiYyNj5CSk5SXmZ2en6Klpq+vsrO3uLu/wMXMzc/S1tbX19fg4eft7u7v7/H09Pb3+f3+/iGfBl4AABcuSURBVHhe7Z1dcBNXlsePaeE4MW23jLyOhASOsPiwwYAXg5kwZAOJcajNkDDxUMkkVcmmaqt2HlJTNTOv+7Cvm62amtpK1W7NJFWbbDIDMyRVmYUAkxAIWcAQMAx2+Su2wY6FQTYysh0jS2bvR7fUfdV2d0stS2Lv70G6ffuclvqv27dvdx+dC8DhcDgcDofD4TwcFLEVOWNlcR9bZRHJlSjGB1T11nGwFblC2Op/m62zyMrmRDH4kKhiA/c6AMp8oVFUDLPrrPEwqTI4CFDvCx9j661TMKqIq93SdNdAFB1rK4pvxn114pEoSOt8UVKnh7DBI00PXSZF7NIiXhmgLsfm8UhQKKo81YhfA6N/moai5mVHdvgAWg97D+C64GHdndzwpIheA9Hr6LXoWRG7eIN/XPHCEoBlf7zPWmspFFWWD6MfunpvVfMnaMHxk+73wN3y3OOdF6D6aXdNJ2uNqG+Gtg4o2tIyR9ZiF3eLu+XxLuyyWs9DxRK2Ik85/fvuKAwehdJivBQ8PjZ2HbWco2Nj31yGNawxoqIJjp4ZGwudhC3EA7tcP4FcjiEXWGMwICkUVcbI61TESRr3FXLMxNrwawhKE1ZJvGVBOvy5V0lHMdjl7lzs4gO8QLWdn0I5ggBcFStcS8UZXIyRE29kKXm7qzZKUAM3aG8T8ksjILtMTRYTlzmVoS6Foor/Wdx34v0zy7hSqFDXmqNQVDkQ+7pnDCpal7IrskKBqCLApXNs3YKElSbiTDYa8xRIb1tE982LDyNThGAz7YSdMzeYVSYokLYSg331E0vW90bMHkHXhl/6p9hAdMn6ecZ4C1MgbQUugrdu/fmv2Or5GT805AjUrY+lI0oe3V8xQJDkMYt5kMuc/nmbw+FwOBxOIVEw45V/YyvU/IKtyJA8GPHbub8LbutXhvdVFAplxL+4cFX04KrowVXRg6uiB1dFD66KHlwVPbgqenBV9OCq6MFV0YOrogdXRQ+uih5cFT24KnpwVfTgqujBVdGDq6IHV0UProoeXBU9uCp6FKAqcdOPANMm/54zrxueZKu07IHP2aoENj1RzT+W/WwZW8WwZw9bYzf5dwRNdm9nqxgurDXSLVPyTxW002wNg7FumaLuV36lKueSx6bZGoYlUtjWLuJt8qchFXnYVmDG6EvNzZSwVfZi9AVywZzhPs+UZPd7Z3fraWK4z9luLEafnxOM9zn6aFa/OBsX96/Mck74m5d/ZzCSgxfvzT+Us8IvdNXVrcw1tw+9aTQi+bgsm0O5vFQFgjcNRyTnN7A1NpKfqqB9NvrTWPAmW2MjeapK8OYmtorlvJFuGZCnqsD5LUY7HTTULX3yVZVgyHCnDXVLn3xVxURjMdYtbfJWFePGYqxb2uStKsY7baxb2uSvKsY7bahb2uSvKvCxbytbpSX459ezJEseq2LcFoybU5rksyrGO22oW5rksypop9kaBmPd0iOvVQmG2BqWLDWWvFbF+Fon+H1WGkt+q2J8rXMpK40lv1UxvtbpykpjyXNVjPc5K40lz1Ux3ueu7/VSC2ZInqvS9dVBtorlgy3r2KqMyYEq9b/8qUHCQxVdQPd5AadLBhcGacA++UiT3Q0fKnn/6kme5tjoZBtOSpwxl7Z2sVUMXfarYk9bcfhhG1OzYu1r2po0URrLAlxiKzLGnrZSK4E3GUkQ/HwWwLdd3HlWbZMuJhrLOiMLq9jSVoQauFRSnVweHxsba/9deNV8PYElTDQW2w8hW1Qpr4z1KNl1E8TaaC7ajDHRmxrqZhFbvrhXRNdxlS42z6pDYsJzxNXrH1zpVpaEjQEHDH4j5/6rDrge3OgiSe+EljJ4cENZIZY43/j+iuIEwgHFabO7v7u60XF6xMRBZhE7VHFsgxvRsORnVHHR5LxJSI5ar7IHFa041eaK7b/GC9JeH3r1Crgnoiu82/+C84Bjp+KqsPcUdaIrV2w/0YnW+COPoPOdNILOQjb3LHaoskvqOwtHWpvGVTm6xcpnxXc0TUXY7w++h35jHxmXoaW+/yDvL3yCF2bel8/j2IyuaNl0OEqd1m394JfKSuwl7N+3BmcVb+qTp3v44FWwVRYbVKkIQA/AxB1xjayK+y3y1qk9fjz+2Cnc8Ef6/XSJZL2O9+16bNrjhy+UwQ02Iysu+t01ndQJtQXipHjF+/zklDdzQnay+xiyobf1ijOD+Juik7OqNvbZUfwm1BCKcfrdO+QmUpxk+Q7ISyTttRPk1N/qFUGcuFd2urSVrldWku4HYDihu4kTlRUybyvotDzhQe9LImIDHaAEUcvf2dgi4aXyPbiTiB0aARdMaZKqyi0KYtK0dpVcfjAuuRSn5PBV8SI9uepWXZetjSVzVcoroeoFWlzVpuxc/LTQsBWrMkFDkkI4raoatPQdLT3QrEo1I9BdxisTXlqubrGzw81clc2i/E3LRPXJ+YxfKo4qxwsirj0hhaEP95fKQim2xcRVZaSl4nT1aXmlykvD7BU7G0vGqqBLoPP0wHG8rj45x44f3MaM+OlFATriAO+w6hIhBE5HUgm5XFRBcmVTs9mAvFLlpcXWxpJxb+uWYv20hN43q/rbocvqJYBeoBcF6IgD3F+qLhGGIyUNKeVaCXXiihMUk5GzxkvL7BUbQ8IyVUVoVM4Zyf2WaS8h9xQURvqhGfXK0gGSJx0vkXh9AdWN90LTTlSW1tIyrvfvgPZpxQmkZY8oXsTph7hSy1Wnm61Km0yPIPTLy1NH4HOppAxZMONQU6tajH9ZKb7S4fQEr+A+Ai89/1w3OD0foqUzlb6mrbfKxfPocuC05P/5QMwrQudZ2Sk06vR89qIge/0cO8W+TW5YZvZK08dsXbqw/27IbrytWDquPjsvd96NJJaXl95SyqIzel8ektAl5LX09T8HqZ3aScNWn3VZSLxtyr8bMm0r1ohENItj6qzpqgWtGV1KNAWNk4arRg9gTZNpv7J4XHUZ9RuzySvrDCkcVVBjYatYrhrpZpbCUcVMYzHUzSQFpIqJxmKom0kKSBUTjcVYN3MUkirGjcVYN3MUkirGO22smzkKShXjnTbUzRwFpQpcGn2RrdIy+9FP7JClsFSBaysNdnryulFzMkOBqWK80xd8BrqZocBUgQsr2RqGyQ4j3UxQaKpMkkdnC3Ghqgy9rlvF1luh0FSBC+Tfqwvs9GRvI1r/BpYmbQpOlcntaJ9fXWinLwQ2vvoGW2mNxb2/Ygdrb2/ayNZp8MZfZausUniqzC68z+u2LqyZKdg7lPlN8WPKrzj1vWaFgpCYeXkeA11S7lAWVr9imP4rHpnncZE1CkwVw51+MB0xUs4EhaUK3mm2iuX+vRhbZZkCUwXttOE+xyYsdCn65F8WPSOW7sFP2T74K1uvpu4Zz8IGBhRcW4HZz/6LDUtMoeMjtsYa+DlloXFnYLnr2m22VsNUycTCBg8jS//ecKhmmBpqIbT9yr9olhj+OVk0a7eQYdLMlBGTd/OB5osnZ0W0Jz1n4fUrlOyeJQpVlezCVdGDq6IHV0UProoeXBU9uCp6cFX04KrowVXRg6uiB1dFD66KHlwVPbgqenBV9OCq6MFV0YOrogdXRQ+uih5cFT24KnpwVTgcjt2YiNQQfJWTcbby4UYbb1u/610ajlfffJT+mb+++TdzzcsOGcbR2APJhwXwG/3/ti8eWlWGS6qpGC6oIO9CDUQXMVLine truncated
-
-[image3]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAPwAAAEqCAYAAAAxuz3hAACAAElEQVR4Xuy9B3QUV57/246AMTkpC6Gcc84555xzQgkFQBIIRAaTsQkGg43JGWMwwQRhG2MbbGzjgMceezyzO7sz7+y+8z9n98y+s+/N9/3ubZUolVpCIAkEut9zvqdbVdW3qqvv5/5+v1vVLZVKSOgR9e3p43hBpcLoF1UYO2YMtKcbQl/bgmyNmbqOmKXvAjMjd1gYe8LazBv2VoFwsA6Cs00YXG0j4WEfA0+HOHg7JcLHOQn+rqkIcEtHkEcWQrxyEOqdg3DfAkT4FSLSvwhR5OigEsQGlyIupALxoZVICJtNrkJSRA2SI2uRElWHtJh67oy4RmTGNyE7cR53TtJ85CTOR25SKz22dDk7obnb86z4FmTGNSMjdj7SY+ZRm01IJSdHNCApvB4JoXWIC6tDbGgtYkJqEBVUhcjA2YgIqES4fxnC/EoR6luCEJ9iBHsXcQd55Xe6AAEeefB3z0GAeza930z4OafBl+zlnAovpxR4OibTeWFOIifCwy4BbnbxcLGNpXPHHANn6yg4WUXC0TIcjhbhsLMIga15MDkQNmYBsLUIoPWBoI/peeXnJiT0SLp7+gQHfsxLz2PSuHHQ1zGBkZ4dge4IE0NXDruliResTH2oA/rDwSoYjtYhcLONgrtdNLwcGewJ8HVJhp9rCgLdM7pgD/PJ47BH+jLYCxEVUIzoQAZ7GYEuwV6NxPBqDjsDPTV6TjfQsxLmdoGem9yMvOQW5KUwL6Dn980GAOlRPRi0EvitHHwGfSpBnxo9FymRaugTw+Zw6JnjybEEfWxwNQ1IlTQ4VSDcr5wDL0GvBr+QO8irEIGezLkEfi78CHx/l3T4kRn0PhL0nfZg8Nsnwd0+Ea4c+jhuJ+sYOFpFwZmgd7KMIOhDYU/Q21kEqW0ZQANDsABeaPD0bSfwr7z8PKZMmAQDHXOC3QnGBiyye8B8lidFdy+C3pt3QDvLIIo6YRx2Ft0Z7CyyM9j9OyN7sGd2t8gexeyvhj0miEX2cm457OrITrBH1yM9tkEz7AS6BHR+ysIuS9DL/2bbdAM+Vh3lGfDMLMpL0MfJgI8OmI1IGfDB3sUEOIvwDPpCsjraM/AZ7Mws0vu5UpR3zaKBrxN6p1RuL0cp2ifBw6En8E7WauCZHcxDYG8Z2gU8y6ZcBPBCgykG/EsvqPDqmJegNVUbM/VsYDrTjWBnoJNn+WBOZSOWtS2jNNMfh/cewNIFy6mDRlDUiqVOHE8dOh6+BHyAezqHXR3d8zvT+EJ6ZLCTA0opnS8lsMrJ6jSeOTmyjiCsQ3Pda1i9ZDt/nhU/9z7wiQz2ZhoE5uGTa7dx8J0zBDWDu60T7s4on0iwdz7PSVxAbbDUXp3S56e24PrlGzRwtCIxlFL6kDk06NQS6LWU2tfQ8VUjMqCKBiYC3p8BX4ni3Ga8vm4XNq3diezkeg5+CIHPonuQpxp4P48c+Lplw8clCz6u6TQApqrtSGaRniJ7kGcGDYapcGMR3lad1jM7WsbAwTKaR3cHy0isWrQKrvYRPLW3swgm+INo+1ABvNDgiQH/MtXv48e+BN3pehTd7dQ1+ywvbjMjL1x67zi+/PQyjzofX76A9pbF1NkT4e4QQ/VuFpKjizpr98zO6J7LozszAz01ppLS+RLkpVL9HFmNyoIFKMtt5ZE9L6WJIno9CtPnY9XiHWitX4uKvEWYXbgUORTVm2avRmV+OwFL0Zpq819//hUH9pxAWc5iAn4Bf6zMX8oje2X+Esyrfa1zAGhFddGyTvib0TZ3A3758Seq61soujdSPd+A0qxFdOz1vI7PjKesIoGl+jUEfgXBXYqOi9fw8ZWr+OLGDRpkjiAquAQRgaWICS2her8YkUH0vj1pcPPLo9cUwI/ef7hfDg1w2QR4GgfemyJ70+wWZCWWEeTxiAjIoOyIojvV8X5u8Qj1SyXwKboT9Ls2vU7Ah/Mob2/JgA8WwAsNru6+dwJjRlE6P340jHRnwdjQkafwViY+vHZf0tqOvPQCHNv3LjxdonD1/PuoKqunx7Norm/D9s3bMa9mIZa0rKYUNgUhBDyr3SP8iuixiOrmKpw59gHam9fh4N5jOLr/JBY1v4bjh89g+6a9+PLmF6gtXYKvb93htfvONw7gP/7+77RuH4F2C3t3HcVP399DTsICAnQx/vzrr3h313H88d7PuHn9C3R8eINeexenjpyn7X7C6qU78ObrR/DNnbu4crED1698ijtffoszxy/g+6/vUjSfQ4NOI76j52to23vf/YAFTevx6fVPsXThFpyi4wom2LMS5+L8+xeQGl+H/XuO4EZHBz66cg2b1mzDv//lD9izfQ+dgwvY99Z+Op49OH34KN7cvBOH9+ynQacNW9e+TmAnID6sBPt3vY20+CJsXLEBZXk12L/zLRRlVmFl2wpU5NdgYWMbwR2JhXMXwdlWAl4NvatNiABeaPDEIvyEV16A1qTRMDGwhKmRG6xMvQl2H2xdtwnnTx7HwuaF+ODkMaQl5GHjqg1U26Zg26atePvN3aitaMKurW+hJK8RLtbx2PXG20iPm40IqtmjKIUPdMvFLz99j2+/+grleQvw5edfoWF2O25+9BnBvZ/A/QP27jxMsH5P9fQc3Lj+Oa5e/Ag7Nu3HX//8Ozas2IUlzVv5JFzT7HW4ce0mDS6v4zOC/U+//Er7O4A///E3rF+5m2DcjZ9/+IkGkW/wze2vsap9F5Uja3Dl/DV8/833+OC9y0iMmIu0uPkUuT9B/ewVdBxf4K3th7Bo3ibUlC3H+uW76P2VoyCjFRcJ+KKsJnz31R1sXL0Tp4+cxm8/36VBaB9uf3qDBrwVlPl8hN1bd1PmUImFTStRlt+E8oJ5mFe9GB52yVRC1OG9w4eRGleG948dR0JkPna/8SYW1i9Cflo5qovrUVFQy2v5mJBMOFoR8BZhZAG80BCIAT9x7AvQnjwGZjNtYW7swWfkrU39CPIT1Klb4GIXhk2rN/NavrFqPuLCcwmQZWisXYgPz52nqH0I8+cspYiWhHWrXuepO0vhowLK4O+Si7//2+84/94FxIfOxu5tB/Htl9/g2qWPKOpvxq8//YrbN29ThP0ciSH1uPXpHRzZ+z7amjbhG4rcn16/hdfXvstr9tqS1fj9l9/x092f+Wt/+u5nitQ/4ucff8bZkx/S9t/jzq1v8da247h390dcufAp5lSsxK9/+JWWf43bn32LBErnU6Ln4S5lAN/c+Y4ymK3YuukgastWYnHzG1RSrKPjrkJMQCUNUnfw1c1blD28h8ykRhzaexz/8tsfsXTBOvzxx+8oU6jEiYMncLPjGhY0rMSyhWuQnjAbzXOWUQSvh4dDKjLiqnD60FEaQOZgdftayiAOU7SnMieqiAaC/Ti5fz+/0uHuEI/lC5ZT2h/ZDXgXWwG80CCKpfRTxr0EvWmvwtLYlUd3W/MAfi3Y3IiBHwAHijo25sE8+rjYxsCNOqe7QwI8HJPoMQmeTqm8fmWpfFxoGcK8C3h0T4upxe7tB7Fy8RYkhFUiIYRACpyNaIIpNqiKauk6xIfU8MieFM5m5ucjLbqJoqX6OjubcGP1N6vVi9IXkxdx8JnZ37lJbeSFXY/ZlPZnJ7QgN3ERvwafGcdm9Bchi+r2lOj5VK/PRWpsM79Ex1J75oQwquFD5lAdTw6uo+OrQWxgLT1W8dQ+MrAc65bt5DV+sHcZAjyKEOxTggA39fV4f8pg/F1z6P1nwd89m8/Y+7qyCbxMPpnn55zOr8fz2XoaEF1t2fV49hhP5zUGDhaRcLaOJUfDjkB3tIrgwLOJOwa8uCwnNKhiwE+b8DL0p42DpakHrM18OfBslpjdBGJnHkIdkzqiNcFuF8UjkYdjPLyck+HtkkKdms1AZyLII5sAyEGoF6vf2TX3Uh7RC7PmUx1bxaN9QiiDuxqJYbUEeC1SIhso0jVyyNlMegbByC6lsfSdAcwm4gpSF6MwjWDvcjvB3s4fC7lpfWo7ClIWIy+JnEwDQWIb1fyLqJ02Dj9rKyN+AdJiFxDsDP5mJEc1UYRuohS/iUf9OAZ8SB3iOqGPJrNIH+ZTSduyQaGKX58P8y1DODnMp5TeazGC+Yx9AQI98hHomYcA987LdDQAcPNLdQQ/m8CjgdHTIQXu7Jq8XSKfxHNml+dsorkZ7ErgnQTwQoOp786cwPSJo2A4YyJFcfXNNeoZYmniKJR3QlZjutrF8sju6UQRyzmFolgaRbcMPjsf5JGDEC92o00RvwzHgI8JKueODZpNwFQTVDU0CNTwyJ4SOYfMZsubCPi5ncDP74zM7G45Nruujt55yYtQQC4kqIsIbmYGeX4Km6knJ9Pz5CVqyMnZ8cyd0JOz4hcS7K3cXcBHzkdKxDwkRzTy7IIBz03QxwQR7IHVHPho6VKdXyV3ONX33N5q4ENYxPcg4Bn0nuwOPAn4bG4fgp3Z25ldskuDN4v0BLwHAe9G0d7FJq7zzrtYfo4d2fV4Swa9+nq8o3WQAF5o8MSAnzFpNAE/mWD34zfX2Fuqo7qDpdqsEzpTBHKzj+NpPAPe2yWVX3dmwLObTlhHD/ZUAx/pX4IIcnRQGcFTQVGzilJmAp4iewKP7vX8bjcJdgn49Bh2K6wG4JPuA1/Aonmn1bCzyN7OrQZ9scxq6DM7gWewq4FvuQ98uBr4eErtlcBLsHcHvoI7zJuVLqUI8SwiqyN8EIEf4M5uub0P/X3gM+DllN4NeHeW3tvEy4BnUT5KDbwli/JB/DZmlQBeaLD0/ZmTBPwYGGlPJ9j9qbNRJ7MKVQNPNbsEO7tRxN0hsVt093FRR3d/Sl0DGfBe+Qgl4EMpwkewSbvASl6zx1N0Z+awR8wh2Ot5Kn8fePY4D2lRc5HBoW+hqMzulOse5ZmlqM6e53aCnpvIovt90LPiWNLine truncated
-
-[image4]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAO0AAAFdCAYAAADv8OpOAABCCElEQVR4Xu2dB3gUxRvGlwRCh9BTCb2KVAWkKiDSiwKiCPxV7IACKlbABohBESk2pAkC0kR6SWjSW+i9Kr0TapL3P7N3c+x911I2l1z4fs/zPrmd/XZmd2fendkaTWMYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYJoUA8OtVqnqTj8rUfappcMmmOfyzNhXJKVJw1hxNWwSXbdo9vFK9IdUb59UYhjGHryrUfapFSIlnS2XPs6NOgdA9Imm3WSqcNfvuanmLrK0WGNxLYxgmZcyr1SL020qNhvlrmdCocDHMr9MR0fWfx+qGL7jVGqEV9Z/TtbJBF/1vVAPny8nYn6s1R818IagaWKhnEa1ITroeDMMkko/K1u1QPTDo4FslquNEszeBZz5AwtP9cbplT6FeLvVfi5641eZd3GjVD9fb9EVs63643fY9nG3Z2yFW6lLrdzC71tMQRUZny6zVoevBMEwi6BRSrmWpHHn3BmfLiXhh1GvCfBVzF5TGSpSWDWoDzOuL7Hn8sPLL9ujfqZpDDFWmTPrf2IjQnG9Pi6yVX2MYJvGUyJ63XW7/zIdr5w8Vpv1ADI076Mbq1KIEpg5vhCmRT1g03FGThz2Oazu7Aadfw8JfmuH6ru44uvJZTPu2kUOsUVPF/O8/fgz+fpkOly0V2puuE8Mw7mkrdLBx4QjsaPwSauYNwdLfWgBnegGHXsPtnS/jg1eroGeXiuhlot7u/hAqlgpEZj8NT7Ro+3vjlq3/R1eMYRgn+BtM+8/jXVExRyFsmN4WOPUmTq3qghfalHYY3pqtXLnzxGXLnn1f7wHfBmoMw7jHaNp1j3fTTbt+mjTtW3i1U3ndVN3efAf9Ph+GPoOGmqp+n32Nd78Yhi6vvomQsHCEFS/1xoABA9i4DOMOl6Y92wuBuTOjRr0ncPQOsP9aQop18AZ00fTj94C1B/9DoSJBl1s/+z++j8sw7vBk2gZPtdGNtufyvRTp0E1g75U47Lsaj/3X7fM7GAusP3YOgQUKXmrS8mk2LcO4w5Np6zdtnSLTHr4FLNq6Dy06dLadw/YdNATbTl+3M+26o2flvIuBgfl72q8hwzB2pMS0+8TQ9thd6L0onae08fgFNGvX0XJu3L40nn6yGDL5Z8ankaP0XpeaVohNyzDuSI5pd1+6hw3HzqPPwK9QskRxNG7eCgfEkPfA9QRi2jj0GfClbtj+r1bWbyHh8OsILxyArNlz68vsEUNmfXh8/LyMOyX0hv0aMgxjR3JMK3vXds93swx3C4Xpf4f9PFE3HzXt+18M0+e/9Exp/TYSTr6JIvkzI3uuQJtpZU+9at/xe8FhRd8SseXtVpBhGHuSYtoj4vx0+op1qFKjJrRSlaENnQNtTTy0OSd1YwYFh+gGlMNmW698ORa9P/5MzPfHIw8VQMVSeVAkJAwT/l6mX5jS870NLIs5dGPBgQNZ6foxDENIrGnlFd+f/pyPEqWsD1uMXgVtRSy0hRegLb8BrVUPPX3k73/aneMeEr3vplOXMOi70fDLkhu5Agvjt3lL7a4gy4tVy2IOxu5JSChAVo9hGEpiTLv3yj30/yrSYtZmYlg87QC0RRcthlWSxpXpIqZ4iZK6WdWFJil10Urek5U97K4Lt7FX3v6R92+tt3zEsleFmtutIMMw9ngy7RMt2uON9z5BtoAAaJ37iqHwKWHQ6/aG1XXRYtxGz+rGHTd3MQ6Tq8qyd5Xq/kZvVKxUCQOHj8bW/67i0C3b1ePrQi3s15BhGDvcmTZ/YFb4+flBKxgCrf/P0JZccWJWouViyPz7Lt245R96WDeu7EnlkPmN9z5G3pw5oFVpCG3A79AKhOhxE/5ernraK4ULF25tv4YMw1DaCR1q5My0eQMsQ+LBs6AtvuxoUKe6aDnXfayFvuy0Zf+Ic+K7mLlyoyWvnHmhTT8EbZnorUcshVa5HoIKFcJb73+MHNmyXQwLC3vTfvUYhrEjOCB72xz+/ocq5SmEnU1eRiWjacXwWCtUTBjsmjDtJScGdaMJ26HVbIpHHquLdz79AqVKl4UWUR7ax+OhLb1qOSdeIYbTU/dCy5YLosdHYGDgjR49ekyh68gwjAEMGOD3aGBwvWyZMs8pn7sA8vtntzftI80sJqOmTIyib0Gr2tDSwxYIhrY2wbn5ZZzozbOKofjSpUuRkJBQiq4nwzCECrkK1dUyaZeFEjbMaGeOaaV+3gDt4ToW4/60ztJr0xg5VJ59HFlFzOzZs6Vp36brxzCMEyZ81ahA+WKBH3dvVwZXdr6MgnmypNy0UtG3oY1ZDS0gK7TIBY7zlwojTz+InH7arSlTpqwUpq1K141hGBe0bhxeI1s2/7stGxRFzqx+0Go1T7lppeTFqS9nQPvziOM8q2kLZvW/2LZ1C/5mFMMklbLFC1fKlyvLy+LnXa1mM8vwVV44Si3J+77CtFrhkMs5m/D7tAyTXLJoWXNt1IKLWe69Lr5iue2TGlp5B9rwhdCCwo/6PdbieboiDMMkkor1Hn9MCys1TcscIM5Hs0PLki115JcZWbLm+Dc8vEAIXQeGYZJIoT7fNG3+ZBM8Xq8uGtatkypq0qBefL0XekTRshmGSSL37t1rGR8fvxxeQpT1XUJCQkm6HgyT4RENv49V7em8xCI8lEWYKIr4KlUR5V0UGkHXhWEyPAYTXBbG/VT8HCj/CtUSv/2E/KXEdFZnEvNyiV62lcrHm8TFxV0U61Dauh4B1r/+dBsZJkNBjeAKYeqrwiSXqcSsKzTWm4j1ui3XQ/y9JHTNmnZEGDiMbivDZAioCdI1wpPnb9DE+1w7fwZnrt6Vpo0Rpo2g28owGQLa8G2cmQGtwPP4S76Jng7YNPYl1KzUBDfpDAMbv2yCoGLNdNNOB4rSbWWYDAFt+IrYqCEo+MxP2HMpgc5KA+LQrHxpNB8UTWfYcWHtp6gZVAL32LRMRoY2fAvnMLNfQ7w5Zy9ui6mXH6uCRzoNRO0yRRFcpAiCitbFxpVj0aNhBRQuVBghD3cRtrIw5IV6KF00DEFBwQgNL4n6Lw615bpqTC80figcwSFhKN+gC16p/yhavT8HuHMKS0e+hvDQMISFh4rlyiLmYrx1qVvY/ctz6D7hMOThI+HKbvz+YRuEFyuGYkVD0azvROy8ZIm9c+Ef9K1REPPuxscMOMymZTIoNkcZubseg9sWx09bzuuTFXNp0HKVwLglG7Bu7rdoV8wPDz/eCj0+m4yZn7REsMjmrHTUjc3Im7cMnhk4GYdiVmDM241RwL8A/pOZJFxFxby5UbH9R4hZ/Qc+eroycmYLQqtv9yDqm2cQmisPJq3ciyNHtmHqhy3Q+puVOCu8GHdlHfrXCULUVcuqbR7fA6VzhWLnieM4tnspcmQqgI5jtuBigozdh8/r58PPV+JjfrzJpmUyKMqnRs4tGISGBTTs0y/6XBZGLI93l57T513a+Qc6BflhtfVU95/IjiiWORyx+lQCDh/Yi107tiFq8SyM7N8RZTLnwRVcx9/9G6L3zL24Y1kMuL0XBct1xZwT/0LzD0erj//Eju3bsV1oR8xi+Id2wsy9N7C036MoUKiyWgo7ZryLGnkz4/XIGdhw8LRIice9e3FIEKa9d3k7PqmdD7/cYNMyGRibGwysG/kCimsB0PvZu6uRv+LrWHHpnpi4g4OL3kc5/8LQ760IfnulBnLkeRLSsJd2z0eVMsUQHhaGsuXKoGRwLmgBFUUXuBlftQzB9Jj7l37jrq5GyZaRiPl3BzIF5EZIibIoWbKkrtJlyyLsiQ8QffwWvm4ahoIlnrUth4QziBrdE/kDNGQNfgT7r6iBuRhIn5mPl0rnxyIeHjMZmftuuM+g5kWgBXXUf1/88w00HroV1+Vp47WD+L1rBAJK3X+WokNpP4R3+Q1357+JMvlz3O9JD89Gt5IactV8TRh/A4a0K4rx2y39Me4dwYTuFdB16jncPbcdOav3wJwLasEE7Jo5BH/tkbeAgedLi3Pmpt/ov+PP7kSn93/FdlXIv7OghXbF3H2WsfOeMW1RJF8JxMXH71jA92mZjIq1+dvRpWwmZHt8EOTQc9WnT+DtJaeh97Mn1+DdatkR3KCvLbZGzmxoMiwatxb2Qrm8WXFKBgq++V9lZNH8UO7pwZBD7DWRHVD9peFYsikGP/ZriQjRUw7blSCvHiGrfwhaD1qAy6f3Y9XkfqhTJDPmH7EYUZo2tPlw/XfChS3Qcj+C92bs16dvn5qFLBV7Yckxy42gb5sXRb6yHeUtn50H2LRMRkVv7YSxg4di+vpj4tcprJwyFidiLU68emIVfo0chT9WHbFGXkXk8D+w+aTlBPfSrvlo07QJmrXvivUxe7Doh4/x1bgl1lhgxe8j8eWgz7A26js0L14IJ+9a0mNPrMPU7z5B15d7YsD307Dh5P27sUObRyC46lu26cOrfsfgPi+iYaMWeO71j3BJjY7jT6CgMOzEffJ6t/5U1E66rQyTIbC5wSSuXbqEaxbf2Di1diI+6NrdNn10THuE5iqCe4m4Bbykb20EFWtBk3H+0nXbbSbJqXn98NyoTbaHL+RjjXRbGSZDYGj3qcaFpV+gjhgOf/LzdPw1bwYeC82OQo/1oWFOOT2vJyqHVqTJ9pxZgj71y2IXecSRbivDZAjsm3kqEheLzevWYNWq1Th+4Rad65qEWzi6YbHLxxcTYk9g3cIFWHvQcuHKCN1WhskQ0IaeXnE5kpY3aF1At5VhmBQQpWmBUjSdYZh0iDRrtKZtk2LjMkw6x2pYGMXGZZh0iqGHtTOtTPtb0/IZYxmGSWOc9bBUbFyGSSe46WGpuMdlmLQmMT2sE23TF2YYxvsI07YVGigVrWlHox0NqnRUxUnZMmAYJu2IFh6OdjSrUtT9SIZh0gXRbFqG8S2i2bQM41Line truncated
-
-[image5]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAR8AAAEwCAYAAACOrXWuAAA5IElEQVR4Xu2dB5xU1fXHH21FsSCisrTV2LBFMAhG/EfFilEES2LDQiIREZKoUdSIGI1ExW6CGAEBFYMoTXrZoQlIbwssbQHpZWlL3z3/d97M2b1z3ntT9s3dmTdzfp/P77Mzt5z35s6737n3vrKGIRKlkHZt2wt+t6lK/HOJRKIUF+/IfrTARyTyoXhHdvLu7ftgz84DtvRUscBHJPKheEd2c+GO/ba0VLHARyTyoXhHdnJg0jRoddvtsGbleji496gtP9kW+IhEPhTvyE4e8r+hYBaFvMUr4ciBYlt+si3wEYl8KN6RyTjNKtwenGp9/+1wyMrKguVL8i34qHmpYIGPSORD8Y6M3rGlEIYOGQGjR461Xg/7bmQQPktXwYE9R2DQV4PNvHGwd9dBW91kWOAjEvlQvCOjt2zcAS2uuRZa33GX9fqHYaMt+KxYthp2bN0Dl15yqZV39GCJra6b9+4qgn279cBK4CMS+VC8I6M3b9gOJ5xQHc4551zr9ajhYyz4rFy2BrZt2mmt/2BeyVGw1eXGKdpOE1jjx0yC774dbsuPx3jKH83TBT4ikQ9FHRjXcMCEyeEDxRZwatQ4Gc4773xX+GBe8RGwAFRyJBxCCBuETvFhjHcctm/eBfUbNLDq4TQO141wW04gcTOeaZs8caq1D/y0v8BHJPKhDu8/bk2Htm7aBT+YkJk1Y27s8DEBgms/Y0dNCAMJwmHrzzthwtjJsH7NJgs4F154kVUPy82aMcfaFk7pOGRU7y88BIfM/cMp25gfJkDbu+6GwYO+M9MPQ9HeI1Y+xhP4iEQ+1EMPtoOB/QfBqhXrLDjc1PJmCxyxwMfq9ZUqWe/xCmha10GgLVu8ErKqZUH/fl9agLjookZWOQTHTTfebL1esmC5VQ/r0GjGOpNmGtM//7QvPPLwozB39kLo93l/qFa1GrzdoycUHy+BF194GT75sBeszd8g8BGJ/CgERFezI6/NX28BAS8mLA98cCSCp+Q/793PGung4vSpp5wKXw0cZINPq9t+G7puKB9mTp9j1ulrgigPdmwutEYz5Kef6gInVj/RHF2Nh/59B8JJJ54EPd96D0pKSqDZVc3hkXaPwdKFKwQ+IpFPBX97rmspfG5v9du44YPGNZxfXdm0FCqrlq+D0049zYTPNzb43N7qDut1ft5aePnFV6z6A8wREq4VNWncBDr88UlYtmgldPxTJ6hSuSqMGjHWBp8mja8EHLUJfEQi/yph8LmqaTPrdTzw+ftL3aw6A7/42oTPXrjil1fAH9s/AUsXrRD4iERprsTA5xjAxY0usfKWLFweM3z+9uwLVv3PPwtO1x5/tD289eY71tktgY9IlMaqV68+vP6PN83OXmAB4Zabb40bPvger3Zev3YTrF25AYr2HLHuA8P1mgH9v7LBB7eBr5eakNqycbsFPtwmTrtwoZkWobs+/xLkNDwHJo7LDYdPcQncdmsr6Nzpz7B8ySqBj0jkR21av80Eyi7rlPjdbe6GHv982zoFHit87v/9g/DgAw+VXgCIxsXidas3Wms30wIzYee2PaXwQbC81aMn3NP2XutMFZ7ZcrveB/cLt4+v+5gjIwQd7h+eTUNYYT4CS+AjEvlQCAoEwvbNu2HKpOmweH5ezNf5lBwHmDF1Nvw4bXYYNBAmeGHhnJnzrfrbt+yG88+/wKqH+TjimTp5hgUPDhzVwRHQIQtQP/04D9547U0ImPu4zxxl4Wn9PTuL5Dofkciv4tBAxwyf0FXKbiMXTEdI7DDB1vHJp+GO394Zth1ePpKxfOGOA471BD4ikQ/FOzIagZOoe7vIa/LXQ37eGlt6IizwEYl8KN6R0TgdwsXcl158xXo94vsfgne1L10Ne8wp0BN/+JOVd2j/EVvdZFjgIxL5ULwjk+k2B3xtf5jYAdvNncm0wEck8qF4Ryar6zJ2+LifoUqGBT4ikQ/FO7KTvx88DKpUqWJduSzPcBaJRAkR78hOxpHOgT2HU2q0o1rgIxKJRCKRSCQSiUQikUhUsco1jMfQPF0kEom0CaETMAxATzKMx3m+SCQSaVHAMEoCIfigcwVAIpFIp0IjnjDwkAVAIpFIi9SplpsFQCKRKKGKNOLhFgCJRCItynWBEBiGXI0sEon0SeAjEomSIoGPSCRKigQ+IpEoKRL4iESipEjgIxKJkiKBj0gkSooEPiKRKCkS+IhEoqRI4CMSiUQiUSqrQYO6IBanm81DuzI/1kUpJv6licXpYEPgk/riX5pYnA42BD6pL/ULg4AhFvvWAh+fSeAjThcLfHwmgY84XSzw8ZkEPuJ0scDHZxL4iNPFAh+fSeAjThcLfHwmgY84XSzw8ZkEPuJ0scDHZxL4iNPFAh+fSeAjThcLfHwmgY84XSzw8ZkEPuJ0scDHZ8oE+Gz+zoBVXxmw7ht7nhf3ed6AKy+0pyfb119hwKz/2NPT3QIfnyld4XNskoEHoKMHdbOXL491wWftIANqnWrAtqHB97gd/hnQWdUMODA2vO6h8cG8NtcG3/d1qUvG/X/29wZ0e9SAklz7vvjJAh+fKV3hc33jYOe6/BcGFIU6KALp1qtSHz4Ig0qGCYPQe9rOgs+DI7gRbxrQ+e4yeBwYE15fHeHhZ8Y66OUDgnXoPY0GEVgIMr+PlgQ+PlM6wodGCjw90dYFHwTP16+UvaftcMhYZSsFP2ssoxYaFRHUVOfUCQKIp/vJAh+fKd3gM+3jYAeLdXSDUxwsr9qpE459xz5d+aiLHT48HsZSQYK+oYl9m+ooB6dMKkwiwYemaDRtoukmvudlI8EH2w1B5rQNv1jg4zOlG3xo1LNtmD3PybVOCU7Nhr4RnIbQdO1rBV7UMdVyWAatwodAgGnqNEeFGcGpR4eyMjiFIiBY6y+PhQMiEnwQNq1blAGrvPDBPBz9+HnqJfDxmdINPndeE+xgtM4TyQgVp1EOjZ4IQAgeXo46vQofp3IIDkzHxWOCGB8Jkd0AEAk+aAQW5ZcXPhQH980tP9Ut8PGZMhk+z/zOPm1CUyd99bHge1yDcSrH13ywDi9HwEH4EIjcIMLPcvHtONVTIZgI+Pj5rJfAx2dKN/jQtGt2L3ueaqvTXlN2SprnYQzMU1/zcip81FP711xm96xeQUhgvlvnLg98qA6NpgQ+Ah/fKN3gg50PRz98+uNk7NQ1qtvTqZPStAtf83KO0y6Hcnx7NArieejyTLv42a7ywoc+j0y7RBWmdIMPmtZsln5hz1NNi788naBEkKAOzuvS4jKlOZVzqoMLyjwPrQJATXeDz8ddgttT96G88HEDn58s8PGZ0hE+aJp+kXHqQ6/V0/A49cI0HCk93qqsjNpBCRqY3qihAb+6MNjh+al2tVyHO4Om2FSGAIBp111hwOmnhG8PwclBwz8LGWNwkJQXPnglNMLTr1MutMDHZ0pX+KjG09nje0ZehMb7vxb2safzMlM+jByHPPqt4Kl0nk7GPLdYCAGnNSadRmhGmjL6wQIfnykT4OM3V/TVxjRa4tM9v1ng4zMJfFLTdLEhT9dhhJ3bIrifLPDxmQQ+qetEPwLEzZGmh36ywMdnEviI08UCH59J4CNOFwt8fCaBjzhdLPDxmQQ+4nSxwMdnEviI08UCH59J4CNOFwt8fCb1CxOL08WGwCf1xb80sTgdbAh8Ul/8SxOL08GGwEckEolEIpFI5D/xMwterDO2ZcOohHFLAkaJLc+LIRhX5CyzvYttbebFIFMnkSnbgeHBOmNb9jl8SnL17rct36NL91vgI9Ih24HhwTpjWxb4hFvgI/KzbAeGB+uMbVngE26Bj8jPsh0YHqwztmWBT7gFPiI/y3ZgeLDO2JYFPuEW+Ij8LNuB4cE6Y1sW+IRb4CPys2wHhgfrjG1Z4BNugY/Iz7IdGB6sM7ZlgU+4BT4iP8t2YHiwztiWBT7hFviI/CzbgeHBOmNbFviEW+Aj8rNsB4YH64xtWeATboGPyM+yHRgerDO2ZYFPuAU+Ij/LdmB4sM7YlgU+4Rb4iPws24HhwTpjWxb4hFvgI/KzbAeGB+uMbVngE26Bj8jPsh0YHqwztmWBT7gFPiI/y3ZgeLDO2JYFPuEW+IhEolgk8AlZ4CMSiUQikUgUr/j/1RGLxWJdNtT/28UzxWKxWJcFPmKxOCl2hU9pokgkEiVIAh+RSJQUCXxEIlFSJPARiURJkcDHX2pvep7pk3hGEoXHRyrtj6rhRnD/SFVD7yvktgtRZAl84ld1I3gAR2oXyufuoxYqhzIBPueaPmD6LMPeftw1QnXcVJHw6Wn6NZ4ocpfAJz7hwTvC9Fum15u+Pzy7VNhmCAlVWN5rR013+FxruMcrMMJBEosqEj6oEkNf7LSTwCc+4a/ybtPNTL9r2AFDcoIPggPTz2Tp8Sjd4YNt6na8FRjueW6qaPgUmG7OE0XOEvjEJxq9oGj65SQn+FDHihU+txrB8mSM1zn0V+3sCES13BHDPiLLDeWpVoVA5fntwkoExcvhvuBfDp9doXR1n0gEAIIxmvYXXw8NveYqMOz7jSMNvt/qVCwafPA9llH3D9uT7/8DSj7FVPefhCM3bJNo00GRYWOMwCeKeOfAzsg7OgrLEXyyjTKQLC4tEVk4tcPyaqemKYkKHyqnSk2jaSKuoVxaWqJMlM+BRp2UPoNbOT5NonJqG/E0is0hg+kIARxVOqnAsH9WDj0EvAr9aPDB1075fP/VOhST7z+KyqqwErlI4BO7aIShwoZ3NBId1KpxsZl3FjdhefXXGOUEAKdy6vSO4OAESBTlO3V42m+UWznqbLQ/buVwnxCAKKqDC8qqcCQZL3y41O2gIsGH2ukRJR/3H/eBT51whEX7SzH5/pOwrMAnBgl8Yhd2fBy5tFL8pBE8ELuXFbOEafQL/GbofffS3KCcpkvU8fLine truncated
-
-[image6]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAdAAAAHUCAYAAACKzHkWAACAAElEQVR4XuydB1wUW5q3Z77dmdndibuzOzM36DXca85iwAiIZDEAKqJiwpxzzllERUUlGlCM16yIIkGSmHPOCkoGQQWhn6+qoBXLcFFE6dvn7+//6+pTp9Ir3U+/p8459ZvfCL0hhIRKWNnZ2bx8+VJZzs3N5cWLF8p7uTwvL08pf/78ORqNRlmfnp5OZmam8l5ISOhtyZ8X+fMjS/4Mye+1r4XryGXy5ygjI0Ox9vP2qVLzQ++lDpCQkJCQUOmWDEctLGVAZmVlKa8yVOUfqNrXzy01P/Re6gAJCRVX8fHxDBgw4JX79++Pi4uLstyvXz/69OmjlMmW3/ft2/eN+sLCuugRI0aQmpqq/jiUiLTZpjYTlQEqt97Ikl+1y59ban7ovdQB0kXJf0zyH5Hc7Kdrfvr06aummF+LLl++zG9/+1v5w8Zv/+23b/n/vaNMWFhXLf89y3/rf/jDH7h//77641BikjNO+btP+/2nvU1SklLzQ++lDpAuKjQ0lI4dO9K5c2ecnJx0yl27dlXO+9ekK1euKAB13biAiznnOJ15Qlj4V+urXGLEvGH8+c9/5sGDB+qPQ4lJBqicgebk5CjNtV+iz4CaH3ovdYB0UcHBwdjY2CjNJ+oMr7T71q1bdOjQQX1JOi0tQGd5Ticq4RhhD48KC/9qHZsazcAp/b84QGXJ2eeXyDy1UvND76UOkC5KC9CSavcvSd25cwc7Ozt1sU5LAFRYn/w1AfqlpeaH3ksdIF2UFqBf8pfY59Lt27cFQIWFddgCoHosdYB0UQKgpUsCoML6ZAFQPZY6QLooAdDSpXcC9NFRjidHEpMkXBxHJ0YQ+iD4rS9x4a9nAVA9ljpAuigB0NKldwH0yK1DjJg9nDELRgt/oscuHCPFdIYCUfWXuPDXswCoHksdIF2UAGjp0rsAeujGQarWrEblimWoWa2C8Ee6Xq2f+Nc//k8BaYyUyau/xIW/ngVA9VjqAOmiPidAk0KXMNRtL68mwdLksX1qJ2ytLTA3N8fcypmDd9LIS7nF9c1jWRKRUXjzj5Y+AbRilSrMneLA7TPzuXl6nnARffvCQk4cnUK92j8xZr4AaGmzAKgeSx0gXdTnA2gq8xpWpYbLWM4nFky6nJfNINNvWHjgLklJSSRHe1Cr6zye3TvBqWnVaLs+4c1dfKT0CaAVqlTGdZajFGYvSPYULqqz/Lh9bj61qlcUAC2FFgDVY6kDpIv6bAC9tgCjgQcInt+Pqfsv5pdpshlsVRH/ywV1EnZT3nAgcfdPcGJ2Xez8E19t/inSN4AumumIRoJCbuIa4SKaDB8lExUALZ0WANVjqQOki/pcAA0d1pZF16SFuI00GhVQUKphYvPfU7ZKHerXr0/tCoZM33mWvIcniJlcUwD0HRIA/bwWAC3dFgDVY6kDpIv6PAB9zhKL+rTuP46xg2z4t6qjySxYM87oTxhaOdGjRw96uszlRhq8fBBLxIQaAqDvkADo57UAaOm2AKgeSx0gXdTnAOiz85uoZupIWFg44cei8OpanaGhGuKfahhsXYG1Z9+cqDn73nEiJgqAvkslDVBSvCR7okl6XaYpKMt7R33ZecleSn3tq1KW9P76Rbenck/3ffspfLxPtQBo6bYAqB5LHSBdVPEBmse+6daYj96NFpPJO50oY7qMVZEPGWD8f7gfS39ji+w7UYSNKoel35M3yj9WAqAfYQl4pMxniFFFatp0ZWP4YglOEjhfrGShWQVqm9lwN2WD9GvIR4KafAypfoavtLyGwJkmTN68jCPjqjLxgC+axJVsXuTIweilkLUWnkr1nkuvadL+Mvykfa5VgJybIME51Sd/XZZUJ2l1wfnIx10PcZMwr96UMKkeyd7528p1n8n78GL/gDqM3bYcTdqaTwapAGjptgCoHksdIF1U8QH6krP7fybibsrrouzb+PsFsCM2gaj967kYn/N6naTcjMfERQdw8Fr+U+E/Vb9GgN64caMEATqXYa1McRnUmJ6+EyQw+cGpfpj3bUU/q3Y8SliE18whHApfhCZhOfu8B3D+6gqOLLRiyvTuOLUsi2FvZ+Zvm8/pXSM4d8WNkOWd8VjsTL/e1uw+PJsL3m1x7NSW4FMSXJ96kxY9mvE9jRg+xZm4RAnQqTIoXVk6xJypY81pVNOUiBQJsremcCOgA10djRkyrBtxEmAPD63PxB3LJTALgP5aLQCqx1IHSBdVfIB+PX1tgEZFRSnPU/1cDg8PZ926dfIHjakrJ5cAQOcw2NgKD3czWjo5kpjlzfHZ7Vm03YWhZm15Ej8R2zo2zPGfDgluTO7Zit0xywieZ8Y0vzlsGlSVIf5LeXB9Ca4DWuMfPouxBj8wZMUM7qyz4O/1m2M9YxSn17THamBvsm4toK9pTdbtm8XOsY3oMGUEEVJmu2tQQ2ymjOXBvu40LGvAqcduuHczwqKNA9ceeRLQryoDNi4ickwDAdBfuQVA9VjqAOmiBEA/XXXq1GH48OGMGjXqs3n06NFYmFkwfM4wohM/P0AHtTDj5+hheHRrRMAJN6YOsef4yeEMMLWVADoJu4a2LNg0AxLdmNHPnL3HJYDObc30ADcOjq3BhL1evHjkjttgczZJAJ1pXZNNZzagOdeXzrY92H9zBZlH+uIwSso4Tw6jdydbHjxZT3rsCOyc7Ri3fhoTLVsRdN8D0ucysLmZlIFKWentedzbNwjf1QOZ37MOvX3nETVWAPTXbgFQPZY6QLooAdBPVyuTVpTEk+zX+a1jwOR+JQTQ1kTfXEBqYDsshndj9BAH7t4YT/9XAG2H65aZEOfKGMfm7It1/yBAF7Srzf5rW8g97YKdTTd+vrycp4ddsB/pTLwE5j5agB4fTofuEkA3TGOCRSsO3csH6KDm5sQ8dGV+t6aYmVmxedtoVgyo+8UBGh4X8laZcMlbAFSPpQ6QLuqjAZp8Etf5c5k7dy4+O6O58mrevqIr4WwQ2zcGka5JZe+2fdzPLOKxVfraADVtbcqDU0GsmJsfD9lu7h4cPH2fW+H++PwcTHrBpEyy0q8E4rk2kPTn+dDNOn+QhfPyt1vitQ/tneI1q9YoXyolBdATcW5w2YUKteszakI/Eh9NUACaEDcB0xotGL5aAmjCfBwb1eHAqeWvALpvdHVmBq+XYLaCxYPyATq/bW32Xt38ToCmnJ9Cd6umXH2yibjtDhh1cWRp5BJ8OzfHLXIVxE+mfa1mRF+fSddWjWk0aJJ09Ts5NKgevdbNJ/oLAVRuKs9/Yot4WsuXtgCoHksdIF3URwH0xgbqGdRk0ro97NmziwltTajWcgZx6nq/oFOzzKnwp9Yc1zzj8rnLpGYXosxH6EMAffbsmbros6uVqSlPbpzh8B45HnvYe+AQQ1r+D108b7LH5U/8ppw1gTde/8LY6vT/+M333bmWlMuj42uoXMmGTT/nb+vZrxF1Bi4jVarnWWIAnY1LQ2MiUj3JfuGK44/fM2WzG5rHY+jZxIK7GX5sG1gF40aVsbNqhpmUKe4/sYygmcaM3+DKmTm1+d7WgZFe01k5xJT1oTOZZVGdXVekDPRUb2xad2HbJQmgh6TlIV3Jlo57YEZrjBpXp0XzOvhHrObp07VknexLk5o1adOzGVY/mnA8dTmbhjbHosyPNG5hyqLRTbGfMYHDo+oxZmsJ9cJ9dJSI+DBOpMbgHehJ7ca1mb92LtEJ4mktX9ICoHosdYB0UUUHaDaetpXovOX1/LWZ57exc0hbdibB4cGGlKlUixZ9l3D16laa/LMyVSpXpUyF+niEPCYvL5WVg834oVJVrJsZ8be/dORa3i16tOnJsYQXnPWfSJ3vK/FjxXK4n0gmowhMVQM0JSVF6cXq6OjI//7v/5ZI82phmZiY8DRTO2WElJxv6kkV65XIfYsPjCxPjTZjcPeP5lyuVJAdSpfydjQd3p/Lj16wpHsdZuwr9IXxIo5B5ob4nkzEa3UJALTAeYW2l/eVpx3XWVAul2lhpayX18ljPuVxoIXGfmqXC5dpVMvafSjHS5L3+3qbwsfQHl8pS8hfVs6r4Ljqa/gYqwEqf2HLEN17cRdjFo6ikXFDpdPWv77/lxJr0ZT7ZS0AqsdSB0gXVXSAnse2ohVHlHbGNDZPGMGAYWMZM3sZUfEaDvc3oM7c/ElvT65zJzI7f6u4vRPpNHobT66tpm7t4TyVyoLGNaPh/1pyJe+2BNBehN26Ri+Diux7ApqTrlRq5MzAQ2naA79XMkC7dOmiLK9atYoGDRrw7//+78oX4o4dO1S1P79MpQz0aUY+QNNPbKBpo04cLxjyumd4JZwmzMdqiBsjj+WRsc0O28WH6e/izPn7DxloWpvttwrtjBy2jnJgxIYzeHt6lhhA9c2FATrRbQKhD47Qb6ILZcp/r/ydaL1400IuZJ/lVMZxybHCX8iX8y4wfM5Q/vKXv5Ce/uZ48V+b3qSHkJ4B9CoOlRsRoMx9kMOjK5eICZiLm93/MjRSQ8iAegzcW5CNnVtJjaqVqVypEuW++z86uYVyP6A/9R39kI9ybdcM2lfqLgFUzkD7cVjKWG2qWXH2mQTQ1Bt0tbfEbPWd14d+j+7evav0hJXn2ZX/O7SWP4z29vZ0796dbt26vdfy+o+1s7Oz4l69elG9enWyNXmk7B7EQGcpEz//OhvdPeRHxu85RR/HmfRfuJtFFsasO3mZrt2duPAgjfF2NdlyvaBynpwpZ+PV14IRARcEQD+jtQCtXeNHuvRz5H/+8T9v/K3I/v1//B4zu9ZYOJhJNhf+grZ2tKJK7cr87ne/w8HBQZnyU/sZ+1RrP6elLaMthA4hWeoA6aKKDlCIHGtIpYFb0d5dTAycxhSzPzMkIh+gQw/kSKUJ9P/ub7jdz69zcmVXnKbsI+GCH63Npynbxiy0o9m37QoA2oLine truncated
-
-[image7]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQwAAAEbCAYAAAAvX7qwAAA00ElEQVR4Xu2dCXgUVbbHC5AgoIgITkIIwYERFB0RFxj1jeuMy3MBlTcgIouKKIIL7jMjqOMyioyCDqCCAgqIIIiyCzQgAsq+LwJhFRDClrAn59Wp7pPcPlXVqXR1JVWd8/u+/5fue2+dqq6u8+97by3RNEHwmH27D0IQpVOOfxZBEDyGJ2JQJIYhCKUAT0Su7D2H4MDeHNj/22FTXWlKDEMQSgGeiKrQLDZv2ApzZ82DxT8tgwO/5ZjalJbEMAShFOCJqOrA3lwITZ8DXbs8Dm+98Q4cOXjC1Ka0JIYhCKUAT0RVB/cdgTGjxsFlTS+Hdm3bw/GcPFOb0pIYhiCUAjwRScdyTkGu3qP4+qtv4IrLr4T27TrCidx8yDlwDI4ePgl7dx0wLVOSEsMQhFKAJyIawq/bfoOrr7oGxo7+Fr4bN9EwjA4PdILfdJN49ZXX4c7b74ITR/JNSWwlnAc5lH3EVO5WYhiCUArwRMw9eBx2bt0DVaueAYM+HgITvplkGEbH9p1g94690PnhLlC/fgPIPwGmJLYS9kT27Mw2lbuVGIYglAI8ERNpGAf35cKUid/DZZddYbw/lH3U1KYo4fAH4/ByMQxBKAWOHT5lDBl27dgH3+nmsH/PYUeGkacbxsRvp8DkCdOMYQclMl6vsXTRSuPsypaNO+CrkV+DvprI9Ry5xjpwyMMNgITtjurbhGdk8PqPlnfdDaNGjIHD+48ZZnZ4/1GjjRiGIJQCbe9rB8OGjIANazcbib12xQbYtX1vkYZhZGy5csYymNjYC8C/aEBdH30cMjPrwZBPPzcmTbENJvz2LbuM1yuWrDHaolHRBWH4F7Vx3RZ44P728GyPF+DUsTyoeFpFePvN3pB3Kh9efP7v8MH7/WHT+q1iGIJQGqRUTIEX9ETctH6LkczrVv5SbMPAX340hmmTZ8Bvv+6HJ7s/Deef3xC+GDYiyjB2bNltvF69fD3M++Fn+GTgYN08Vhtmgz0H1Mqla6Hy6ZXhkj9eApAHUKVyFej97z6Qn58PV17RDB5o18FoI4YhCKUDPPvMC3EbBgr04Qleq3Ff67aGGTz1RA9o2LCRbhgjLQ1j/epN8PcX/2ksO1TvhYwbMx4ubXIpdH6oi7EdFcqfBhdecKHJMC5t0hSwRySGIQilR0IMA9u0bXO/Y8P4x0svG8sO+2w4jB093uhRPNTpYWO4IYYhCP4FntCHEG4M40TuKXj6qWeMxMaJzqIMY62+jqGffQF33dkCpk2aYdyvMnXSdFjw4yJYtWyd0SazbiZAfrRhNNSHOffc3cqYAxHDEIRSID29Drz26huwcV1WOJnjmPTES8i3bNoB2zb/CrkHjkO3rt3h97+vD0OHfGFpGCuXroFft+0xTArXhZOdOAmKWrNivW4W9eCG626A/JP5hYaRlw+33HyrHvsJvc0GMQxBKA0wiXfv2Gf0DO5ucbeR9Hja04lhtP7bfXBfm7bGaU4STlwOGTwMnu3xPMwJzTPuRdFXY5gBToje0/JeY9iBJqGejiXhhV54WvdX3UjwqlM0pTdff9s4+4LmgtuKbcQwBKEUUK9/OHLohCEn12HghVt4dgTFkx7PeuToPYocvbcxauQY0PSkxyTHthjfyijsdCwnzzj9yu9dEcMQhFKAJyhd6Vmp0unwyUefmQzjoQc7Q7165zm60hOFvYL1qzcar4tjFEVJDEMQSgGeiGQY5cqVh4H9B0UZxq7tv0EH/W/t2ukAJ50ZBg5Fjuc6u1GtOBLDEIRSgCciCec09up/6fb2Du06GvMIO7fuhq2bdia0txCPxDAEoRTgiUgybkvfdzTqeRj4AB2a3OTtS1piGIJQCvBEVIWnS7+WJ24JgkDwRFSF8w/zf/gJev7zFRjw34+N05y8TWlJDEMQBEEQBEEQBEEQhEQxU9M68DJBEAQTIU3L1wUoViUIglBISDELUlQDQRAEBIchIWYWKL28Y1RDQRDKNiGLngVXQWNBEMouIQdmQYosIgiCYJiHySR0l5ArMwVBMBMSwxAEwSkhMQxBEJwSEsMQBMEpITEMQRCcEhLDEATBKSExDEEQnBISwxAEwSkhMQxBEARBsCEjozaIRH6XfqiW58euUArwL0Yk8qM0MQx/wL8YkciP0sQw/IH6pUBIE4l8o8y6Yhi+QwxD5FeJYfgQMQyRXyWG4UPEMER+lRiGDxHDEPlVYhg+RAxD5FeJYfgQMQyRXyWG4UPEMER+lRiGDxHDEPlVYhg+RAxD5FeJYfgQMQyRXyWG4UOSzTDWDDX+gVCUFvQ3tyuuBj2nQdPzzeVuNKefBuXKaZAzKRxf3earLtKgzY3mZVA1qoW3BZcbzJbjmtpbg7uu0SB/pjmO3yWG4UOSyTC63W1OGNSIl81tiysvDAMTv+rp4dfcMEgpFXVjmBy9XGaqBi0iJlCUYaCpoCm93N68fr9LDMOHJIth3HFVOEGGvmSuS4QSbRjYu6BeAr6n+PSe1K97YeLzGFxHp4bb5rNyNBU0jaD1MsQwfEgyGAYlCi+30/Q+4S4/6q3O5nrUzjHh+msvCb+3MoyT0zW4+YrYcbDXQ3E2jywsx97F7rGF7+0MA9Xjb+Gexvz/ht8PeFqDsf8yt7MzDIoRtF6GGIYPSQbDwGSjrn1RqnGmBhf/XoMNX4R1XROz2eCvMZZhUmIbfI1SDWPTiHDZkkHhNndeFU5qXv9m53A9zq307FCYzOW06MSOZRhoTBgLhyH4/s6rw8nPewyxDAPr1O0LgsQwfEgyGMbT/2f+9bcSJZ46J8DL6L0ajyYnqQzbYNJSAqvLURJjvZ2JYVuevE4Mg9Yfj2EUbJ9FnV8lhuFDksEwcP7CiWHYGQtOIuKvP74uSFw20agOSXr8X8RkWHKjqdAwg85+8HWhsPfBzcRrw0AFbR5DDMOHJINhYLLxX2wuo1dwVXSvgETlmGjqa74ONWH1XVcwD6Jqfn9zb4OruIZBw5vh/yxcvxiGUCokg2FQghZ1vYXdXAf2MIZHTr1iLwTbqBOSNAQhw8A4OAehtuFSextcVvMJsQyDTouqw53iGoYMSYSEkAyGgdI/ipHoKz8z15Hol5qXq8mP8xXYhgyElqOLpagNJrHahgvbv9zBXI6i5FXL7AzD6rRqPIZhZVJ+lxiGD0kWw0DpH6dAODyg1+qFWzjkwDK8irLjreHXnW6LjoPJjuWN6mpw2fnhRO7bPXr+g8wHk7DzHeF2akJS8mIZnlI9+8ywgRScJWFzHGgY6vaTcHluAPEYBl6LgVd88nI/SwzDhySTYZDwGgq8JDqXTVyqmvW+BksHmct5m1gxUHjKdOK/7dthvVUcNAi6WpMvk2hRD8luiORXiWH4kGQ0jKAIexlWk7CJlnpKOEgSw/AhYhilJ7z6kl/AlWjhMIVP4gZFYhg+RAxD5FeJYfgQMQyRXyWG4UPEMER+lRiGDxHDEPlVYhg+RAxD5FeJYfgQMQyRXyWG4UNUwxCJ/CpNDMMf8C9GJPKjNDEMf8C/GJHIj9LEMARBEIRSJT+k5fPJs3jFYwuC4ACeSG7kdVwxDG/g+yZugQwtkh7Tl+5CXscNkmHw9cUt0Mp5ErdnYXKb6uKVGEbyY/rSXcjruGIYWuLiimEI8WD60l3I67hiGFri4ophCPFg+tJdyOu4Yhha4uKKYQjxYPrSXcjruGIYWuLiimEI8WD60l3I67hiGFri4ophCPFg+tJdyOu4Yhha4uKKYQjxYPrSXcjruGIYWuLiimEI8WD60l3I67hiGFri4ophCPFg+tJdyOu4Yhha4uKKYQjxYPrSXcjruGIYWuLiimEI8WD60l3I67hiGFri4ophCPFg+tJdyOu4Yhha4uKKYQjxYPrSXcjruGIYWuLiimEI8WD60l3I67hiGFri4ophCPFg+tJdyOu4Yhha4uKKYQiCIAiCIAiCIJjgjxQXiUQiVZr6bwZ4pUgkEqmyNYyCQkEQyjS6H+SJYQiC4AgxDEEQHCOGIQiCY8QwiuZdXcd5oQLuHxK2GxpdXWw6aeFYVXhFKZDI7bhG1yIten9xYX1VWsCC07RwO/4+6qpKl+TrGssLhTBiGLE5XdcWLfog5fCDHjUoqkXxSFbD2KfrXM28r1T5wTB6a2HTSGTMpEEMIzbjtfABiabRmtURPKnQZHhZcUhGw6DeBSdLizaAoigJw0DwO5BehgViGLGhXz084O2GJVZJhWVXsjI70nTNjaiFZm8YXbRwm+ct6pAbtMI42EYFE2twpC6kWS+P20HrwO1AYm0HSq3DdQyMvKZtQc7TCnsXnCzNbBgYZ7IWvR1qXVGG8Y6uD7TCnooaj+8X/Cy0jkm6mil10suwQAzDHjSJHF21Iu+tkgfh5ZggTvcZrgPb4kHbQNcbkfdqTIyXrYWNC9tQr0c1JKxHQ8N6VF+ljraH1oHJjO/VHhNtx/JIG6vtQNTtaBSpp+2g5MU2IS2cjAjOAdntjywtug7NEj/HJVp4HbQNRFGGgcMJ1djpszfRwvG+YfX4frgW3uZWum5X6rK0aAMRNDGMWOCBrnZL8aCyGpaoSYW/0tgOE88JmPy8q84TlQyCwCTBMto2SprGBS0KobZ8HVSmrkN9j5CJUBlPVsRqO3hXHtvY9c6ytOiYf9ait4HMRu0t8H1BhvF65HUbpR4NQd0ebjBYb7XNCNapsQRNDCMW+Lmpd4Hgr5/a4yCwnap20dXGQXorE4Htea+FD0nwNU946v0g2J7XE5RwfB00z9Ir8t6qDaKWYyy+HnU7eDITOByhNpwszXoZBHswX2nhehrO8HXQezK3XkodgmV8EhWHGhSPDMVq6IG9lVd4YVlHDMMa6spaiSeNmlT0Xh0u2BkGP/gJ1TCoDYrmDUihSL36K8/hvRNC7Q3YbQeibgfFstoOxC5OcQyDfvGxR/Line truncated
-
-[image8]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVwAAAGJCAYAAADG5YuwAACAAElEQVR4XuydBXgV19aG23tv7//f+u3f21svFHd3d5fgkgRCcHeXIIFAkKDB3d1dgwa3lhYo7m4tUOT797fCnJ6chJaE5JCTs77nWU8ye/bImVnzztpr75l5CyqVSqVyit5yLFCpVCpV7EiBq1KpVE6SAlelUqmcJAWuSqVSOUkKXJVKpXKSFLgqlUrlJClwVSqVyklS4KpUKpWTpMBVqVQqJ0mBq1KpVE6SAlelUqmcJAWuSqVSOUkKXJVKpXKSFLgqlUrlJClwVSqVyklS4KpUKpWTpMBVqVQqJ0mBq1KpVE6SAlelUqmcJAWuSqVSOUkKXJVKpXKSFLgqlUrlJClwVSqVyklS4KpUKpWTpMB1Ez1//hyzZ89G+vTp8d133yFlypSoWrUqLl686Fg1zmjJkiVInDix7G/r1q3lN/yZqlWrJnXtrXTp0o7VIojHJUOGDFI/VapUmD9/Pq5du+ZYLdaUMWNGpE2bFvfv33ecpYpnUuC6iaZOnYoECRLgrbfestk//vEP5MqVCzdu3HCs/sb1448/IlGiRLZ9/fDDDzFy5EjHauH07bffhvt9tKRJkzpWC6d58+ZFOC4JEyZEvnz58PDhQ8fqsaJ//vOf+Nvf/oY7d+44zlLFMylw3UATJkywwaRGjRro0qULatasKREdy/7973/j3r17jou9MS1YsABJkiSRffP29kabNm3k5kAonTlzxrG66ObNmwIuRsT8fZb9GaQZQVvHpVKlSlK/Vq1atrKvvvoKjx8/dlwsxjVmzBgEBwc7ZVuqNysFbjzXgwcP8MknnwhA5s6dG27ekydP8P3338u8Dh06YOXKlfJ/y5Ytbc3b/v37S1n+/PltyzECZBmb+RSb5O+//76U5cmTBz///LOt+U94spwwIzj5/7vvvovy5cvj9u3btnXaq0yZMlIvderUtjJfX18p27Ztm13NP7R161aZ37x5c+zdu1fsz5rohNvXX38ty/CGZK/Vq1fLtjmvR48eUpY7d26ZXrp0qS2S/u9//4vz589j//79tmMcGBhoW8/169flRsEoe/HixfK7WYfbPXjwoK3exx9/LMfv7t27tjJV/JQCN56LMOBFzmZzZLKASKDwgv/ggw+QLl06LFq0SOYToBZc2MS+cuWKTNMOHz4s0Zk1/fe//13+/u///q9EihTXwzJGp/Z1aFYdRzHFce7cOdmWJQKf6yDMI1NQUFC47dAYuR86dMixqujEiRNS54svvnCcJdq4caPMZ36VKlCggG29b7/9tu13MNVhRd/WfOumwN9BkFrz7Otwu7///rvU+89//iP7qsCN/1LgxnMxIuMFXrhwYcdZoj179sj8ZMmSyXSOHDkEDJ6ennj06JHAwILF8ePHbemJzz77TKJERsisY0W0zIlyPoHEnKQF3G+++caWoyTQWcZtvYp69uwp9QsWLPjSjrPixYtLHUbdY8eORf369WWa0Xhk2rx5s8zPnj274yzRsWPHbMs/ffrUBlwrqqesqHbYsGEy3aRJE5m2ImYLuCzj9ij+Ft68mP5gBEwpcN1HCtx4ru3bt8sFz86xyLRz506Zb6UMrIiVAN63b5/8X7duXfnLPCNHAjCiq1y5sqQk3nvvPZkXmbHZbAGXALfk4eEhQH7ZPtmrVatWsjwjTSsifFVZnWGMlh1l/TaO2ohMR44csR0HQt4CLlMVltKkSYN33nkHFy5ckOk+ffpInUmTJsm0BVymDCwtXLhQ0hX/8z//Y+usVOC6jxS48VzMMRIKbAZbOVNGYhMnTpSOJquTaOjQoTKPEGBdRrT16tWTeadOnRJwMO3AiI+Q2rBhgwCQwGWUy0432m+//SbljH6fPXtmA26jRo1s+8T87asAl5DmsnXq1JF1vUycR4hzP7h9S1YkffLkSbvaYeLvZE6Vv5XHyF6MxFu0aCHLcr2UBdyjR4/a6hGcPLYW0CMDLo+PfdqCKRwFrvtKgRvPxeisUKFCAgJe6JcuXZJOLk5b5ti0z5Ytm20eYUBVr17dVtaxY0dbj3qKFCkEztOmTZNpNueLFi2KEiVKSEoiOsC9deuWdLJxOXbavYoyZ84s9fv27SvTjNwZiX/66acvhXXFihVlGUbCVq6XoyA4Rpkg5jxGwpQCVxUTUuC6gWbOnGmDJWHHiNG+84pDqUJDQ231vby8bPOsHKefn5+tzH60A1MMLCN4CVVrBANBx5RDdIDL/bXG4FaoUAGNGzeW5WkHDhyQOuzU4vSmTZtkmjcBC54sz5s3r0wTqi/TihUrbL+JNyUuxwclrDKORrDSGApcVUxIgesmsi50CyY09rCzM4v/sxPHGkbFIUzWOFhrmNOOHTtsy9k32wlVC240RrvdunXD1atXZT7hzHL7EQnswGMZn65yFDuoPv/883D7aW+zZs2SetZwtQEDBtiWK1myZLi6HGHxV+OLV61aJSkS++V48yhVqlS4Bx+sCNp+1IM1PMwaG9y1a1eZZh6cYqcYp5m6sMT9t5azOs2slI8++BD/pcB1IzG9wIuaTXb+tfKtgwcPRtOmTW312AT/9ddfpZ4V4XFZ5oBfFoWxnPUdx75yeZZzfZY4NpjrigyG9vsYmVn7w3QFp/nXXtZ+RAVejsclshQEfxfnE+yWuC3+Dqs+Ac06VrrFOmb2+8L9t5azRlxwvv20Kv5KgatSqVROkgJXpVKpnCQFrkqlUjlJClyVSqVykhS4KpVK5SQpcFUqlcpJUuCqVCqVk6TAValUKidJgatSqVROkgJXpVKpnCQFrkqlUjlJClyVSqVykhS4KpVK5SQpcFUqlcpJUuCqVCqVk6TAValUKidJgatSqVROkgLXiJ+DuXbtWpw061M1KpXK9aXANfrXv/6Fd/7xT/zj7/+IO/a3f+Cdv78jHyFUqVTxQwpco0/+/QmWHVmMvfdC44wd+HUPNp9bj/f+V4EbX8QPbNauXdtl7JNPPnH8CarXlAIXYcCds3MmtlzYGGds6+XNchN4/1/vO+6uykXFT7+7kpImTepYpHpNKXChwFU5RwpclQIXClyVc6TAVSlwocBVOUcKXJUCF+GBu+vmdoTe3IHQWzuw68a2CCB0lilw458UuCoFLv4AbsilTchdKBcy58iEDFkzoGTlkthxfauUb720OQyE5q/1v9jFsLKQi5sinW8/zTr2da35XL/Mt9+OAjfeSYGrUuAiPHDzl8iH/Q9244enh9BnbC8MnDEA7fu3hVeTmth2dQtqNfeGb2sfiYQJxfl75qK8ZzlMWjcB265sQZ1WPmjQoR52394p8PRtUwc1GlbH9qshmL19BsrXLIvBMwOx49pWAatn4xroMqSTLNt5cCczXRMbzqzFtstbIgXumDFjsHTp0nBlKtfQXwH38OHDSJEiBVq1aoXOnTsjffr0ePDggWO119a9e/fQtm1bx+IIUuDGvBS4CA9cRrf9Jvqj1xg/lKlWGnNDZ5toN73As7V/SwTNGSwgbtW7JTadW48iZQtj373daBvQBu0D2mLapskYNCMQ7cz0kNmDMGrJSCw9vAhjVwQjb9Hc+OHZYfQZ10tAXaVOZcwImQafFrUQbOplz5sN60+tCRfhvvcCuIsXL5Zxke+++67D3qtcRX8F3KNHj6Jdu3a26caNG+PHH3/EtGnTkDlzZlSsWFHKL168iGTJksHT0xN169bFpEmTsGnTJuzYsUNA/fjxY9SsWRNZs2bFyJEjcfPmTWTKlAnp0qXDDz/8gJ49e+LLL7/E9evXbduKTArcmJcCF+GBm6tgTizYO1ds5Y/LsOn8BhQuUwg7r2+DdxNPgeea4yslch27YjQqeHlItLrv/m54mig4VfrUSJc5nUS6y39YigIl8iN3odySRhi1eASSp0oGv5HdMXrZKOQpkhvrTq7G8AVD0SWoE0pXLhkupbDcAPfL/34poH3rrbfEeNE2atRILsYmTZqgefPmaNmyJdq0McBv3x4dO3ZEly5d0L17d/To0QO9evWCv78/AgICEBgYiEGDBiEoKAjDhw/HiBEjEBwcLFHz+PHj5cKdOnUqpk+fjpkzZ2L27NmYN28eFi5cKMBftmwZVq5ciTVr1mDt2rXYsGGDXOghISHYvn07du7cid27d2Pfvn3Yv38/Dh06JBAhNH7++WecPHkSp0+fxrlz53D+/HlcunQJV65cwY0bNwQK/Buf7fvvv3d0vXA6duwYPvroIyRMmBBfffWVnFsqQ4YMEpXyYQQe4759++LgwYO4fPkyGjZsKOeL54DHvnfv3vK3WbNmuH//PgoUKCDnjOf49u3bePLkiRzrTp06OWw9ohS4MS8FLsIDN2/RPNhsIGvlWDeeDYtit18LMVFtc4xYOBQBE/uiVa8WWHNiFfIWyYODv+1Dsx5N0K5/W8zdNdtExbPQZ0xPDJ8/FGtNnWmbJqHjwPZo2bM5TuMkvJt5Yf7uOahsItzZO2bCt3UdjFgwDCUrFRfQ2ke4H77/IUJDQ5E/f34BboIECeQdC7zYaIx2Lly4IBA7e/asAO2XX34RuJ04cUJAxwuZ0CP8jhw5IiA8cOCAGOG4d+9euUi5HV7QhOe2bduwdetWbNmyBZs3b8bGjRsFsOvWrRPg0latWiUX8/LlyyXNsWTJEixatEgAvWDBAsyfPx9z584VcBPgM2bMkGhtypQpmDx5sgB+4sSJmDBhgoB+3Lhx8dq+++47R9cLJ54f3jQpRqKEKcXolseIx5jnkhEqzysBypsvgctztn79egEuz2OdOnXkJsrzRvF8MXrm+eWNztrOn0mBG/NS4MIOuCYKbd2nZTjgbjq3QXK4jDy3XwlBtXpVJe/KEQzMu07eMAnFKxTDhDXjJO/q2agGKvlUlBzuZrNs1bpVJP/LacKadRnpMirm/MqmbruAtrKujoHtbR1okXWaEYDM8RGYKtLine truncated
-
-[image9]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOsAAAEWCAYAAACHcWCmAAB5cklEQVR4XuydB3gVVfrG4/5XRXonjYQQ0nvvvffee++kQwKkQOgdpIogiPQmIBBAJXRUFERQV1111+7aV1d33fX9f2duJlwmNw0IIXDe53lz751y5szM+c33nZlzb5SUuLgUSH3cSAz4sxIe//MjUBmrifEq+mQjTFCzwCRNW+hq2cFgkiNM9F1hZugBC2Nv2JgEwM40CA7moXCyjICzVSTcbGLgYZcAT/tEeDsmw9c5FX4u6Qhwy0QgOcg9CyFeOeRchPkUINy3kFyESP8SRAVMRnRgKWKDyxEXUoH40EokhlcjKWKK4OSIqUiJnCY4OaJWcFJ4zS3vE0JrEB8yVXB0UBViAqsQHVCJSL9yRPiWIdS3FCE+kxHsXYJAzyIEeBTC370Afm658HHJFuzllNnqDHg6ppHT4W6fCje7ZLjbJcHdNgGuVrFwIjtaxcDRIpqOQRQcLMjmEbA3DYetaRisTULIwbAyJhsFwtIwABYGfjDX94GpvjdM9DxgrOtOr+4wM/AAnYY/Sc8LF1c7qY8dgSce/RMGPvZ/GK86CVrqppg4noFqA72J9gKoxnoMVHeYG3pRA/RrBTWEQA2Hi3UUXG2iCdR4eDkktYHq75KBAGYGqkc2gj1zCNJ8weG+xYjwKxZAZZDGBJUJoDJIE8KqBFCTIwnQqBrBqdG15OkyR8nMwBVfZRBPR2JYLUE7BXHBU6jMagK2GlH+FYLDfUoFh5FDCNhgjyK6iBTA3zWfQL0JrLdzlmAZsBnwcGBOIWhT4ErQulnHwZXswqC1jIGDZbRgewasWSTszCJgQ8AyW5uEwtIoGBaGgbAiYC0N/GEhAOvZZgsjTw4rV/ekNno4Bg34Pwx54lFoqhlCW8MaOhPsCFQH6Gs7wUjXRYgAFkbe8HKJkIFqFkxRJUSIqK4UUd1s44SI6uOUQqCmUESVQSoDNYtAzaOIWohQ7wKCpVAAVYimAaUEZhUyE6ZRRKVoGiYXSaOYa5ERV4fJ2XOQHlNHnoG0aNF15OnC+9QoGaxJ4bVURg1iKbrmpTYgO7GOQK0iV9IFokwANZRFV69iqlchRft8BLgWEKQ58CZQ/d1zCVCKrI4M2NYIy0ywutkxJ9P+JsDZmlkGrDMB60wR1p4irL15pGBbs3CCNRTWxgzWIHIAweoPawLW1sRXiK43YfXisHJ1T2pjhmMwgTp00OOYoG5MoLLUl4HqQJHUA2UFlQjwIiAdwrBzyzZK+UIJ1hDYUwRxsAiDC0VVd7s4IarKYE2jaJVBziRnERDZCKKoysyADfMpEtLfKAKVwVpTuhDzZ66VgRtW3Zb+soiaQPBWFC7A9s2HCMgZAqyyaNoaXSNkn5mTCNLEsKkCrNEBVTj94nls23QQET7lBGs5XSgmU0SdTOlwCYFaDH+3QoqshXRByRciaVZKDW1nF5KiyimKpsHbKUuIqp5kIaraJ8PFNokgTRRAZVHV0TyGUmKClSKrp0M8XbRiCFSKrCZhAqgsHbYwCIa5QZAQVaOD0jGvfh6lwF4EqhelwF4cVq7uazz1WYcPeQyjhw2iqGoupL762o706ohdW5/F06tWYW7DbJTkV+Hci8dw4vAhikxJKMgox9kXTyI3pZIibDSlvwzUdAFUb+rvHdx7BFs27MSWp3Zidt1yHNx9BG++fhUb12zH4b0nkJM0HYf3HUdN2SLMrV+FG9fewUH6/MqZ13Hj6g2CeLEQMZfO3YRrr7+FF4+cQUHaLJw/fRmXL13Fivlb8NqFN7D72WMoypiNK6+8hS3r96Ewo0lY/rOPP0bDlBUUxauQHj8DV169imc37kU8XQzOtVygZa5i6Zz18HXJw47N+3D9yhWKsJm4ePo0VixZj2kV8/DysZNYsXA9ntu4Azkp1bQ/W/HchuewZM4KHNq1H80HDlCkjCAww3Fs3z4saVqCyIB0PL9zB3Y9s4VS4mBse3oj9j37HF0oEhAVlIrFTQsoU/EmUJkZrN4cVq7uSVNlFMYOHwDVUUMwaYKV0Ec1mOSMEL9YnD5+COtXPonnqMFtWruBGvUWrF22DhtWr0dL8zFkJpehurgRjmZRlOpm3RJRd23dj5NHTqD5hRO4cPoCDuw6is0E7xuvvY6rr72BGZVL8PabbyGM+o8H9x7Hm5evEZgbCaLrOPPSRQJzJooz5+Kl5nOUBs/HX999D9uf2SdAeuPKX+gisgtvvnYNjTWrcebkBYLsNSxbsBmvE8hZKTNx+uQ55KU1ISakChfOvIpkSqk/ePd9PLlkM9at2Ipzpy7g2Q27KeXNweXzr2L7lj0UdYvwwr4XUFU8my4kB+iicJb2eTvt/3O42PIiTp9oxqJZy3D80CHUVc/FC3v2CrDamERj61MbEeydiv3btlMfNoyWW4CmaU2IDqYLmEcc5tbNo0gdicykYoLUp9XeHFau7muC6mgojxgAtdHDoKNlC0MdJwHW1YuXY+HseZhRM4Mi4S4c2L4Dy+Ytxb5tu/HM+o0ESDNB+zSyUythbRSGDaueodQxldLLLAS65xAQW/DBX25gYdNavP7KG3h+z2FcPHMJL+xvFsBjEfcvN24g3KeMouUrePn4Gaxb/hz2bDtEEW+D0B+tKlxMcL+FmbVP4tLpVykq76LtbMPf//oRlhDYyxdsxN7tR/D6xTexZumzlLIvwI0336Z6P4WjB19EZmIDYkOn4pVzr6O8cA5t/1U8vXYH6qcsp4vDm1gyewNF0zxcOHUexw83o2nGciyb/xQBXoKP3n9LiKSXL5xHTfkcvH7hDDau3oiE0AKcOt6M3LQqvEhZhr1pFMEaiYM7dyImNBeHCeDwgDRs37QVM8rrkRabh+KscuSnT6Y+bwQBnQAzfT8yh5Wrh9JWH0ugDoSm8lgY6DhSf8pVeKwQ6hsPw0keBK8nwgNTEOafQpEhGjHB2fBwikNSdCE1zjyE++XAzyWVQKmjVwYq9VE9cnHsUDNFzjdpWib1PStRUTCPUuY6RPoXUzRegHL6XJI1W7hzW1W0kFLZJmTE1mNew3qKSKvpfR2yExtRkb8QCxqfRmZcPXKTZmH29KcoKq9Cbeky6v/RxSJhJvJSZmN+w9PIim9Aac4CSq2XE2ArkBJTT/3EqRRVZ6CBIjB7pJMeT9OCqwigWYgNrkQI9V+jg8pQmDkTb1+9RhlCPnyc81CUU4/IwBIUZs2Al1M64kInU9l1lMoW0r5PpmkpVG4J9VkTqBsQT/tYQOuxm2tJlOJTH9knnfr14UiPK6F+eA4s9APh5xaHMN8U6sP688jK1XPpaSpj/Jgh0FbXbLvzy25+mOixGyA+MDf0haVxIGyp/2VnHkYpXgScrKPhbBMLN1v2XDUJno7J8HFMRUBrVGV3fLOSa6lhFlGaW0TRs5hcgki/ycLNnpjASsQSOAzU+JAaJIWzu7nsEcwMpEezu74NyIyVc1wjvcqcQZ8zYhppuQakRjbQOnVIiahDcjh7rad+bh3BMR2xIdPpwlJLINaQp1BfsgoR/lV0cakkkMqEG06hXqUI9ihBIAEb6F6EIM8i6lsWURqfS33ZXAKPXp1zaN+y4CXcbEqHh32acMPJXbg7nARX20S42JCtE+BiFSN7nGPO7g5Hws6U+rMmYbBij3CM2V3hQFgY+guwyu4IewmPw5Q4rFzdkcEEZWiMpRRYQ1t4nmqq7yE0JNkNEB9qXH6wooZmaxbSCmsUwRoDV5s4YZCAl0My9ftYRGEDILIFWFlk9XFmAyHyCAgGa4ng6IAyMoO1Sg5WdgeXDWyQPX5Ji64XnE7OJCiZs2JnEqgzZZCS0wjUtCgCNaKBIK1HUtgMMgHLoI2YgfiwaRJYp9JFopqiOhskUUEXkzKCspSAJVg9b8Lq5yJ77hroWgA/1zz4UYT1c6J9ccyEt30mvOwJVgasQ6oArPAoxzapDVbn1mevThbRcGSDJcwiKUUOlwFrHCw8xrEwDGiFVTZAgt1kUuKwcnVHhloq0Bw3EgZaxjA1YCNqvIVoam7A7Cc0LjYax848HPYWkXC0YlE1jmCNF545skbr5cBGK2VSfzVbcJBnLkI8CwRQQ3wpqvpORqRvqTA4gY0qig2qFsxgjQtmI4/YCCQG7HQBNhYt0yLrkRHdGkUlkKZGNgpOCmOgNrSaRVf2uY5SVhmoMlhrERU4FdH+UxDl1wqrL0tTWXS9CWuQOxskUSg8d2X2c8kX7Euw+jrlwNuBgG2NrB52FF3t2MimlDZY2bNXJ0qHHS3j2mC1p8hqR7DaGjNYQwSLwJoZyJ63cli5ui2jiarQUhkLI21rgtOTQJWlvuYUUVnaJouqYQKosqgaSw2TQKWo6kYpoAeD1TFNeOzhx2ClPmughwzUMO9i4W5vpH8ZgVompL+yqMpgZa9TEBtYjXgBWDaggUXXGW3ApkaxKNsgmL1PITNIUyJmEpgySBND69ucQPDGE6yxITMQFzSdLEbWmtY0uJLSYDaaiSIrpcChXpPbYA0gWNmzVz8hqlKEJVCZb8KaRWlwJu0vG4KYRqDeCqsIqqNFnDCayc4sWoistqZsRBOLrqFtwAqw6t8cHKHEYeXqjlhknaiqAmMdWwK0NaoSqGZG/kI/i0VVNiKHgcrsSLD6O0YhwDmK0uAkYXSPj1MafF2zCNYsgjXnFlhZ+hvlz6IqS4Erb0ZWgjQyoAbTs9IJ4GkCsAxW+egqAis6uS31pahKsCYyWMkM0gSCNT6URdX6DmEVQG1NgxmkIZ4M1snCIAmWBjNYRWBvwppHaXB2K6xZt8DqaptCoLKBEjejqoN5rDBgxI5AFSKrMPzwJqyy4Yc3YeWRlavbMqbIqjNeEya6DrAwZg/p/ZFs6IEDZq7wNKL+FcHKRuLYsyF1FDGivSOwOF4PJX7mcKSIwm66+DqmI8E/Fb4uORSh8gjWPASz0UreDNLine truncated
-
-[image10]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXYAAAFiCAYAAADiGfPoAAB7FElEQVR4XuydB1gV19aGc2tuv3/KTUwxsSb2rthFELtSFMUKduzYC1aw94rSRFBBUUBFEVEEsffeYktssbfEWKJ+//4WzvF4wIIVcH/Ps55zZs2ePYXDO2vW7PIBtLS0tLQylT6wdGhpaWlpZWxpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBraWlpZTJpsGtpaWllMmmwa2lpaWUyabBrvbJOnDiBDh06IEuWLPjkk09QpkwZxMTE4N69e5ZF37muXbsmx2dtbY2HDx9artbSyhTSYNd6JY0YMQIffPBBqlaqVCmBfnrS1atXkTdvXhQpUkSDXSvTSoNd66UVGhpqgnixYsWwefNmHDt2DD169DD5K1WqhN9++02i999///2J7R88eJDCd//+/RQ+LtNY3lzG9gQ0zSj3LKUGdvPtUquDx0RLTUZ54zhSk3mdxvlZnotR5mn70dJKizTYtV5Kd+7cQenSpQXe/LQE1ahRo/DHP/5R1q9evRr//ve/5fuyZctkPSFYoEAB8a1bt058BNv//vc/8d29excnT56Eo6Oj6Sbx1Vdf4fDhw6Z9uLm5ib9hw4ZyYzHKtW3bVlIuqckS7KdOncIf/vAH2W7q1KmmOpo2bYpffvkF48aNk+U//elPst4c3tHR0abyxnXYvn27af2NGzfQrl0703pnZ2dUrFhRvk+ePNlULjIy0lTmP//5j1w7nr+W1stKg13rpXTw4EEBEaG4atUqy9USpWfLlk3K9OzZE35+fvLd1tZW1u/cudMEMwKaioqKkuXixYtj3759JuDyCYCAtbKykuXExEQp37FjR1Mdffv2Ffvb3/4my23atDEdi7kswX7mzBnTDejDDz/E2LFj8fXXX5vq5bFw33/+85/leBYvXiz1+Pv7y/r4+HhZDgkJkeWPP/5Y9nHz5k3897//FR/3xzrc3d1N9c6cOVO243FyuXXr1lKGuX8uFy1aVK6hltbLSINd66WUkJAgAGIku2nTJsvVknr57rvvpAyBxuibAP373/8uwDJy8/R98803EvG3atVKfN7e3ujSpYt8b968uUTfBOWWLVvER9BTBti5H0NNmjQRn4uLi8lnrmeBPSAgQMrwBsHl//u//8OFCxfElzt3bvEx/UTdvn0bt27dkjp4bEuWLJH1fHnMbfjymDeCf/7zn3LuhniDYDne6BiVcx9cvnLlitRz9OhR09PNgQMHTNtpaaVFGuxaL6VDhw4JfAivuLg4y9UCPiNinzZtmgDQSEMwHWNnZ4d//etfqFmzpvi2bdsmqZm//vWvcqNwcHAQf2pWvnx52YcBdnt7e9N+27dvL75GjRqZfOZ6Fth37NghZYz0C1/+GqkX4yYVFhZmqisiIkKOkzcr49gMsDPVwmWmiMzTN4ULFxY/I/5Lly7JU4Ll+RnGVI+W1stIg13rpcQI88svvxQAeXh4iI8Aq1evnsCSKQpC+h//+Ad2794t6728vEwgzpo1K/LkyWOK3F1dXSV6zZcvn+S2jdx6586dpT7mws+ePYuff/5Z9k0ZYGeO3ZCR7ngZsDP9QzEdw+WSJUs+FeyEP29qjMC5r9GjR8t6A+w+Pj6yzHM0fyHKfdNPsF++fFnAzqce4xz5yXOk8T2GltbLSINd66VE4BnRMSPvBQsWSPqFIKOPLzr56enpadpm48aNphw4jfBes2aN3AAMX4sWLaRsnz59ZJkpkIsXL4qPTScJxL1798qyAfYGDRqY9vEqYDfqfR7YCeoKFSrIsgHtDRs2yDLBzuPluRrvCJYuXSplmHoxfEzF8Hp9+umnsswUDMWbF1NCQUFBcoPT0noZabBrvbTOnz9vitoJR0bbRqsWw9gZyGgpQrB9//33pnVM0TBqNeBGW758uZQ9fvy4KU3BfRQsWFCgyWWjvncFdvr4Etg4Fp4X00Nc5svT06dPC/D54pU+5tl5/N9++63pPI2Xp0OGDJFlXjeW+fzzz2WZzUR1xK71stJg13plnTt3Dl27dpXWJART9erVpZWIAUOmUwzNmjVLyhByRkqFL0Ppq1y5sqkcRThOmjQJ2bNnl/VsTmjexpwvOeln80ZDvXv3Fh8Bn5quX78uECaYCWge+xdffCHbsKUPxTQKl6tVq2YCe9myZcXHljsUj437582GrXUYsbM5I8vwqcIQmzLmzJlT9sGnGsKb14TXwRCPgWW4LdfzKUZL61Wkwa71RsWUQnrrffo2xDbsQ4cOlTbp5m3bjRYvvBFoab0pabBrab0B8cnAaC3DF6wcR+ezzz6TZUbmT+tApaX1OqTBrqX1hsS27IUKFTLl1Wls/mh0sNLSelPSYNfSeoNip6MffvhB2v3TdEsXrbchDXYtLS2tTCYNdi0tLa1MJg12LS0trUwmDXYtLS2tTCYNdi0tLa1MJg12LS0trUwmDXYtLS2tTCYNdi0tLa1MJg12LS0trUwmDXYtLS2tTCYNdi0tLa1MJg12LS0trUwmDXYtLS2tTCYNdi0tLa1MJg12LS0trUymNw72n3/+GQcOHMDhw4fTrZ06dcrysLW0tLQyrN442Ldt24YCBQogd+7c6c9y5ZaJkhMSEiwPW0tLSyvD6o2DfdeuXciZKydGBAzDpAUTMDFsfLqxyeGT4NzaGevWrbM8bK10pNu3b8t0cosXL053xtmRtLTSm94K2HPkzAGfxdOw4XwS1p5Zk25s06X1cGjuoMGeznXmzBm0b99eAP/rr7+mG7ty5QpGjx5tebhaWu9cGuwa7OleBHubNm0s3e9cvNFosGulR2mwa7Cne6VXsP/2228a7FrpUhrsGuzpXhrsWlppkwa7Bnu6lwa7llba9NbBvu5cIjZcSML682tlOelsQgrgvi3TYM8Y0mDX0kqb3irYN15ch4FTPNGqR0ssO7AU/cb1wZDpg5FwKh7L1fKKQ8uw+sRKLNkThVXHYwX6SecSsOLwMsQeXi43BQKZ62NUWa5fdSxWLUdi5dEYKRt/Mg5L90YhUdXJ5dXHWV+k1JtwarXaLhrR+xYj8XR8qmDnC7HAwEDEx8ebnYXWu9TLgD0kJAQ5c+bEZ599hhw5cqBjx46WRZ4QW7i4uLhYup8pDXat9Kp3AvZeo3ti373dGDV7BDoP6giPoV1QuqIVBk31RHXHarBvXBe2tSoL0BdsCkXdRnVQsWoFLNq2AF29uqKWc02UrmAlkHZs6oD6bvVQt3FtKd+oXSNUc6wq++HNoUptW9m+qr0d5q4NwbfZvoVr52aI/zEOmy9vELDzGO/evQtfX198//33+NOf/oStW7danorWO9LLgN1QlSpVTN8J+759++LYsWMIDw/HzZs3xT9//ny5mfMGcP/+fVP550mDXSu96q2DfXiAt3RYKlKyCKxrWCN6/2J0GdwJg6cNkvVNOjTG1qsb0bZ3Gwzz90KP4d0wOXwiVv4QgzmJs1FZbcPIfXrkVFXnVNRr7iTRNyG/4nA0qjtUk2h9xZFl0gmp04AO2HFzK5p3aga/5TNRzaGqKdfPiL1eCye0bNkSefLkwQcffCBGGBw8eBA7d+7E7t27sWfPHuzfv198HILg6NGjOH78OE6ePImffvoJp0+fxtmzZ3H+/HlcuHABly9flgjw+vXruHHjBm7duiUQeJrxKeFpdufOnWcab0hPs3v37qVqv//++1ONYHuaPXjw4Kn28OHDZ9qr6Ny5cy8N9kqVKsm1iIqKQpcuXZCUlIQKFSrI37B3795YsmQJpk+fLh2N6E/LsWqwa6VXvXWwS8Q+qid2/7YD265vxrqfE9F9mAcmhU9A/Ik4tOjmho2X1sFzUn94zRiCVt1bwCdqGrZe24SI7QtRupIV7JvYo55rPSzbvwQzlk5HdadqaNaxCVafjMP8DfMkOh8ROAwj1E2kx4ju2Hp9k9pnD8xc6gOXNg0lv2+AvX6LeqhXrx4++ugjE9hz5cqFfv36oUePHujWrRs8PDzQuXNndOrUSTrKtG3bFq1bt0arVq3QokULuLq6olmzZmjSpAkaNWqEhg0bwtnZWep1dHRE3bp1YW9vL+bg4CBGP83JyUmMZevXry/WoEEDMdZDY4qAxrpp3I9hTZs2FeP+mzdvLsbjcXNzE+Px0XisNB43jaCk8Vxo7dq1g7u7uxjPsUOHDmJMYfC8abwGNAKya9euYrw2NF6n7t27yzWj9ezZU6xXr15ihGifPn0kYjasf//+Yp6enmIDBgwQGzRoEAYOHCifgwcPlv3w+ryMCHbezLhvXq/x48fLcBKUv78/6tSpI995U65Ro4b5ps+VBrtWetU7AXtnFaEb+XLmwbt5d5Xomi9VrWtUknw44cwIu8+Y3ug/sR96j+mFoT6DUc2xmmw7ft5YTI+YqsDdDbtubYd7v7YCbkb6i7YuUJ+tVYQfLPWs+XEVKtpVQHB8EBq0cn4C7EzFbNq0SaJuAurvf/87smbNikuXLlmeyjuXEf0yQmYEbUTZBBejUkbwRqRP6PBJgcZekr/88oukHmh8iuDTxLVr18SuXr0qTxh80uB5X7x4UYxPHwQeB3KjMXLmkwlTIzReMw6gRuOTy48//ihPMSdOnJAnGhrTHoyOGRHTjhw5Ik89hw4dEuNTEAeJo/GpaN++fdi7d688JdH4xBQbGys3q5eRAXYvLy9MnDgRGzZskCcxasyYMXKz5fXi+Wqwa2UWvV2wX1iHKQsnYeSs4abWMPwcEzwSs+ICJHof5jcUFezKq4i9n9wIWLine truncated
-
-[image11]: <data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABM8AAARlCAYAAACukOBGAACAAElEQVR4Xuy9B7hV13WuHSd5ch3Zia17rcQl13+K88dyXOLYlmPLRb13CVUkUEMIEB3Rkeiig+iiI3rvvffeO4gmeu+dM+56xz4TbRZ7IxDt7M33Ps98zll97rXGnGvOb4055l+YEEIIIYQQQgghhBAiJX8RXyGEEEIIIYQQQgghhEgg8UwIIYQQQgghhBBCiDRIPBNCCCGEEEIIIYQQIg0Sz4QQQgghhBBCCCGESIPEMyGEEEIIIYQQQggh0iDxTAghhBBCCCGEEEKINEg8E0IIIYQQQgghhBAiDRLPhBBCCCGEEEIIIYRIg8QzIYQQQgghhBBCCCHSIPFMCCGEEEIIIYQQQog0SDwTQgghhBBCCCGEECINEs+EuAhnz5jlKCnF09k0Kb49/B9fH98/1bazSfaXYhvr025PPm98e3xd8nJ826VcK36eVNeN/e/nih+Xajl+vaT1550jvu+lbIv/TfV/fDn5nPHz5yYhhBBCCCFE9iHxTIh05JgdP5Bjx3YrKcXSntTpeG5KXk61/lzafZFtsXPE07n1KfJz7pwp8nvB9ZL2uWBb8j7ptuWm+DWT973geqmOS943zfW+7Den+j++HD9Hym25/8eXk895wfmj8506mhOvRYQQQgghhBBZgMQzIdKBeLY/x45sz7GjO5SUvkhH0qT49vB/fH18/1TbLnbclx2bvC2+Pb4ueTm+Lb5Pqm2p9onvG///Upfj10ten25b/P/4cvzcqbalukb8nBecf7vEMyGEEEIIIbIViWdCXIQT+y3ROd6ppKSklD5RT0g8E0IIIYQQIjuReCZEOnIQz3ISHigpOstKSkpKIUk8E0IIIYQQInuReCZEOsKwTYlnSkpKX5IkngkhhBBCCJG9SDwTIh3yPFNSUrrEJPFMCCGEEEKI7EXimRDpkOeZkpLSJSaJZ0IIIYQQQmQvEs+ESIfEMyUlpUtMEs+EEEIIIYTIXiSeCZEOiWdKSkqXmCSeCSGEEEIIkb1IPBPiIpzYbxLPbmA6vjt6CEej57Dni+dgh83OHkgsu2CxL9rnpPlflk9wzIlon4Nf7HNyb2IfO56bonMe22UXXC+euD7nuZR9SX79KK/k8cuOYd/TkX3lHJKNZUOSeCaEEEIIIUT2IvFMiHRowoAbmhC8tq3bby0afWKfLd3mQhPrenToZ2MGT3ax7PRBs3lTllmJImVsxbz1dvpQ4pjypSrb8H7jfDv7LZu91t59s5gVzk1Vy9ew3RtQuOycoIZQhuCFsBYEtu1r91vfbkNs/+fsYC7K2ZGEKOYiGeuOJY4NQt6GKK9d2va0PZuOfnGu4wkB0MXAIOBFl58zabGNHDDBBTpSOD/58GXyd+ILYZB74PvkCm5nDnyxHL9/Stc3STwTQgghhBAie5F4JkQ6NGzzhiaEorWLt9hD9z9mcycvdVGJde++Ucwa1m7m4hMi1IAew+373/uBNW/Y1h/bkN6jfblRneYuPuHZNbDHCLvvrgds5viFtmDaUlsya5Xt2XjE5kxaZD079bcB3YfZwa2nXOBCdOvZaUC03zL7fNVuGzFgvK1bssUmjphhndv0sCmjZrvnG/v36NjPBvYc4fk9sSfHBa/5U5ZZwfxv2cblO23RjOXWu8tA69NlkO3//IQd3n7Gl7u172N7N+2zSmWq2aMPPm57Nh2wxTNXW6fW3WzCsOkuni2ZtSbK93Dr3qFvlI89djr6vRuWbY/y0N1mTljowtyaRZs9T3OnLPU8xe+h0vVLEs+EEEIIIYTIXiSeCXERNGzzxiWEMjzO8j39ki2YtsLFMtaVKVbeWjRs66IYAtKwvmPtyUeftVJFy9mhHcetecM2lu+pF619y09dcEPQGtpnjL30XH4XoXatP2CHo/3WR+f+nzvudE+0P995t/XpOtjGD5saXe9Fe+O1QvbgfY/YzAkLrGzxCtamWQe741f/YyXeLWP3/Pk+Wz5njbVq3N5KFStnBV9509o07eBeYORp0fSV5zzh7r/nQXvztXeiY+73/HRq3d3ee6ekVS73gQtolcpWs8ceftKFOfLfpF5Le+bxfDZ93Hx75YUC9nz025945Gmr8n4N2711X/S3ur1fspIVi84xacxUK/1eeav9QT17KV9+mzNxkZ2WgHbDksQzIYQQQgghsheJZ0JcBIlnNy5dqng2uNcoK1mkjL1fqrIN6D7CalerbzUq17HWTTu4eMZ+YwZNstv/4z8t/wsFrMDLr1vLxm1s7aLP7bmnXrAd6w5Yp1bdrH7NJlauRAUfErp1zT4r9EYRmzZmrpUrWTHa/xN7r3ApP1fZaJ9P2/exJx992lo1aW8fVKhlTz/+nB3YctLzE8SzZXPWuaj12dKt1rfrYKsV5avtx52sepS3kQNG277NB21g9+HWqM7HdmLP2SjvQ21Qr5H2wrMvW99Ph1iRQiVsanT9GePmW8WyVaPtw/28eNMd2XPUOrfuYX+68y4b2meYPfLA4+5pR/7i91Hp+iSJZ0IIIYQQQmQvEs+ESIeGbd7QhFC2bulWe+HZV1zock4kxLMOLbuee0yDeo6wmlU/cq8uvLVqVatnrZu0TwzjzPU8Q2B7u0DhRKyxXH1j3eIt7nW2c/2h6HyfutcXItzEETNt+7oDVrRQiYR4VqKifdK8k4tfDKdkqGW39n3tuadfsJpVPrKPG7T0oZOHtp2xnMNfiGcMu+TvphU7rVfHAVbnw4Z2cNux6Nge7s3W9ZNePuyzWb1WNnviYnv5+VetWf1W9uC9D/tQ0MrlPrSF01fYhOHTo99X14eSFn+39Lk4bd2jPNz9p/tcCKxfs7FNGjnTTu298D4qXZ8k8UwIIYQQQojsReKZEOmQeHZDE+LZ+mXb7Q+/v8uFq95detnMCXOtyFvF7eV8r1qvzr1sxMARLnzh/TV2yBS77Tu32bC+Y6xFo7ZWv1ZT9wTDGwvx7Df//VsX2Lq1722Deg6xuZOXWIGX37Bta/db8wZtrHHdFja490h7q0Bhq/NBA3vmiXw2bew8K/p2cR8KWrFMNc8XotqwfmPdY6xhnWY+bJLjT+xNzARKzLO3Cha2hdNX+vDPz5ZstS5te9iHlWpbuxado3O19mM/ad7Z+n06xAq88oa1b9HFr9e/21B75IHHXIwrWbSsTygwatBEK1uivK2cv87PhyhYqlhZ69Wlr4tzvToPsBLvlrbJo2b5PYvfR6XrkySeCSGEEEIIkb1IPBMiHRLPbmg6vjvH9m85YZ+262N1PmxgtarWs8G9R3nA/no1Gvsyccfw2po1YaHt3njExadd6w/ZwmkrfBbOk/uYtTPHA+03rdfSaler555pzeq3ttULNrlXF9fAW2zBtOW2d9Mxa1i7oRV45U2PZcZxU0fPskUzVvrwSfJFfLItq/b49Zi4oEGtprZpxS47tS8xacC2Nfv8vMzUOX7YNNuz8agtn7POxbrN0X51qzf0vByIrrt55S6Pp8YQT2boRFQbPWiSX2/K6Nm2dfVej82GMMaEALMmLvJhqUP6jLacI4nZOvk9vToNiOz0rN+z+H1Uuj5J4pkQQgghhBDZi8QzIdIh8eyGJ4ZJMlTTTuemI5aYZTMsn0gMy8yJ0ond5sMZmTGT4P2ITTw70sm9SceQTpqd2pc4F9dgf87F7JYli5axd94o4gIY52afMBkAeWKZ83EdO5VILIdr+frDudujv8d3J2K1cS7PB8ecTKz35ZOJ7f47o//5jZ6fQ4lznGKfQ4nze5y33PtwlGXyx/KxxO+I3z+l65ckngkhhBBCCJG9SDwTIh0Sz266hACFCIewpiGQSpeTJJ4JIYQQQgiRvUg8EyIdEs9uyoSAJi8upctNEs+EEEIIIYTIXiSeCZEOiWdKSkqXmCSeCSGEEEIIkb1IPBMiHRLPlJSULjFJPBNCCCGEECJ7kXgmRDoknikpKV1ikngmhBBCCCFE9iLxTIh0SDxTUlK6xCTxTAghhBBCiOxF4pkQ6ZB4pqSkdIlJ4pkQQgghhBDZi8QzIdIh8UxJKU8kZj89nsdnQJV4JoQQQgghRPYi8UyIdCCe7Ys6xdsTHWMlJaXrn45G6cRus5N77YJteSpF9cSpIxLPhBBCCCGEyEYkngmRhpyoH4wnCR1iJaXz0lG7cDms+7JtydtTLce3pTrvxY5LtRzfFj82+RrxY+P7xrelOzbVvun+j58nadvZE2Zd2/ewMsUr2IlDZ+x0/Nj4NeLnSZe/VMelOybVtvj/UTpzQuKZEEIIIYQQ2YjEMyGEEHmacePHWdu2bS0n52x8kxBCCCGEEEJccySeCSGEEEIIIYQQQgiRBolnQgghhBBCCCGEEEKkQeKZEEIIIYQQQgghhBBpkHgmhBAiTzNixAj76KOP7OxZxTwTQgghhBBCXH8kngkhhMjTFC9e3G699VY7ffp0fJMQQgghhBBCXHMkngkhhMjTVKxY0f7t3/7NTp06Fd8khBBCCCGEENcciWdCCCHyNJUqVbJ//dd/lXgmhBBCCCGEuCFIPBNCCJGnqVy5sv3Lv/yLxDMhhBBCCCHEDUHimRBCiDwNwzZ/+MMfSjwTQgghhBBC3BAkngkhhMjT4Hkm8UwIIYQQQghxo5B4JoQQIk+za9cuW7NmjeXk5MQ3CSGEEEIIIcQ1R+KZEEIIIYQQQgghhBBpkHgmhBBCCCGEEEIIIUQaJJ4JIYQQQgghhBBCCJEGiWdCCCHyNK1bt7Z77rlHEwYIIYQQQgghbggSz4QQQuRpqlSpYn//939vJ0+ejG8SQgghhBBCiGuOxDMhhBB5mpo1a9p3v/tdiWdCCCGEEEKIG4LEMyGEEHmaWrVq2T/+4z9KPBNCCCGEEELcECSeCSGEyNMgnv3DP/yDxDMhhBBCCCHEDUHimRBCiDxNtWrV7JZbbpF4JoQQQgghhLghSLine truncated

docs/architecture/implementation_architecture.md

@@ -1,353 +0,0 @@
-# vLLM-omni Implementation Architecture
-
-## 1. Package Structure
-
-```
-vllm_omni/
-├── __init__.py                 # Main package exports
-├── config/                     # Configuration management
-│   ├── __init__.py
-│   ├── stage_config.py        # OmniStageConfig implementation
-│   ├── dit_config.py          # DiT-specific configurations
-│   └── cache_config.py        # Caching configurations
-├── core/                       # Core processing components
-│   ├── __init__.py
-│   ├── stage_manager.py       # Multi-stage orchestration
-│   ├── dit_cache_manager.py   # DiT caching system
-│   └── sched/                 # Schedulers
-│       ├── __init__.py
-│       ├── scheduler.py       # Base scheduler interface
-
-├── engine/                     # Engine components
-│   ├── __init__.py
-│   ├── processor.py           # Output processing
-│   └── output_processor.py    # Multimodal output handling
-├── executor/                   # Executor implementations
-│   ├── __init__.py
-│   ├── base_executor.py       # Base executor interface
-│   └── diffusers_executor.py  # Diffusers pipeline executor
-├── model_executor/            # Model execution components
-│   ├── __init__.py
-├── worker/                     # Worker implementations
-│   ├── __init__.py
-│   ├── gpu_diffusion_model_runner.py  # Loads and runs diffusers pipelines
-│   └── gpu_diffusion_worker.py        # Worker facade around the diffusion runner
-├── entrypoints/               # Entry points and CLI
-│   ├── __init__.py
-│   ├── omni.py               # OmniServeCommand
-│   ├── omni_llm.py            # OmniLLM and AsyncOmniLLM
-│   └── cli/                  # CLI integration
-│       ├── __init__.py
-│       └── main.py           # CLI main entry point
-├── request.py                 # Request handling
-├── dit_cache_interface.py     # DiT cache interface
-└── utils/                     # Utility functions
-    ├── __init__.py
-    ├── multimodal.py         # Multimodal utilities
-    └── vae.py                # VAE utilities for image processing
-```
-
-## 2. Core Module Dependencies
-
-### 2.1 vLLM Integration Points
-
-```python
-# Key vLLM imports and extensions
-from vllm.v1.engine.llm_engine import LLMEngine
-from vllm.v1.engine.async_llm import AsyncLLM
-from vllm.v1.core.sched.scheduler import SchedulerInterface
-from vllm.v1.executor.multiproc_executor import MultiprocExecutor
-from vllm.v1.worker.gpu_worker import Worker as GPUWorker
-from vllm.v1.worker.gpu_model_runner import GPUModelRunner
-from vllm.v1.outputs import ModelRunnerOutput
-```
-
-### 2.2 Internal Dependencies
-
-```python
-# Internal module dependencies
-vllm_omni.config → vllm_omni.core
-vllm_omni.core → vllm_omni.engine, vllm_omni.executor, vllm_omni.worker
-vllm_omni.engine → vllm_omni.model_executor
-vllm_omni.entrypoints → vllm_omni.core
-```
-
-## 3. Configuration System
-
-### 3.1 Stage Configuration
-
-```python
-# vllm_omni/config/stage_config.py
-@dataclass
-class OmniStageConfig:
-    stage_id: int
-    engine_type: Literal["AR", "DiT"]
-    model_path: str
-    input_modalities: List[str]
-    output_modalities: List[str]
-    vllm_config: Optional[VllmConfig] = None
-    dit_config: Optional[DiTConfig] = None
-    executor_class: Optional[type[Executor]] = None
-    stage_output: Optional[Any] = None
-```
-
-### 3.2 DiT Configuration
-
-```python
-# vllm_omni/config/stage_config.py
-@dataclass
-class DiTConfig:
-    model_config: Optional[ModelConfig] = None
-    scheduler_config: Optional[Any] = None
-    device_config: Optional[DeviceConfig] = None
-    load_config: Optional[LoadConfig] = None
-    compilation_config: Optional[CompilationConfig] = None
-    dit_cache_config: Optional[DiTCacheConfig] = None
-    num_inference_steps: int
-    guidance_scale: float = 7.5
-    use_diffusers: bool = False
-    diffusers_pipeline: Optional[str] = None
-    height: int = 512
-    width: int = 512
-```
-
-### 3.3 Cache Configuration
-
-```python
-# vllm_omni/config/cache_config.py
-@dataclass
-class DiTCacheConfig:
-    cache_tensors: List[DiTCacheTensor]
-    max_cache_size: int
-    cache_strategy: str = "fifo"
-    enable_optimization: bool = True
-```
-
-## 4. Core Implementation Details
-
-### 4.1 OmniLLM Implementation
-
-```python
-# vllm_omni/core/omni_llm.py
-class OmniLLM(LLM):
-    def __init__(self, stage_configs: List[OmniStageConfig]):
-        super().__init__()
-        self.stage_configs = stage_configs
-        self.engine_list: List[LLMEngine] = []
-        self.output_processor = MultimodalOutputProcessor()
-        self._initialize_stage_engines()
-    
-    def _initialize_stage_engines(self) -> None:
-        """Initialize LLMEngine instances for each stage"""
-        for stage_config in self.stage_configs:
-            if stage_config.engine_type == "AR":
-                engine = self._create_ar_engine(stage_config)
-            elif stage_config.engine_type == "DiT":
-                engine = self._create_dit_engine(stage_config)
-            self.engine_list.append(engine)
-    
-    def generate(self, stage_args_list: List[Dict], **kwargs) -> List[RequestOutput]:
-        """Main generation interface - orchestrates multi-stage processing"""
-        current_output = None
-        
-        for i, (stage_config, stage_args) in enumerate(zip(self.stage_configs, stage_args_list)):
-            stage_engine = self.engine_list[i]
-            
-            # Prepare input for this stage
-            processed_input = self._process_stage_inputs(stage_config, stage_args, current_output)
-            
-            # Execute stage
-            stage_output = self._execute_stage(stage_engine, processed_input)
-            
-            # Update for next stage
-            current_output = stage_output
-            stage_config.stage_output = stage_output
-        
-        # Process final output
-        return self.output_processor.process_output(current_output)
-```
-
-### 4.2 AsyncOmniLLM Implementation
-
-```python
-# vllm_omni/core/omni_llm.py
-class AsyncOmniLLM(AsyncLLM):
-    def __init__(self, stage_configs: List[OmniStageConfig]):
-        super().__init__()
-        self.stage_configs = stage_configs
-        self.async_engine_list: List[AsyncLLM] = []
-        self.output_processor = MultimodalOutputProcessor()
-        self._initialize_async_stage_engines()
-    
-    async def generate_async(self, stage_args_list: List[Dict], **kwargs) -> List[RequestOutput]:
-        """Async generation interface"""
-        # Similar to OmniLLM but with async/await patterns
-        pass
-``
-
-### 4.3 Model Runners
-
-```python
-# vllm_omni/model_executor/dit_model_runner.py
-class OmniDiffusionModelRunner(GPUModelRunner):
-    def execute_model(
-        self,
-        scheduler_output: "SchedulerOutput",
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> Union[ModelRunnerOutput, IntermediateTensors]:
-        # DiT model execution logic
-        return ModelRunnerOutput(
-            req_ids=[...],
-            req_id_to_index={...},
-            sampled_token_ids=[],
-            spec_token_ids=None,
-            logprobs=None,
-            prompt_logprobs_dict={},
-            pooler_output=[tensor, ...],  # DiT output tensors
-            kv_connector_output=None,
-            num_nans_in_logits=None,
-        )
-
-# vllm_omni/model_executor/ar_model_runner.py
-class OmniARModelRunner(GPUModelRunner):
-    def execute_model(
-        self,
-        scheduler_output: "SchedulerOutput",
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> Union[ModelRunnerOutput, IntermediateTensors]:
-        # AR model execution with hidden state output
-        return ModelRunnerOutput(
-            req_ids=[...],
-            req_id_to_index={...},
-            sampled_token_ids=[],
-            spec_token_ids=None,
-            logprobs=None,
-            prompt_logprobs_dict={},
-            pooler_output=[hidden_states, ...],  # Hidden states
-            kv_connector_output=None,
-            num_nans_in_logits=None,
-        )
-```
-
-### 4.5 Output Processing
-
-```python
-# vllm_omni/engine/output_processor.py
-class MultimodalOutputProcessor(OutputProcessor):
-    def __init__(self):
-        self.output_handlers: Dict[str, Callable] = {
-            "image": self._process_image_output,
-            "text+image": self._process_text_image_output,
-            "latents": self._process_latents_output,
-            "text": self._process_text_output,
-        }
-    
-    def process_outputs(self, engine_core_outputs: List[EngineCoreOutput], ...):
-        """Process multimodal outputs based on type"""
-        for engine_core_output in engine_core_outputs:
-            output_type = self._detect_output_type(engine_core_output)
-            handler = self.output_handlers.get(output_type, self._process_pooling_output)
-            handler(engine_core_output)
-```
-
-## 5. CLI Integration
-
-### 5.1 Entry Point Override
-
-```python
-# vllm_omni/entrypoints/cli/main.py
-def main():
-    """Main CLI entry point that intercepts vLLM commands"""
-    if "--omni" in sys.argv:
-        omni_args = [arg for arg in sys.argv[1:] if arg != "--omni"]
-        omni_serve = OmniServeCommand()
-        omni_serve.run(omni_args)
-    else:
-        from vllm.entrypoints.cli.main import main as vllm_main
-        vllm_main()
-```
-
-### 5.2 Package Configuration
-
-```toml
-# pyproject.toml updates
-[project.scripts]
-vllm = "vllm_omni.entrypoints.cli.main:main"
-vllm-omni = "vllm_omni.entrypoints.cli.main:main"
-
-[project.entry-points."vllm.plugins"]
-omni = "vllm_omni.plugin:OmniPlugin"
-```
-
-## 6. Testing Strategy
-
-### 6.1 Unit Tests
-- Individual component testing
-- Mock vLLM dependencies
-- Configuration validation
-
-### 6.2 Integration Tests
-- End-to-end pipeline testing
-- vLLM compatibility testing
-- Multi-stage processing validation
-
-### 6.3 Performance Tests
-- Benchmarking against native vLLM
-- Memory usage profiling
-- Latency measurements
-
-## 7. Installation and Setup
-
-### 7.1 Package Installation
-```bash
-pip install vllm>=0.10.2
-pip install vllm-omni
-```
-
-### 7.2 Development Setup
-```bash
-git clone https://github.com/hsliuustc0106/vllm-omni
-cd vllm-omni
-pip install -e ".[dev]"
-```
-
-### 7.3 Usage
-```bash
-vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000
-```
-
-## 8. Memory Management
-
-### 8.1 DiT Cache Management
-- Tensor caching for intermediate results
-- Memory pooling for efficient allocation
-- Cache eviction strategies
-
-### 8.2 Multi-Stage Memory
-- Inter-stage data passing optimization
-- Memory sharing between stages
-- Garbage collection optimization
-
-## 9. Error Handling
-
-### 9.1 Stage Failure Handling
-- Graceful degradation on stage failures
-- Error propagation and reporting
-- Recovery mechanisms
-
-### 9.2 vLLM Compatibility
-- Version compatibility checks
-- API change detection
-- Fallback mechanisms
-
-## 10. Monitoring and Logging
-
-### 10.1 Performance Metrics
-- Stage execution times
-- Memory usage per stage
-- Throughput measurements
-
-### 10.2 Debug Information
-- Request tracing across stages
-- Cache hit/miss ratios
-- Error logging and reporting

examples/basic/api_client.py

@@ -1,128 +0,0 @@
-#!/usr/bin/env python3
-"""
-Simple API client example for vLLM-omni server.
-
-This example shows how to interact with the vLLM-omni API server
-using HTTP requests.
-"""
-
-import requests
-import json
-import time
-
-
-def test_health(host="localhost", port=8000):
-    """Test if the server is healthy."""
-    url = f"http://{host}:{port}/health"
-    try:
-        response = requests.get(url, timeout=5)
-        if response.status_code == 200:
-            print("✅ Server is healthy")
-            return True
-        else:
-            print(f"❌ Server health check failed: {response.status_code}")
-            return False
-    except requests.exceptions.RequestException as e:
-        print(f"❌ Cannot connect to server: {e}")
-        return False
-
-
-def generate_text(prompt, host="localhost", port=8000, **kwargs):
-    """Generate text using the API server."""
-    url = f"http://{host}:{port}/generate"
-    
-    # Default parameters
-    params = {
-        "prompts": [prompt],
-        "max_tokens": kwargs.get("max_tokens", 100),
-        "temperature": kwargs.get("temperature", 0.7),
-        "top_p": kwargs.get("top_p", 1.0),
-        "frequency_penalty": kwargs.get("frequency_penalty", 0.0),
-        "presence_penalty": kwargs.get("presence_penalty", 0.0),
-    }
-    
-    # Add optional parameters
-    if "stop" in kwargs:
-        params["stop"] = kwargs["stop"]
-    
-    try:
-        response = requests.post(url, json=params, timeout=30)
-        response.raise_for_status()
-        return response.json()
-    except requests.exceptions.RequestException as e:
-        print(f"❌ API request failed: {e}")
-        return None
-
-
-def main():
-    """Main example function."""
-    print("vLLM-omni API Client Example")
-    print("=" * 40)
-    
-    # Test server health
-    if not test_health():
-        print("Please start the server first:")
-        print("  vllm serve Qwen/Qwen3-0.6B --omni --port 8000")
-        return
-    
-    # Example 1: Simple generation
-    print("\n1. Simple text generation:")
-    result = generate_text("Hello, how are you?", max_tokens=50)
-    if result:
-        text = result["outputs"][0]["text"]
-        print(f"   Input: Hello, how are you?")
-        print(f"   Output: {text}")
-    
-    # Example 2: Creative writing
-    print("\n2. Creative writing:")
-    result = generate_text(
-        "Write a short story about a robot learning to paint",
-        max_tokens=150,
-        temperature=0.9
-    )
-    if result:
-        text = result["outputs"][0]["text"]
-        print(f"   Input: Write a short story about a robot learning to paint")
-        print(f"   Output: {text}")
-    
-    # Example 3: Question answering
-    print("\n3. Question answering:")
-    result = generate_text(
-        "What is the capital of France?",
-        max_tokens=30,
-        temperature=0.3
-    )
-    if result:
-        text = result["outputs"][0]["text"]
-        print(f"   Input: What is the capital of France?")
-        print(f"   Output: {text}")
-    
-    # Example 4: Multiple prompts
-    print("\n4. Multiple prompts:")
-    prompts = [
-        "Tell me a joke",
-        "What's 2+2?",
-        "Describe the color blue"
-    ]
-    
-    for prompt in prompts:
-        result = generate_text(prompt, max_tokens=30)
-        if result:
-            text = result["outputs"][0]["text"]
-            print(f"   Q: {prompt}")
-            print(f"   A: {text}")
-    
-    # Example 5: Show full response structure
-    print("\n5. Full response structure:")
-    result = generate_text("Hello world", max_tokens=20)
-    if result:
-        print("   Full API response:")
-        print(json.dumps(result, indent=2))
-    
-    print("\n" + "=" * 40)
-    print("API client examples completed!")
-
-
-if __name__ == "__main__":
-    main()
-

examples/basic/text_generation.py

@@ -1,137 +0,0 @@
-#!/usr/bin/env python3
-"""
-Simple vLLM-omni usage example.
-
-This example demonstrates how to use vLLM-omni for basic text generation
-with a single AR (Autoregressive) stage.
-"""
-
-import asyncio
-from vllm_omni.entrypoints.omni_llm import OmniLLM, AsyncOmniLLM
-from vllm_omni.config import create_ar_stage_config
-
-
-def sync_example():
-    """Synchronous usage example."""
-    print("=== Synchronous vLLM-omni Example ===")
-    
-    # Create a simple AR stage configuration
-    stage_config = create_ar_stage_config(
-        stage_id=0,
-        model_path="Qwen/Qwen3-0.6B",
-        input_modalities=["text"],
-        output_modalities=["text"]
-    )
-    
-    # Initialize OmniLLM
-    omni_llm = OmniLLM([stage_config])
-    
-    # Prepare stage arguments
-    stage_args = [{
-        "prompt": "Hello, how are you today?",
-        "max_tokens": 50,
-        "temperature": 0.7
-    }]
-    
-    # Generate text
-    print(f"Input: {stage_args[0]['prompt']}")
-    outputs = omni_llm.generate(stage_args)
-    
-    # Display results
-    for i, output in enumerate(outputs):
-        print(f"Output {i}:")
-        if hasattr(output, 'outputs') and output.outputs:
-            for completion in output.outputs:
-                print(f"  Text: {completion.text}")
-                print(f"  Finished: {completion.finish_reason != 'length'}")
-                print(f"  Tokens: {len(completion.token_ids)} tokens")
-
-
-async def async_example():
-    """Asynchronous usage example."""
-    print("\n=== Asynchronous vLLM-omni Example ===")
-    
-    # Create a simple AR stage configuration
-    stage_config = create_ar_stage_config(
-        stage_id=0,
-        model_path="Qwen/Qwen3-0.6B",
-        input_modalities=["text"],
-        output_modalities=["text"]
-    )
-    
-    # Initialize AsyncOmniLLM
-    omni_llm = AsyncOmniLLM([stage_config])
-    
-    # Prepare stage arguments
-    stage_args = [{
-        "prompt": "What is artificial intelligence?",
-        "max_tokens": 100,
-        "temperature": 0.8
-    }]
-    
-    # Generate text asynchronously
-    print(f"Input: {stage_args[0]['prompt']}")
-    outputs = await omni_llm.generate_async(stage_args)
-    
-    # Display results
-    for i, output in enumerate(outputs):
-        print(f"Output {i}:")
-        if hasattr(output, 'outputs') and output.outputs:
-            for completion in output.outputs:
-                print(f"  Text: {completion.text}")
-                print(f"  Finished: {completion.finish_reason != 'length'}")
-                print(f"  Tokens: {len(completion.token_ids)} tokens")
-
-
-def multi_prompt_example():
-    """Example with multiple prompts."""
-    print("\n=== Multi-Prompt Example ===")
-    
-    # Create stage configuration
-    stage_config = create_ar_stage_config(
-        stage_id=0,
-        model_path="Qwen/Qwen3-0.6B",
-        input_modalities=["text"],
-        output_modalities=["text"]
-    )
-    
-    # Initialize OmniLLM
-    omni_llm = OmniLLM([stage_config])
-    
-    # Multiple prompts
-    prompts = [
-        "Tell me a joke",
-        "What's the weather like?",
-        "Explain quantum computing"
-    ]
-    
-    for prompt in prompts:
-        print(f"\nInput: {prompt}")
-        stage_args = [{
-            "prompt": prompt,
-            "max_tokens": 30,
-            "temperature": 0.9
-        }]
-        
-        outputs = omni_llm.generate(stage_args)
-        
-        if outputs and hasattr(outputs[0], 'outputs') and outputs[0].outputs:
-            completion = outputs[0].outputs[0]
-            print(f"Output: {completion.text}")
-
-
-if __name__ == "__main__":
-    print("vLLM-omni Simple Usage Examples")
-    print("=" * 40)
-    
-    # Run synchronous example
-    sync_example()
-    
-    # Run asynchronous example
-    asyncio.run(async_example())
-    
-    # Run multi-prompt example
-    multi_prompt_example()
-    
-    print("\n" + "=" * 40)
-    print("Examples completed!")

examples/offline_inference/qwen_2_5_omni/README.md

@@ -1,37 +0,0 @@
-# Offline Example of vLLM-omni for Qwen2.5-omni
-
-## Installation
-
-Set up basic environments
-```bash
-uv venv --python 3.12 --seed
-source .venv/bin/activate
-```
-Install certain version of vllm with commitid: 808a7b69df479b6b3a16181711cac7ca28a9b941
-
-```bash
-git clone https://github.com/vllm-project/vllm.git
-cd vllm
-git checkout 808a7b69df479b6b3a16181711cac7ca28a9b941
-VLLM_USE_PRECOMPILED=1 uv pip install --editable .
-```
-
-## Run examples
-
-Get into the example folder
-```bash
-cd vllm_omni
-cd examples/offline_inference/qwen2_5_omni
-```
-Modify PYTHONPATH in run.sh as your path of vllm_omni. Then run.
-```bash
-bash run.sh
-```
-The output audio is saved in ./output_audio
-
-## To-do list
-- [x] Offline inference example for Qwen2.5-omni with single request
-- [ ] Adaptation from current vllm branch to stable vllm v0.11.0
-- [ ] Offline inference example for Qwen2.5-omni with streaming multiple requests
-- [ ] Online inference support
-- [ ] Support for other models

examples/offline_inference/qwen_2_5_omni/end2end.py

@@ -1,130 +0,0 @@
-import argparse
-import os
-import soundfile as sf
-import random
-import numpy as np
-import torch
-
-from vllm.sampling_params import SamplingParams
-
-import os as _os_env_toggle
-_os_env_toggle.environ["VLLM_USE_V1"] = "1"
-
-from vllm_omni.entrypoints.omni_llm import OmniLLM
-from utils import make_omni_prompt
-
-
-SEED = 42
-# Set all random seeds
-random.seed(SEED)
-np.random.seed(SEED)
-torch.manual_seed(SEED)
-torch.cuda.manual_seed(SEED)
-torch.cuda.manual_seed_all(SEED)
-
-# Make PyTorch deterministic
-torch.backends.cudnn.deterministic = True
-torch.backends.cudnn.benchmark = False
-
-# Set environment variables for deterministic behavior
-os.environ["PYTHONHASHSEED"] = str(SEED)
-os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
-
-
-def parse_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--model', required=True, help='Path to merged model directory (will be created if downloading).')
-    parser.add_argument('--thinker-model', type=str, default=None)
-    parser.add_argument('--talker-model', type=str, default=None)
-    parser.add_argument('--code2wav-model', type=str, default=None)
-    parser.add_argument('--hf-hub-id', default='Qwen/Qwen2.5-Omni-7B', help='Hugging Face repo id to download if needed.')
-    parser.add_argument('--hf-revision', default=None, help='Optional HF revision (branch/tag/commit).')
-    parser.add_argument('--prompts', required=True, nargs='+', help='Input text prompts.')
-    parser.add_argument('--voice-type', default='default', help='Voice type, e.g., m02, f030, default.')
-    parser.add_argument('--code2wav-dir', default=None, help='Path to code2wav folder (contains spk_dict.pt).')
-    parser.add_argument('--dit-ckpt', default=None, help='Path to DiT checkpoint file (e.g., dit.pt).')
-    parser.add_argument('--bigvgan-ckpt', default=None, help='Path to BigVGAN checkpoint file.')
-    parser.add_argument('--dtype', default='bfloat16', choices=['float16', 'bfloat16', 'float32'])
-    parser.add_argument('--max-model-len', type=int, default=32768)
-
-    parser.add_argument("--thinker-only", action="store_true")
-    parser.add_argument("--text-only", action="store_true")
-    parser.add_argument("--do-wave", action="store_true")
-    parser.add_argument('--prompt_type',
-                        choices=[
-                            'text', 'audio', 'audio-long', 'audio-long-chunks',
-                            'audio-long-expand-chunks', 'image', 'video',
-                            'video-frames', 'audio-in-video', 'audio-in-video-v2',
-                            "audio-multi-round", "badcase-vl", "badcase-text",
-                            "badcase-image-early-stop", "badcase-two-audios",
-                            "badcase-two-videos", "badcase-multi-round",
-                            "badcase-voice-type", "badcase-voice-type-v2",
-                            "badcase-audio-tower-1", "badcase-audio-only"
-                        ],
-                        default='text')
-    parser.add_argument('--use-torchvision', action='store_true')
-    parser.add_argument('--tokenize', action='store_true')
-    parser.add_argument('--output-wav', default="output.wav", help='Output wav file path.')
-    parser.add_argument('--thinker-hidden-states-dir', default="thinker_hidden_states", help='Path to thinker hidden states directory.')
-    args = parser.parse_args()
-    return args
-
-
-def main():
-    args = parse_args()
-    model_name = args.model
-    omni_llm = OmniLLM(model=model_name)
-    thinker_sampling_params = SamplingParams(
-                                            temperature=0.0,    # Deterministic - no randomness
-                                            top_p=1.0,          # Disable nucleus sampling
-                                            top_k=-1,           # Disable top-k sampling
-                                            max_tokens=2048,
-                                            seed=SEED,          # Fixed seed for sampling
-                                            detokenize=True,
-                                            repetition_penalty=1.1,
-                                            )
-    talker_sampling_params = SamplingParams(
-                                            temperature=0.0,    # Deterministic - no randomness
-                                            top_p=1.0,          # Disable nucleus sampling
-                                            top_k=-1,           # Disable top-k sampling
-                                            max_tokens=2048,
-                                            seed=SEED,          # Fixed seed for sampling
-                                            detokenize=True,
-                                            repetition_penalty=1.1,
-                                            stop_token_ids=[8294]
-                                            )
-    code2wav_sampling_params = SamplingParams(
-                                            temperature=0.0,    # Deterministic - no randomness
-                                            top_p=1.0,          # Disable nucleus sampling
-                                            top_k=-1,           # Disable top-k sampling
-                                            max_tokens=2048,
-                                            seed=SEED,          # Fixed seed for sampling
-                                            detokenize=True,
-                                            repetition_penalty=1.1,
-                                            )
-
-    sampling_params_list = [thinker_sampling_params,
-                            talker_sampling_params,
-                            code2wav_sampling_params]
-    
-    prompt = [make_omni_prompt(args, prompt) for prompt in args.prompts]
-    omni_outputs = omni_llm.generate(prompt, sampling_params_list)
-
-    os.makedirs(args.output_wav, exist_ok=True)
-    for stage_outputs in omni_outputs:
-        if stage_outputs.final_output_type == "text":
-            for output in stage_outputs.request_output:
-                request_id = output.request_id
-                text_output = output.outputs[0].text
-                print(f"Request ID: {request_id}, Text Output: {text_output}")
-        elif stage_outputs.final_output_type == "audio":
-            for output in stage_outputs.request_output:
-                request_id = output.request_id
-                audio_tensor = output.multimodal_output["audio"]
-                output_wav = os.path.join(args.output_wav, f"output_{output.request_id}.wav")
-                sf.write(output_wav, audio_tensor.detach().cpu().numpy(), samplerate=24000)
-                print(f"Request ID: {request_id}, Saved audio to {output_wav}")
-
-
-if __name__ == "__main__":
-    main()

examples/offline_inference/qwen_2_5_omni/processing_omni.py

@@ -1,373 +0,0 @@
-from __future__ import annotations
-
-import base64
-import logging
-import math
-import os
-import time
-import warnings
-from functools import lru_cache
-from io import BytesIO
-import requests
-import torch
-import torchvision
-from packaging import version
-from PIL import Image
-from torchvision import io, transforms
-from torchvision.transforms import InterpolationMode
-
-
-logger = logging.getLogger(__name__)
-
-IMAGE_FACTOR = 28
-MIN_PIXELS = 4 * 28 * 28
-MAX_PIXELS = 16384 * 28 * 28
-MAX_RATIO = 200
-
-VIDEO_MIN_PIXELS = 128 * 28 * 28
-VIDEO_MAX_PIXELS = 768 * 28 * 28
-VIDEO_TOTAL_PIXELS = 24576 * 28 * 28
-FRAME_FACTOR = 2
-FPS = 2.0
-FPS_MIN_FRAMES = 4
-FPS_MAX_FRAMES = 768
-
-temporal_patch_size = 2
-spatial_patch_size = 14
-spatial_merge_size = 2
-
-
-def round_by_factor(number: int, factor: int) -> int:
-    """Returns the closest integer to 'number' that is divisible by 'factor'."""
-    return round(number / factor) * factor
-
-
-def ceil_by_factor(number: int, factor: int) -> int:
-    """Returns the smallest integer greater than or equal to 'number' that is divisible by 'factor'."""
-    return math.ceil(number / factor) * factor
-
-
-def floor_by_factor(number: int, factor: int) -> int:
-    """Returns the largest integer less than or equal to 'number' that is divisible by 'factor'."""
-    return math.floor(number / factor) * factor
-
-
-def smart_resize(height: int,
-                 width: int,
-                 factor: int = IMAGE_FACTOR,
-                 min_pixels: int = MIN_PIXELS,
-                 max_pixels: int = MAX_PIXELS) -> tuple[int, int]:
-    """
-    Rescales the image so that the following conditions are met:
-
-    1. Both dimensions (height and width) are divisible by 'factor'.
-
-    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
-
-    3. The aspect ratio of the image is maintained as closely as possible.
-    """
-    if max(height, width) / min(height, width) > MAX_RATIO:
-        raise ValueError(
-            f"absolute aspect ratio must be smaller than {MAX_RATIO}, got {max(height, width) / min(height, width)}"
-        )
-    h_bar = max(factor, round_by_factor(height, factor))
-    w_bar = max(factor, round_by_factor(width, factor))
-    if h_bar * w_bar > max_pixels:
-        beta = math.sqrt((height * width) / max_pixels)
-        h_bar = floor_by_factor(height / beta, factor)
-        w_bar = floor_by_factor(width / beta, factor)
-    elif h_bar * w_bar < min_pixels:
-        beta = math.sqrt(min_pixels / (height * width))
-        h_bar = ceil_by_factor(height * beta, factor)
-        w_bar = ceil_by_factor(width * beta, factor)
-    return h_bar, w_bar
-
-
-def fetch_image(ele: dict[str, str | Image.Image],
-                size_factor: int = IMAGE_FACTOR) -> Image.Image:
-    if "image" in ele:
-        image = ele["image"]
-    else:
-        image = ele["image_url"]
-    image_obj = None
-    if isinstance(image, Image.Image):
-        image_obj = image
-    elif image.startswith("http://") or image.startswith("https://"):
-        image_obj = Image.open(requests.get(image, stream=True).raw)
-    elif image.startswith("file://"):
-        image_obj = Image.open(image[7:])
-    elif image.startswith("data:image"):
-        if "base64," in image:
-            _, base64_data = image.split("base64,", 1)
-            data = base64.b64decode(base64_data)
-            image_obj = Image.open(BytesIO(data))
-    else:
-        image_obj = Image.open(image)
-    if image_obj is None:
-        raise ValueError(
-            f"Unrecognized image input, support local path, http url, base64 and PIL.Image, got {image}"
-        )
-    image = image_obj.convert("RGB")
-    ## resize
-    if "resized_height" in ele and "resized_width" in ele:
-        resized_height, resized_width = smart_resize(
-            ele["resized_height"],
-            ele["resized_width"],
-            factor=size_factor,
-        )
-    else:
-        width, height = image.size
-        min_pixels = ele.get("min_pixels", MIN_PIXELS)
-        max_pixels = ele.get("max_pixels", MAX_PIXELS)
-        resized_height, resized_width = smart_resize(
-            height,
-            width,
-            factor=size_factor,
-            min_pixels=min_pixels,
-            max_pixels=max_pixels,
-        )
-    image = image.resize((resized_width, resized_height))
-
-    return image
-
-
-def smart_nframes(
-    ele: dict,
-    total_frames: int,
-    video_fps: int | float,
-) -> int:
-    """calculate the number of frames for video used for model inputs.
-
-    Args:
-        ele (dict): a dict contains the configuration of video.
-            support either `fps` or `nframes`:
-                - nframes: the number of frames to extract for model inputs.
-                - fps: the fps to extract frames for model inputs.
-                    - min_frames: the minimum number of frames of the video, only used when fps is provided.
-                    - max_frames: the maximum number of frames of the video, only used when fps is provided.
-        total_frames (int): the original total number of frames of the video.
-        video_fps (int | float): the original fps of the video.
-
-    Raises:
-        ValueError: nframes should in interval [FRAME_FACTOR, total_frames].
-
-    Returns:
-        int: the number of frames for video used for model inputs.
-    """
-    assert not ("fps" in ele
-                and "nframes" in ele), "Only accept either `fps` or `nframes`"
-    if "nframes" in ele:
-        nframes = round_by_factor(ele["nframes"], FRAME_FACTOR)
-    else:
-        fps = ele.get("fps", FPS)
-        min_frames = ceil_by_factor(ele.get("min_frames", FPS_MIN_FRAMES),
-                                    FRAME_FACTOR)
-        max_frames = floor_by_factor(
-            ele.get("max_frames", min(FPS_MAX_FRAMES, total_frames)),
-            FRAME_FACTOR)
-        nframes = total_frames / video_fps * fps
-        nframes = min(max(nframes, min_frames), max_frames)
-        nframes = round_by_factor(nframes, FRAME_FACTOR)
-    if not (FRAME_FACTOR <= nframes and nframes <= total_frames):
-        raise ValueError(
-            f"nframes should in interval [{FRAME_FACTOR}, {total_frames}], but got {nframes}."
-        )
-    return nframes
-
-
-def _read_video_torchvision(ele: dict, ) -> torch.Tensor:
-    """read video using torchvision.io.read_video
-
-    Args:
-        ele (dict): a dict contains the configuration of video.
-        support keys:
-            - video: the path of video. support "file://", "http://", "https://" and local path.
-            - video_start: the start time of video.
-            - video_end: the end time of video.
-    Returns:
-        torch.Tensor: the video tensor with shape (T, C, H, W).
-    """
-    video_path = ele["video"]
-    if version.parse(torchvision.__version__) < version.parse("0.19.0"):
-        if "http://" in video_path or "https://" in video_path:
-            warnings.warn(
-                "torchvision < 0.19.0 does not support http/https video path, please upgrade to 0.19.0."
-            )
-        if "file://" in video_path:
-            video_path = video_path[7:]
-    st = time.time()
-    video, audio, info = io.read_video(
-        video_path,
-        start_pts=ele.get("video_start", 0.0),
-        end_pts=ele.get("video_end", None),
-        pts_unit="sec",
-        output_format="TCHW",
-    )
-    total_frames, video_fps = video.size(0), info["video_fps"]
-    total_duration = round(total_frames / video_fps, 3)
-    logger.info(
-        f"torchvision:  {video_path=}, {total_frames=}, {video_fps=}, duration={total_duration}s, time={time.time() - st:.3f}s"
-    )
-    nframes = smart_nframes(ele,
-                            total_frames=total_frames,
-                            video_fps=video_fps)
-    idx = torch.linspace(0, total_frames - 1, nframes).round().long()
-    video = video[idx]
-    return video, total_duration, nframes
-
-
-def is_decord_available() -> bool:
-    import importlib.util
-
-    return importlib.util.find_spec("decord") is not None
-
-
-def _read_video_decord(ele: dict, ) -> torch.Tensor:
-    """read video using decord.VideoReader
-
-    Args:
-        ele (dict): a dict contains the configuration of video.
-        support keys:
-            - video: the path of video. support "file://", "http://", "https://" and local path.
-            - video_start: the start time of video.
-            - video_end: the end time of video.
-    Returns:
-        torch.Tensor: the video tensor with shape (T, C, H, W).
-    """
-    import decord
-    video_path = ele["video"]
-    st = time.time()
-    vr = decord.VideoReader(video_path)
-    # TODO: support start_pts and end_pts
-    if 'video_start' in ele or 'video_end' in ele:
-        raise NotImplementedError(
-            "not support start_pts and end_pts in decord for now.")
-    total_frames, video_fps = len(vr), vr.get_avg_fps()
-    total_duration = round(total_frames / video_fps, 3)
-    logger.info(
-        f"decord:  {video_path=}, {total_frames=}, {video_fps=}, time={time.time() - st:.3f}s"
-    )
-    nframes = smart_nframes(ele,
-                            total_frames=total_frames,
-                            video_fps=video_fps)
-    idx = torch.linspace(0, total_frames - 1, nframes).round().long().tolist()
-    video = vr.get_batch(idx).asnumpy()
-    video = torch.tensor(video).permute(0, 3, 1, 2)  # Convert to TCHW format
-    return video, total_duration, nframes
-
-
-VIDEO_READER_BACKENDS = {
-    "decord": _read_video_decord,
-    "torchvision": _read_video_torchvision,
-}
-
-FORCE_QWENVL_VIDEO_READER = os.getenv("FORCE_QWENVL_VIDEO_READER", None)
-
-
-@lru_cache(maxsize=1)
-def get_video_reader_backend() -> str:
-    if FORCE_QWENVL_VIDEO_READER is not None:
-        video_reader_backend = FORCE_QWENVL_VIDEO_READER
-    elif is_decord_available():
-        video_reader_backend = "decord"
-    else:
-        video_reader_backend = "torchvision"
-    # print(f"qwen-vl-utils using {video_reader_backend} to read video.", file=sys.stderr)
-    return video_reader_backend
-
-
-def fetch_video(
-        ele: dict,
-        image_factor: int = IMAGE_FACTOR) -> torch.Tensor | list[Image.Image]:
-    if isinstance(ele["video"], str):
-        video_reader_backend = get_video_reader_backend()
-        video, total_dur, nframes = VIDEO_READER_BACKENDS[
-            video_reader_backend](ele)
-        frame_timestamps = total_dur * torch.arange(1, nframes + 1) / nframes
-        grid_timestamps = frame_timestamps[::FRAME_FACTOR]
-        second_per_grid = grid_timestamps[1] - grid_timestamps[0]
-        nframes, _, height, width = video.shape
-        factor = spatial_patch_size * spatial_merge_size
-        min_pixels = ele.get("min_pixels", VIDEO_MIN_PIXELS)
-        total_pixels = ele.get("total_pixels", VIDEO_TOTAL_PIXELS)
-        max_pixels = max(
-            min(VIDEO_MAX_PIXELS, total_pixels / nframes * FRAME_FACTOR),
-            int(min_pixels * 1.05))
-        max_pixels = ele.get("max_pixels", max_pixels)
-        # min_pixels = (factor ** 2) * 52
-        # max_pixels = (factor ** 2) * min(768, (16384 / nframes * temporal_patch_size))
-        if "resized_height" in ele and "resized_width" in ele:
-            resized_height, resized_width = smart_resize(
-                ele["resized_height"],
-                ele["resized_width"],
-                factor=image_factor,
-            )
-        else:
-            resized_height, resized_width = smart_resize(
-                height,
-                width,
-                factor=image_factor,
-                min_pixels=min_pixels,
-                max_pixels=max_pixels,
-            )
-        video = transforms.functional.resize(
-            video,
-            [resized_height, resized_width],
-            interpolation=InterpolationMode.BICUBIC,
-            antialias=True,
-        ).float()
-        return video, total_dur, nframes, second_per_grid
-    else:
-        assert isinstance(ele["video"], (list, tuple))
-        process_info = ele.copy()
-        process_info.pop("type", None)
-        process_info.pop("video", None)
-        images = [
-            fetch_image({
-                "image": video_element,
-                **process_info
-            },
-                        size_factor=image_factor)
-            for video_element in ele["video"]
-        ]
-        nframes = ceil_by_factor(len(images), FRAME_FACTOR)
-        if len(images) < nframes:
-            images.extend([images[-1]] * (nframes - len(images)))
-        return images, None, None, None
-
-
-def extract_vision_info(
-        conversations: list[dict] | list[list[dict]]) -> list[dict]:
-    vision_infos = []
-    if isinstance(conversations[0], dict):
-        conversations = [conversations]
-    for conversation in conversations:
-        for message in conversation:
-            if isinstance(message["content"], list):
-                for ele in message["content"]:
-                    if ("image" in ele or "image_url" in ele or "video" in ele
-                            or ele["type"] in ("image", "image_url", "video")):
-                        vision_infos.append(ele)
-    return vision_infos
-
-
-def process_vision_info(
-    conversations: list[dict] | list[list[dict]],
-) -> tuple[list[Image.Image] | None, list[torch.Tensor | list[Image.Image]]
-           | None]:
-    vision_infos = extract_vision_info(conversations)
-    ## Read images or videos
-    image_inputs = []
-    video_inputs = []
-    for vision_info in vision_infos:
-        if "image" in vision_info or "image_url" in vision_info:
-            image_inputs.append(fetch_image(vision_info))
-        elif "video" in vision_info:
-            video_inputs.append(fetch_video(vision_info))
-        else:
-            raise ValueError("image, image_url or video should in content.")
-    if len(image_inputs) == 0:
-        image_inputs = None
-    if len(video_inputs) == 0:
-        video_inputs = None
-    return image_inputs, video_inputs

examples/offline_inference/qwen_2_5_omni/run.sh

@@ -1,9 +0,0 @@
-export PYTHONPATH=/path/to/vllm-omni:$PYTHONPATH
-export HF_ENDPOINT=https://hf-mirror.com
-python end2end.py --model Qwen/Qwen2.5-Omni-7B \
-                                 --prompts "Explain the system architecture for a scalable audio generation pipeline. Answer in 15 words." \
-                                 --voice-type "m02" \
-                                 --dit-ckpt none \
-                                 --bigvgan-ckpt none \
-                                 --output-wav output_audio \
-                                 --prompt_type text

examples/offline_inference/qwen_2_5_omni/utils.py

@@ -1,305 +0,0 @@
-import tempfile
-from urllib.request import urlopen
-import librosa
-import soundfile as sf
-import resampy
-from typing import Dict, Optional
-import torch
-import requests
-import torchvision.io
-
-from typing import Union, List
-from vllm.inputs import TextPrompt
-from vllm_omni.inputs.data import OmniTokensPrompt
-from processing_omni import fetch_image, fetch_video
-
-
-def get_system_prompt():
-    
-    return {
-        'role':
-        'system',
-        'content': [{
-            'type':
-            'text',
-            'text':
-            'You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.'
-        }]
-    }
-
-
-def resample_wav_to_16khz(input_filepath):
-    data, original_sample_rate = sf.read(input_filepath)
-    # Only use the first channel
-    if len(data.shape) > 1:
-        data = data[:, 0]
-    # resample to 16kHz
-    data_resampled = resampy.resample(data,
-                                      sr_orig=original_sample_rate,
-                                      sr_new=16000)
-    return data_resampled
-
-
-def fetch_and_read_video(args, video_url: str, fps=2):
-
-    def read_video_with_torchvision(video_file_name: str):
-        video, audio, info = torchvision.io.read_video(
-            video_file_name,
-            start_pts=0.0,
-            end_pts=None,
-            pts_unit="sec",
-            output_format="TCHW",
-        )
-
-        total_frames, video_fps = video.size(0), info["video_fps"]
-        total_duration = round(total_frames / video_fps, 3)
-        nframes = int(total_frames / video_fps * fps)
-
-        frame_timestamps = total_duration * torch.arange(1,
-                                                         nframes + 1) / nframes
-        grid_timestamps = frame_timestamps[::2]
-        second_per_grid = grid_timestamps[1] - grid_timestamps[0]
-
-        idx = torch.linspace(0, video.size(0) - 1, nframes).round().long()
-        video = video[idx]
-
-        if args.legacy_omni_video:
-            return [video, total_duration, nframes, second_per_grid.item()]
-        else:
-            return video
-
-    def read_video_with_transformers(video_file_name: Union[str, List[str]]):
-        video, total_duration, nframes, second_per_grid = fetch_video(
-            {'video': video_file_name})
-        if total_duration is None and nframes is None:
-            nframes = len(video)
-            total_duration = 0.5 * nframes
-            second_per_grid = 1.0
-        if args.legacy_omni_video:
-            return [video, total_duration, nframes, second_per_grid]
-        else:
-            return video
-
-    def read_video(video_file_name: str):
-        if args.use_torchvision:
-            return read_video_with_torchvision(video_file_name)
-        else:
-            return read_video_with_transformers(video_file_name)
-
-    if isinstance(video_url, str) and video_url.startswith("http"):
-        with tempfile.NamedTemporaryFile(delete=True) as temp_video_file:
-            resp = requests.get(video_url)
-            assert resp.status_code == requests.codes.ok, f"Failed to fetch video from {video_url}, status_code:{resp.status_code}, resp:{resp}"
-
-            temp_video_file.write(urlopen(video_url).read())
-            temp_video_file_path = temp_video_file.name
-            video_file_name = temp_video_file_path
-            return read_video(video_file_name)
-    else:
-        video_file_name = video_url
-        return read_video(video_file_name)
-
-
-def make_inputs_qwen2_omni(
-    args,
-    messages: List[Dict[str, Union[str, List[Dict[str, str]]]]],
-    use_audio_in_video: Optional[bool] = False,
-    tokenize: bool = False,
-) -> Union[OmniTokensPrompt, TextPrompt]:
-    
-    from transformers import AutoConfig, AutoProcessor, AutoTokenizer
-    processor = AutoProcessor.from_pretrained(args.model)
-    tokenizer = AutoTokenizer.from_pretrained(args.model)
-
-    try:
-        config = AutoConfig.from_pretrained(args.model)
-        if 'Qwen2_5OmniModel' in config.architectures:
-            args.legacy_omni_video = False
-        else:
-            args.legacy_omni_video = True
-    except:
-        args.legacy_omni_video = True
-
-    audios, images, videos = [], [], []
-    for message in messages:
-        if not isinstance(message['content'], list):
-            message['content'] = [{
-                'type': 'text',
-                'text': message['content'],
-            }]
-        index, num_contents = 0, len(message['content'])
-        while index < num_contents:
-            ele = message['content'][index]
-            if 'type' not in ele:
-                if 'text' in ele:
-                    ele['type'] = 'text'
-                elif 'audio' in ele:
-                    ele['type'] = 'audio'
-                elif 'audio_url' in ele:
-                    ele['type'] = 'audio_url'
-                elif 'image' in ele:
-                    ele['type'] = 'image'
-                elif 'image_url' in ele:
-                    ele['type'] = 'image_url'
-                elif 'video' in ele:
-                    ele['type'] = 'video'
-                elif 'video_url' in ele:
-                    ele['type'] = 'video_url'
-                else:
-                    raise ValueError(f'Unknown ele: {ele}')
-
-            if ele['type'] == 'audio' or ele['type'] == 'audio_url':
-                if 'audio_url' in ele:
-                    audio_key = 'audio_url'
-                    with tempfile.NamedTemporaryFile(
-                            delete=True) as temp_audio_file:
-                        temp_audio_file.write(urlopen(ele[audio_key]).read())
-                        temp_audio_file_path = temp_audio_file.name
-                        audios.append(
-                            resample_wav_to_16khz(temp_audio_file_path))
-                        ele['audio'] = temp_audio_file_path
-                elif 'audio' in ele:
-                    audio_key = 'audio'
-                    audios.append(resample_wav_to_16khz(ele[audio_key]))
-                else:
-                    raise ValueError(f'Unknown ele {ele}')
-            elif use_audio_in_video and (ele['type'] == 'video'
-                                         or ele['type'] == 'video_url'):
-                # use video as audio as well
-                if 'video_url' in ele:
-                    audio_key = 'video_url'
-                    with tempfile.NamedTemporaryFile(
-                            delete=True) as temp_video_file:
-                        temp_video_file.write(urlopen(ele[audio_key]).read())
-                        temp_video_file_path = temp_video_file.name
-                        ele[audio_key] = temp_video_file_path
-                        audios.append(
-                            librosa.load(temp_video_file_path, sr=16000)[0])
-                        videos.append(
-                            fetch_and_read_video(args, temp_video_file_path))
-                        ele['video'] = temp_video_file_path
-                elif 'video' in ele:
-                    audio_key = 'video'
-                    audios.append(librosa.load(ele[audio_key], sr=16000)[0])
-                    videos.append(fetch_and_read_video(args, audio_key))
-                else:
-                    raise ValueError("Unknown ele {}".format(ele))
-                # insert a audio after the video
-                message['content'].insert(index + 1, {
-                    "type": "audio",
-                    "audio": ele[audio_key],
-                })
-                # no need to load the added audio again
-                index += 1
-            elif ele['type'] == 'video' or ele['type'] == 'video_url':
-                if 'video_url' in ele:
-                    video_key = 'video_url'
-                    with tempfile.NamedTemporaryFile(
-                            delete=True) as temp_video_file:
-                        temp_video_file.write(urlopen(ele['video_url']).read())
-                        temp_video_file_path = temp_video_file.name
-                        videos.append(fetch_and_read_video(args, temp_video_file))
-                        ele['video'] = temp_video_file_path
-                else:
-                    video_key = 'video'
-                    videos.append(fetch_and_read_video(args, ele[video_key]))
-            elif ele['type'] == 'image' or ele['type'] == 'image_url':
-                images.append(fetch_image(ele))
-
-            # move to the next content
-            index += 1
-
-    prompt = processor.apply_chat_template(
-        messages,
-        tokenize=tokenize,
-        add_generation_prompt=True,
-        add_vision_id=True,
-    )
-
-    audios = audios if len(audios) > 0 else None
-    images = images if len(images) > 0 else None
-    videos = videos if len(videos) > 0 else None
-
-    multi_modal_data = {}
-    if audios:
-        multi_modal_data["audio"] = audios
-    if images:
-        multi_modal_data["image"] = images
-    if videos:
-        multi_modal_data["video"] = videos
-
-    if isinstance(prompt, list) and isinstance(prompt[0], (list, str)):
-        prompt = prompt[0]
-
-    if tokenize:
-        return OmniTokensPrompt(
-            prompt_token_ids=prompt,
-            multi_modal_data=multi_modal_data,
-        )
-    else:
-        return TextPrompt(
-            prompt=prompt,
-            multi_modal_data=multi_modal_data,
-        )
-
-
-def make_text_prompt(args, prompt):
-    messages = [
-        get_system_prompt(),
-        {
-            "role": "user",
-            "content": [
-                {
-                    "type": "text",
-                    "text": prompt
-                },
-            ]
-        },
-    ]
-
-    prompt = make_inputs_qwen2_omni(args, messages, tokenize=args.tokenize)
-    return prompt
-
-
-def make_audio_in_video_v2_prompt(args):
-    messages = [
-        {
-            'role':
-            'system',
-            'content': [{
-                'type':
-                'text',
-                'text':
-                'You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.'
-            }]
-        },
-        {
-            "role":
-            "user",
-            "content": [
-                {
-                    "type":
-                    "video_url",
-                    "video_url":
-                    "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw_small.mp4"
-                },
-            ]
-        },
-    ]
-    prompt = make_inputs_qwen2_omni(
-        args,
-        messages,
-        use_audio_in_video=True,
-        tokenize=args.tokenize,
-    )
-    return prompt
-
-
-def make_omni_prompt(args, prompt = None) -> Union[OmniTokensPrompt, List[OmniTokensPrompt]]:
-    if args.prompt_type == 'text':
-        prompt = make_text_prompt(args, prompt)
-    elif args.prompt_type == 'audio-in-video-v2':
-        prompt = make_audio_in_video_v2_prompt(args)
-    else:
-        raise ValueError(f'Unsupported prompt type: {args.prompt_type}')
-    return prompt

examples/omni/README.md

@@ -1,31 +0,0 @@
-# Omni Examples
-
-Examples showcasing multi-stage AR + DiT pipelines powered by vLLM-omni.
-
-## AR ➜ DiT with Diffusers Backend
-
-```bash
-# Optional: customise models
-export AR_MODEL=Qwen/Qwen3-0.6B
-export DIT_MODEL=./models/stable-diffusion-2-1/
-
-# Run with defaults (20 steps @ 512x512)
-python examples/omni/ar_dit_diffusers.py
-
-# Faster run
-python examples/omni/ar_dit_diffusers.py --steps 14 --height 384 --width 384 --guidance 4.5
-```
-
-Arguments:
-- `--ar-model`: Override AR stage model (default `Qwen/Qwen3-0.6B`).
-- `--dit-model`: Override DiT stage model (default env `DIT_MODEL`, else diffusers repo).
-- `--steps`: Number of diffusion steps (default 20).
-- `--guidance`: Guidance scale (default 5.0).
-- `--height` / `--width`: Output resolution (default 512).
-- `--seed`: Optional RNG seed.
-- `--prompt`, `--temperature`, `--max-tokens`: Control AR stage generation.
-- `--output`: Destination path for the generated image file.
-
-Generated images are saved next to the script as `omni_dit_output.png` by default.
-
-See the script for additional tweaks (scheduler, prompt chaining, etc.).

examples/omni/ar_dit_diffusers.py

@@ -1,175 +0,0 @@
-"""Run the AR → DiT (diffusers) pipeline using YAML configuration defaults."""
-
-from __future__ import annotations
-
-import argparse
-import os
-from pathlib import Path
-from typing import Dict, List
-
-import yaml
-
-from vllm_omni import (
-    OmniLLM,
-    DiTConfig,
-    create_ar_stage_config,
-    create_dit_stage_config,
-)
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Run a two-stage AR → DiT pipeline with config-derived defaults."
-    )
-    parser.add_argument(
-        "--config",
-        type=Path,
-        default=Path(__file__).resolve().parent / "configs" / "ar_dit_local.yaml",
-        help="Path to a YAML configuration describing the pipeline stages.",
-    )
-    parser.add_argument(
-        "--prompt",
-        default="A scenic watercolor painting of a lighthouse at sunset",
-        help="Prompt passed to the AR stage.",
-    )
-    parser.add_argument(
-        "--negative-prompt",
-        default=None,
-        help="Optional negative prompt forwarded to the diffusion stage.",
-    )
-    parser.add_argument(
-        "--seed",
-        type=int,
-        default=None,
-        help="Optional seed for deterministic diffusion sampling.",
-    )
-    parser.add_argument(
-        "--output",
-        type=Path,
-        default=Path("./omni_dit_output.png"),
-        help="Destination path for the generated image.",
-    )
-    return parser.parse_args()
-
-
-def _normalize_modalities(values: List[str] | None, default: List[str]) -> List[str]:
-    if not values:
-        return default
-    return list(values)
-
-
-def _load_stage_configs_from_yaml(config_path: Path):
-    with config_path.open("r", encoding="utf-8") as f:
-        config = yaml.safe_load(f) or {}
-
-    stages = config.get("stages", [])
-    if not stages:
-        raise ValueError(f"No stages defined in config: {config_path}")
-
-    stage_configs = []
-    for stage in stages:
-        stage_id = stage["stage_id"]
-        engine_type = stage["engine_type"]
-        default_stage_args = stage.get("default_stage_args")
-
-        if engine_type == "AR":
-            stage_configs.append(
-                create_ar_stage_config(
-                    stage_id=stage_id,
-                    model_path=stage["model_path"],
-                    input_modalities=_normalize_modalities(
-                        stage.get("input_modalities"), ["text"]
-                    ),
-                    output_modalities=_normalize_modalities(
-                        stage.get("output_modalities"), ["text"]
-                    ),
-                    default_stage_args=default_stage_args,
-                )
-            )
-        elif engine_type == "DiT":
-            dit_cfg_dict: Dict = dict(stage.get("dit_config", {}))
-            if "num_inference_steps" not in dit_cfg_dict:
-                raise ValueError(
-                    "DiT stage requires 'num_inference_steps' in dit_config"
-                )
-            dit_cfg = DiTConfig(**dit_cfg_dict)
-
-            stage_configs.append(
-                create_dit_stage_config(
-                    stage_id=stage_id,
-                    model_path=stage["model_path"],
-                    input_modalities=_normalize_modalities(
-                        stage.get("input_modalities"), ["text"]
-                    ),
-                    output_modalities=_normalize_modalities(
-                        stage.get("output_modalities"), ["image"]
-                    ),
-                    dit_config=dit_cfg,
-                    default_stage_args=default_stage_args,
-                )
-            )
-        else:
-            raise ValueError(f"Unsupported engine_type '{engine_type}' in config")
-
-    return stage_configs
-
-
-def _apply_env_overrides(stage_configs):
-    ar_override = os.environ.get("AR_MODEL")
-    dit_override = os.environ.get("DIT_MODEL")
-
-    for stage_config in stage_configs:
-        if ar_override and stage_config.engine_type == "AR":
-            stage_config.model_path = ar_override
-        if dit_override and stage_config.engine_type == "DiT":
-            stage_config.model_path = dit_override
-
-
-def main():
-    args = parse_args()
-
-    stage_configs = _load_stage_configs_from_yaml(args.config)
-    _apply_env_overrides(stage_configs)
-
-    omni = OmniLLM(stage_configs)
-
-    stage_overrides: Dict[int, Dict[str, object]] = {}
-    if args.seed is not None or args.negative_prompt:
-        for stage_config in stage_configs:
-            if stage_config.engine_type != "DiT":
-                continue
-            override = stage_overrides.setdefault(stage_config.stage_id, {})
-            if args.seed is not None:
-                override["seed"] = args.seed
-            if args.negative_prompt:
-                override["negative_prompt"] = args.negative_prompt
-
-    outputs = omni.generate(
-        prompt=args.prompt,
-        stage_overrides=stage_overrides if stage_overrides else None,
-    )
-
-    image = None
-    for request_output in outputs:
-        for completion in getattr(request_output, "outputs", []) or []:
-            candidate = getattr(completion, "image", None)
-            if candidate is not None:
-                image = candidate
-                break
-        if image is not None:
-            break
-
-    if image is None:
-        print("No image found in outputs.")
-        return
-
-    out_path = args.output.resolve()
-    try:
-        image.save(out_path)
-        print(f"Saved image to {out_path}")
-    except Exception:
-        print("Generated output is not a PIL.Image; skipping save.")
-
-
-if __name__ == "__main__":
-    main()

examples/omni/configs/ar_dit_local.yaml

@@ -1,28 +0,0 @@
-# Example config for running OmniLLM with local AR + DiT models.
-stages:
-  - stage_id: 0
-    engine_type: AR
-    model_path: Qwen/Qwen3-0.6B
-    input_modalities: [text]
-    output_modalities: [text]
-    default_stage_args:
-      max_tokens: 16
-      temperature: 0.7
-  - stage_id: 1
-    engine_type: DiT
-    model_path: stabilityai/stable-diffusion-2-1
-    input_modalities: [text]
-    output_modalities: [image]
-    default_stage_args:
-      height: 512
-      width: 512
-    dit_config:
-      num_inference_steps: 20
-      guidance_scale: 5.0
-      use_diffusers: true
-      diffusers_pipeline: auto
-      height: 512
-      width: 512
-      device_config:
-        device: mps
-        dtype: float16

pyproject.toml

@@ -1,139 +0,0 @@
-[build-system]
-requires = ["setuptools>=61.0", "wheel"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "vllm-omni"
-version = "0.1.0"
-description = "vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures"
-readme = "README.md"
-requires-python = ">=3.8"
-license = {text = "Apache-2.0"}
-authors = [
-    {name = "vLLM-omni Team", email = "hsliuustc@gmail.com"}
-]
-keywords = ["vllm", "multimodal", "diffusion", "transformer", "inference", "serving"]
-classifiers = [
-    "Development Status :: 3 - Alpha",
-    "Intended Audience :: Developers",
-    "Intended Audience :: Science/Research",
-    "License :: OSI Approved :: Apache Software License",
-    "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
-    "Topic :: Scientific/Engineering :: Artificial Intelligence",
-    "Topic :: Software Development :: Libraries :: Python Modules",
-]
-
-dependencies = [
-    "vllm>=0.2.0",
-    "torch>=2.0.0",
-    "transformers>=4.30.0",
-    "numpy>=1.21.0",
-    "pillow>=8.0.0",
-    "opencv-python>=4.5.0",
-    "PyYAML>=6.0",
-    "gradio>=3.0.0",
-    "fastapi>=0.100.0",
-    "uvicorn>=0.20.0",
-    "pydantic>=2.0.0",
-    "asyncio-mqtt>=0.11.0",
-    "ray>=2.0.0",
-]
-
-[project.optional-dependencies]
-dev = [
-    "pytest>=7.0.0",
-    "pytest-asyncio>=0.21.0",
-    "pytest-cov>=4.0.0",
-    "black>=23.0.0",
-    "isort>=5.12.0",
-    "flake8>=6.0.0",
-    "mypy>=1.0.0",
-    "pre-commit>=3.0.0",
-]
-docs = [
-    "sphinx>=6.0.0",
-    "sphinx-rtd-theme>=1.2.0",
-    "myst-parser>=1.0.0",
-]
-examples = [
-    "jupyter>=1.0.0",
-    "matplotlib>=3.5.0",
-    "seaborn>=0.11.0",
-]
-
-[project.urls]
-Homepage = "https://github.com/hsliuustc0106/vllm-omni"
-Repository = "https://github.com/hsliuustc0106/vllm-omni"
-Documentation = "https://vllm-omni.readthedocs.io"
-"Bug Tracker" = "https://github.com/hsliuustc0106/vllm-omni/issues"
-
-[project.scripts]
-vllm = "vllm_omni.entrypoints.cli.main:main"
-vllm-omni = "vllm_omni.entrypoints.cli.main:main"
-
-[tool.setuptools.packages.find]
-where = ["."]
-include = ["vllm_omni*"]
-
-[tool.black]
-line-length = 88
-target-version = ['py38', 'py39', 'py310', 'py311']
-include = '\.pyi?$'
-extend-exclude = '''
-/(
-  # directories
-  \.eggs
-  | \.git
-  | \.hg
-  | \.mypy_cache
-  | \.tox
-  | \.venv
-  | build
-  | dist
-)/
-'''
-
-[tool.isort]
-profile = "black"
-multi_line_output = 3
-line_length = 88
-known_first_party = ["vllm_omni"]
-
-[tool.mypy]
-python_version = "3.8"
-warn_return_any = true
-warn_unused_configs = true
-disallow_untyped_defs = true
-disallow_incomplete_defs = true
-check_untyped_defs = true
-disallow_untyped_decorators = true
-no_implicit_optional = true
-warn_redundant_casts = true
-warn_unused_ignores = true
-warn_no_return = true
-warn_unreachable = true
-strict_equality = true
-
-[tool.pytest.ini_options]
-testpaths = ["tests"]
-python_files = ["test_*.py", "*_test.py"]
-python_classes = ["Test*"]
-python_functions = ["test_*"]
-addopts = [
-    "--strict-markers",
-    "--strict-config",
-    "--cov=vllm_omni",
-    "--cov-report=term-missing",
-    "--cov-report=html",
-    "--cov-report=xml",
-]
-markers = [
-    "unit: Unit tests",
-    "integration: Integration tests",
-    "benchmark: Benchmark tests",
-    "slow: Slow tests",
-]

requirements.txt

@@ -1,27 +0,0 @@
-# Core dependencies
-vllm>=0.10.2
-torch>=2.7
-transformers>=4.30.0
-numpy>=1.21.0
-pillow>=8.0.0
-opencv-python>=4.5.0
-PyYAML>=6.0
-
-# API and serving
-fastapi>=0.100.0
-uvicorn>=0.20.0
-pydantic>=2.0.0
-gradio>=3.0.0
-
-# Async and distributed computing
-asyncio-mqtt>=0.11.0
-ray>=2.0.0
-
-# Development dependencies
-pytest>=7.0.0
-pytest-asyncio>=0.21.0
-pytest-cov>=4.0.0
-black>=23.0.0
-isort>=5.12.0
-flake8>=6.0.0
-mypy>=1.0.0

scripts/README.md

@@ -1,109 +0,0 @@
-# vLLM-omni Testing Scripts
-
-This directory contains automated testing scripts for vLLM-omni serving functionality.
-
-## Scripts Overview
-
-### 1. `test_serving.sh` - Comprehensive Test Suite
-Full-featured testing script that performs complete validation of the vLLM-omni serving functionality.
-
-**Features:**
-- Model existence validation
-- Environment setup verification
-- Import functionality testing
-- Server startup and health checks
-- Text generation testing
-- Performance benchmarking
-- API client integration testing
-- Comprehensive logging and error handling
-
-**Usage:**
-```bash
-# Use default model and port
-./scripts/test_serving.sh
-
-# Use specific model and port
-./scripts/test_serving.sh ./models/Qwen3-0.6B 8000
-
-# Use HuggingFace model
-./scripts/test_serving.sh Qwen/Qwen3-0.6B 8001
-
-# Show help
-./scripts/test_serving.sh --help
-```
-
-**Test Coverage:**
-- ✅ Model loading and server startup
-- ✅ Health and info endpoints
-- ✅ Text generation functionality
-- ✅ Performance metrics
-- ✅ API client integration
-- ✅ All imports working correctly
-
-### 2. `quick_test.sh` - Fast Validation
-Lightweight script for quick validation after making changes.
-
-**Features:**
-- Fast import testing
-- Basic server startup
-- Health endpoint validation
-- Simple text generation test
-- Retry mechanism for reliability
-
-**Usage:**
-```bash
-# Use default port (8000)
-./scripts/quick_test.sh
-
-# Use specific port
-./scripts/quick_test.sh 8001
-```
-
-**Test Coverage:**
-- ✅ Import functionality
-- ✅ Server startup
-- ✅ Health endpoint
-- ✅ Basic text generation
-
-## Prerequisites
-
-1. **Conda Environment**: Ensure `vllm_omni` environment is activated
-2. **Model Available**: Qwen3-0.6B model should be available in `./models/Qwen3-0.6B/`
-3. **Dependencies**: All vLLM-omni dependencies should be installed
-
-## Environment Setup
-
-```bash
-# Activate conda environment
-conda activate vllm_omni
-
-# Verify installation
-python -c "import vllm_omni; print('vLLM-omni ready')"
-```
-
-## Model Setup
-
-The scripts expect the Qwen3-0.6B model to be available. You can:
-
-1. **Use local model**: Place model in `./models/Qwen3-0.6B/`
-2. **Use HuggingFace model**: Pass `Qwen/Qwen3-0.6B` as model path
-3. **Download model**: Use the `download_models.py` script
-
-```bash
-# Download model using the provided script
-python scripts/download_models.py
-```
-
-## Output and Logging
-
-### Test Results
-- **Success**: Green `[SUCCESS]` messages
-- **Info**: Blue `[INFO]` messages  
-- **Warnings**: Yellow `[WARNING]` messages
-- **Errors**: Red `[ERROR]` messages
-
-### Log Files
-- `server.log`: Server startup and runtime logs
-- `api_client_test.log`: API client test results
-- `simple_usage_test.log`: Simple usage test results
-

scripts/test_serving.sh

@@ -1,363 +0,0 @@
-#!/bin/bash
-
-# vLLM-omni Serving Functionality Test Script
-# This script tests the complete serving functionality of vLLM-omni
-# Usage: ./scripts/test_serving.sh [model_path] [port]
-
-set -e  # Exit on any error
-
-# Configuration
-MODEL_PATH=${1:-"./models/Qwen3-0.6B"}
-PORT=${2:-8000}
-HOST="localhost"
-TIMEOUT=30
-SERVER_STARTUP_WAIT=15
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-# Logging functions
-log_info() {
-    echo -e "${BLUE}[INFO]${NC} $1"
-}
-
-log_success() {
-    echo -e "${GREEN}[SUCCESS]${NC} $1"
-}
-
-log_warning() {
-    echo -e "${YELLOW}[WARNING]${NC} $1"
-}
-
-log_error() {
-    echo -e "${RED}[ERROR]${NC} $1"
-}
-
-# Cleanup function
-cleanup() {
-    log_info "Cleaning up..."
-    if [ ! -z "$SERVER_PID" ]; then
-        log_info "Stopping server (PID: $SERVER_PID)..."
-        kill $SERVER_PID 2>/dev/null || true
-        wait $SERVER_PID 2>/dev/null || true
-    fi
-    # Kill any remaining vllm processes
-    pkill -f "vllm serve" 2>/dev/null || true
-    log_info "Cleanup completed"
-}
-
-# Set up trap for cleanup on exit
-trap cleanup EXIT
-
-# Check if model exists
-check_model() {
-    log_info "Checking if model exists at: $MODEL_PATH"
-    if [ ! -d "$MODEL_PATH" ]; then
-        log_error "Model directory not found: $MODEL_PATH"
-        log_info "Available models:"
-        ls -la models/ 2>/dev/null || log_warning "No models directory found"
-        exit 1
-    fi
-    log_success "Model found: $MODEL_PATH"
-}
-
-# Check if conda environment is activated
-check_environment() {
-    log_info "Checking conda environment..."
-    if [[ "$CONDA_DEFAULT_ENV" != "vllm_omni" ]]; then
-        log_warning "vllm_omni conda environment not activated"
-        log_info "Attempting to activate vllm_omni environment..."
-        source $(conda info --base)/etc/profile.d/conda.sh
-        conda activate vllm_omni
-        if [[ "$CONDA_DEFAULT_ENV" != "vllm_omni" ]]; then
-            log_error "Failed to activate vllm_omni environment"
-            exit 1
-        fi
-    fi
-    log_success "Environment check passed: $CONDA_DEFAULT_ENV"
-}
-
-# Test import functionality
-test_imports() {
-    log_info "Testing imports..."
-    python -c "
-import vllm_omni
-from vllm_omni.entrypoints.omni_llm import OmniLLM, AsyncOmniLLM
-print('✅ All imports successful')
-" || {
-        log_error "Import test failed"
-        exit 1
-    }
-    log_success "Import test passed"
-}
-
-# Start the server
-start_server() {
-    log_info "Starting vLLM-omni server on port $PORT..."
-    log_info "Command: vllm serve $MODEL_PATH --omni --port $PORT"
-    
-    # Start server in background
-    vllm serve "$MODEL_PATH" --omni --port "$PORT" > server.log 2>&1 &
-    SERVER_PID=$!
-    
-    log_info "Server started with PID: $SERVER_PID"
-    log_info "Waiting $SERVER_STARTUP_WAIT seconds for server to initialize..."
-    sleep $SERVER_STARTUP_WAIT
-    
-    # Check if server is still running
-    if ! kill -0 $SERVER_PID 2>/dev/null; then
-        log_error "Server failed to start"
-        log_info "Server logs:"
-        cat server.log
-        exit 1
-    fi
-    
-    log_success "Server appears to be running"
-}
-
-# Test health endpoint
-test_health() {
-    log_info "Testing health endpoint..."
-    for attempt in {1..10}; do
-        local response=$(curl -s -w "%{http_code}" http://$HOST:$PORT/health 2>/dev/null || echo "000")
-        local http_code="${response: -3}"
-        local body="${response%???}"
-
-        if [ "$http_code" = "200" ]; then
-            log_success "Health check passed: $body"
-            return 0
-        fi
-
-        log_warning "Health check attempt $attempt failed (HTTP $http_code). Retrying in 3s..."
-        sleep 3
-    done
-
-    log_error "Health check failed after multiple attempts"
-    return 1
-}
-
-# Test info endpoint
-test_info() {
-    log_info "Testing info endpoint..."
-    local response=$(curl -s -w "%{http_code}" http://$HOST:$PORT/info 2>/dev/null || echo "000")
-    local http_code="${response: -3}"
-    local body="${response%???}"
-    
-    if [ "$http_code" = "200" ]; then
-        log_success "Info endpoint working"
-        echo "$body" | python -m json.tool 2>/dev/null || log_warning "Info response not valid JSON"
-    else
-        log_error "Info endpoint failed (HTTP $http_code): $body"
-        return 1
-    fi
-}
-
-# Test text generation
-test_generation() {
-    log_info "Testing text generation..."
-    
-    # Test 1: Simple generation
-    local response=$(curl -s -X POST http://$HOST:$PORT/generate \
-        -H "Content-Type: application/json" \
-        -d '{"prompts": ["Test the server functionality"], "max_tokens": 20, "temperature": 0.7}' \
-        2>/dev/null || echo "{}")
-    
-    if echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); exit(0 if 'outputs' in data and len(data['outputs']) > 0 else 1)" 2>/dev/null; then
-        log_success "Text generation test passed"
-        echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); print('Generated text:', data['outputs'][0]['text'][:100] + '...' if len(data['outputs'][0]['text']) > 100 else data['outputs'][0]['text'])" 2>/dev/null
-    else
-        log_error "Text generation test failed"
-        echo "Response: $response"
-        return 1
-    fi
-}
-
-# Test API client example
-test_api_client() {
-    log_info "Testing API client example..."
-    if [ -f "examples/basic/api_client.py" ]; then
-        python examples/basic/api_client.py > api_client_test.log 2>&1 || {
-            log_error "API client test failed"
-            log_info "API client logs:"
-            cat api_client_test.log
-            return 1
-        }
-        log_success "API client test passed"
-    else
-        log_warning "API client example not found, skipping"
-    fi
-}
-
-# Test AR → DiT pipeline example
-test_ar_dit_pipeline() {
-    log_info "Testing AR → DiT diffusers pipeline example..."
-
-    if ! python - 2>/dev/null <<'PY'
-import importlib
-for pkg in ("torch", "diffusers"):
-    if importlib.util.find_spec(pkg) is None:
-        raise ImportError(pkg)
-PY
-    then
-        log_warning "Required packages for diffusers pipeline not found, skipping AR → DiT test"
-        return 0
-    fi
-
-    if [ ! -f "examples/omni/ar_dit_diffusers.py" ]; then
-        log_warning "AR → DiT example not found, skipping"
-        return 0
-    fi
-
-    local output_path="logs/ar_dit_pipeline.png"
-    local log_path="logs/ar_dit_pipeline.log"
-    rm -f "$output_path"
-
-    local cmd=(python examples/omni/ar_dit_diffusers.py
-        --prompt "Smoke test prompt of a lighthouse"
-        --seed 0
-        --output "$output_path")
-
-    local env_prefix=()
-    if [ -n "$AR_MODEL" ]; then
-        env_prefix+=("AR_MODEL=$AR_MODEL")
-    fi
-    if [ -n "$DIT_MODEL" ]; then
-        env_prefix+=("DIT_MODEL=$DIT_MODEL")
-    fi
-
-    if ! env "${env_prefix[@]}" "${cmd[@]}" >"$log_path" 2>&1; then
-        log_error "AR → DiT pipeline test failed"
-        log_info "AR → DiT logs:"
-        cat "$log_path"
-        return 1
-    fi
-
-    if [ ! -f "$output_path" ]; then
-        log_error "AR → DiT pipeline did not produce an image"
-        log_info "AR → DiT logs:"
-        cat "$log_path"
-        return 1
-    fi
-
-    log_success "AR → DiT pipeline test passed (output: $output_path)"
-}
-
-# Test simple usage example
-test_simple_usage() {
-    log_info "Testing simple usage example..."
-    if [ -f "examples/basic/simple_usage.py" ]; then
-        # Run with timeout to avoid hanging
-        timeout 60 python examples/basic/simple_usage.py > simple_usage_test.log 2>&1 || {
-            local exit_code=$?
-            if [ $exit_code -eq 124 ]; then
-                log_warning "Simple usage test timed out (60s), but this might be normal for model loading"
-            else
-                log_error "Simple usage test failed (exit code: $exit_code)"
-                log_info "Simple usage logs:"
-                cat simple_usage_test.log
-                return 1
-            fi
-        }
-        log_success "Simple usage test passed"
-    else
-        log_warning "Simple usage example not found, skipping"
-    fi
-}
-
-# Performance test
-test_performance() {
-    log_info "Running performance test..."
-    
-    local start_time=$(date +%s)
-    local response=$(curl -s -X POST http://$HOST:$PORT/generate \
-        -H "Content-Type: application/json" \
-        -d '{"prompts": ["Performance test prompt"], "max_tokens": 50, "temperature": 0.7}' \
-        2>/dev/null)
-    local end_time=$(date +%s)
-    local duration=$((end_time - start_time))
-    
-    if echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); exit(0 if 'outputs' in data else 1)" 2>/dev/null; then
-        log_success "Performance test passed (${duration}s response time)"
-    else
-        log_error "Performance test failed"
-        return 1
-    fi
-}
-
-# Main test function
-run_tests() {
-    log_info "Starting vLLM-omni serving functionality tests..."
-    log_info "Model: $MODEL_PATH"
-    log_info "Port: $PORT"
-    log_info "Host: $HOST"
-    echo "=========================================="
-    
-    # Run all tests
-    check_model
-    check_environment
-    test_imports
-    start_server
-    
-    # Wait a bit more for server to be fully ready
-    sleep 5
-    
-    test_health
-    test_info
-    test_generation
-    test_performance
-    test_api_client
-    test_ar_dit_pipeline
-
-    # Note: Simple usage test is commented out as it starts its own server
-    # test_simple_usage
-
-    log_success "All tests completed successfully!"
-    echo "=========================================="
-    log_info "Test Summary:"
-    log_info "✅ Model loading and server startup"
-    log_info "✅ Health and info endpoints"
-    log_info "✅ Text generation functionality"
-    log_info "✅ Performance metrics"
-    log_info "✅ API client integration"
-    log_info "✅ AR → DiT diffusers pipeline"
-    log_info "✅ All imports working correctly"
-}
-
-# Show usage
-show_usage() {
-    echo "Usage: $0 [model_path] [port]"
-    echo ""
-    echo "Arguments:"
-    echo "  model_path    Path to the model directory (default: ./models/Qwen3-0.6B)"
-    echo "  port          Port to run the server on (default: 8000)"
-    echo ""
-    echo "Examples:"
-    echo "  $0                                    # Use default model and port"
-    echo "  $0 ./models/Qwen3-0.6B 8001          # Use specific model and port"
-    echo "  $0 Qwen/Qwen3-0.6B 8000              # Use HuggingFace model"
-}
-
-# Main execution
-main() {
-    # Check for help flag
-    if [[ "$1" == "-h" || "$1" == "--help" ]]; then
-        show_usage
-        exit 0
-    fi
-    
-    # Change to script directory
-    cd "$(dirname "$0")/.."
-    
-    # Create logs directory if it doesn't exist
-    mkdir -p logs
-    
-    # Run tests
-    run_tests
-}
-
-# Run main function with all arguments
-main "$@"

tests/__init__.py

@@ -1,5 +0,0 @@
-"""
-Test suite for vLLM-omni.
-
-This package contains unit tests, integration tests, and benchmarks for vLLM-omni.
-"""

tests/conftest.py

@@ -1,82 +0,0 @@
-"""
-Shared fixtures and configuration for vLLM-omni tests.
-"""
-
-import pytest
-import torch
-from unittest.mock import Mock
-from vllm_omni.config import OmniStageConfig, DiTConfig, DiTCacheConfig, create_ar_stage_config, create_dit_stage_config
-
-
-@pytest.fixture(scope="session")
-def device():
-    """Get available device for testing."""
-    return "cuda" if torch.cuda.is_available() else "cpu"
-
-
-@pytest.fixture
-def sample_ar_stage_config():
-    """Sample AR stage configuration for testing."""
-    return create_ar_stage_config(
-        stage_id=0,
-        model_path="test-ar-model",
-        input_modalities=["text"],
-        output_modalities=["text"]
-    )
-
-
-@pytest.fixture
-def sample_dit_stage_config():
-    """Sample DiT stage configuration for testing."""
-    dit_config = DiTConfig(
-        model_type="dit",
-        scheduler_type="ddpm",
-        num_inference_steps=10,
-        guidance_scale=7.5
-    )
-    
-    return create_dit_stage_config(
-        stage_id=1,
-        model_path="test-dit-model",
-        input_modalities=["text"],
-        output_modalities=["image"],
-        dit_config=dit_config
-    )
-
-
-@pytest.fixture
-def sample_stage_configs(sample_ar_stage_config, sample_dit_stage_config):
-    """Sample stage configurations for testing."""
-    return [sample_ar_stage_config, sample_dit_stage_config]
-
-
-@pytest.fixture
-def mock_vllm_config():
-    """Mock vLLM configuration."""
-    config = Mock()
-    config.model = "test-model"
-    config.tensor_parallel_size = 1
-    config.pipeline_parallel_size = 1
-    return config
-
-
-@pytest.fixture
-def mock_dit_cache_config():
-    """Mock DiT cache configuration."""
-    from vllm_omni.config import DiTCacheTensor
-    
-    cache_tensors = [
-        DiTCacheTensor(
-            name="test_tensor",
-            shape=[1, 512, 512],
-            dtype="float32",
-            persistent=True
-        )
-    ]
-    
-    return DiTCacheConfig(
-        cache_tensors=cache_tensors,
-        max_cache_size=1024 * 1024 * 1024,  # 1GB
-        cache_strategy="fifo",
-        enable_optimization=True
-    )

tests/unit/test_config.py

@@ -1,179 +0,0 @@
-"""
-Unit tests for configuration modules.
-"""
-
-import pytest
-from vllm_omni.config import (
-    OmniStageConfig,
-    DiTConfig,
-    DiTCacheConfig,
-    DiTCacheTensor,
-    create_ar_stage_config,
-    create_dit_stage_config,
-)
-
-
-class TestOmniStageConfig:
-    def test_ar_stage_config_creation(self):
-        """Test AR stage configuration creation."""
-        config = OmniStageConfig(
-            stage_id=0,
-            engine_type="AR",
-            model_path="test-model",
-            input_modalities=["text"],
-            output_modalities=["text"]
-        )
-        assert config.engine_type == "AR"
-        assert config.stage_id == 0
-        assert config.model_path == "test-model"
-        assert config.input_modalities == ["text"]
-        assert config.output_modalities == ["text"]
-    
-    def test_dit_stage_config_creation(self):
-        """Test DiT stage configuration creation."""
-        dit_config = DiTConfig(
-            model_type="dit",
-            scheduler_type="ddpm",
-            num_inference_steps=50
-        )
-        config = OmniStageConfig(
-            stage_id=1,
-            engine_type="DiT",
-            model_path="test-dit-model",
-            input_modalities=["text"],
-            output_modalities=["image"],
-            dit_config=dit_config
-        )
-        assert config.engine_type == "DiT"
-        assert config.dit_config is not None
-        assert config.dit_config.model_type == "dit"
-    
-    def test_invalid_engine_type(self):
-        """Test validation of engine type."""
-        with pytest.raises(ValueError):
-            OmniStageConfig(
-                stage_id=0,
-                engine_type="INVALID",
-                model_path="test-model",
-                input_modalities=["text"],
-                output_modalities=["text"]
-            )
-    
-    def test_empty_modalities(self):
-        """Test validation of empty modalities."""
-        with pytest.raises(ValueError):
-            OmniStageConfig(
-                stage_id=0,
-                engine_type="AR",
-                model_path="test-model",
-                input_modalities=[],
-                output_modalities=["text"]
-            )
-        
-        with pytest.raises(ValueError):
-            OmniStageConfig(
-                stage_id=0,
-                engine_type="AR",
-                model_path="test-model",
-                input_modalities=["text"],
-                output_modalities=[]
-            )
-    
-    def test_dit_requires_config(self):
-        """Test that DiT engine requires dit_config."""
-        with pytest.raises(ValueError):
-            OmniStageConfig(
-                stage_id=1,
-                engine_type="DiT",
-                model_path="test-dit-model",
-                input_modalities=["text"],
-                output_modalities=["image"]
-            )
-
-
-class TestDiTConfig:
-    def test_dit_config_defaults(self):
-        """Test DiT configuration with defaults."""
-        config = DiTConfig(
-            model_type="dit",
-            scheduler_type="ddpm",
-            num_inference_steps=50
-        )
-        assert config.use_diffusers is False
-        assert config.diffusers_pipeline is None
-        assert config.guidance_scale == 7.5
-        assert config.height == 512
-        assert config.width == 512
-    
-    def test_diffusers_config(self):
-        """Test DiT configuration with diffusers."""
-        config = DiTConfig(
-            model_type="dit",
-            scheduler_type="ddpm",
-            num_inference_steps=50,
-            use_diffusers=True,
-            diffusers_pipeline="stable-diffusion"
-        )
-        assert config.use_diffusers is True
-        assert config.diffusers_pipeline == "stable-diffusion"
-
-
-class TestDiTCacheConfig:
-    def test_cache_config_creation(self):
-        """Test cache configuration creation."""
-        cache_tensors = [
-            DiTCacheTensor(
-                name="test_tensor",
-                shape=[1, 512, 512],
-                dtype="float32",
-                persistent=True
-            )
-        ]
-        
-        config = DiTCacheConfig(
-            cache_tensors=cache_tensors,
-            max_cache_size=1024 * 1024 * 1024,
-            cache_strategy="fifo"
-        )
-        
-        assert len(config.cache_tensors) == 1
-        assert config.max_cache_size == 1024 * 1024 * 1024
-        assert config.cache_strategy == "fifo"
-        assert config.enable_optimization is True
-
-
-class TestHelperFunctions:
-    def test_create_ar_stage_config(self):
-        """Test create_ar_stage_config helper function."""
-        config = create_ar_stage_config(
-            stage_id=0,
-            model_path="test-model"
-        )
-        
-        assert config.stage_id == 0
-        assert config.engine_type == "AR"
-        assert config.model_path == "test-model"
-        assert config.input_modalities == ["text"]
-        assert config.output_modalities == ["text"]
-    
-    def test_create_dit_stage_config(self):
-        """Test create_dit_stage_config helper function."""
-        dit_config = DiTConfig(
-            model_type="dit",
-            scheduler_type="ddpm",
-            num_inference_steps=50
-        )
-        
-        config = create_dit_stage_config(
-            stage_id=1,
-            model_path="test-dit-model",
-            dit_config=dit_config
-        )
-        
-        assert config.stage_id == 1
-        assert config.engine_type == "DiT"
-        assert config.model_path == "test-dit-model"
-        assert config.input_modalities == ["text"]
-        assert config.output_modalities == ["image"]
-        assert config.dit_config is not None
-

vllm_omni/__init__.py

@@ -1,45 +0,0 @@
-"""
-vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures.
-
-This package extends vLLM beyond traditional text-based, autoregressive generation
-to support multi-modality models with non-autoregressive structures and non-textual outputs.
-
-Architecture:
-- 🟡 Modified: vLLM components modified for multimodal support  
-- 🔴 Added: New components for multimodal and non-autoregressive processing
-"""
-
-__version__ = "0.1.0"
-__author__ = "vLLM-omni Team"
-__email__ = "hsliuustc@gmail.com"
-
-# Main entry points
-from .entrypoints.omni_llm import OmniLLM, AsyncOmniLLM
-from .config import (
-    OmniStageConfig,
-    DiTConfig,
-    DiTCacheConfig,
-    create_ar_stage_config,
-    create_dit_stage_config,
-)
-
-__all__ = [
-    # Version info
-    "__version__",
-    "__author__", 
-    "__email__",
-    
-    # Main components
-    "OmniLLM",
-    "AsyncOmniLLM",
-    
-    # Configuration
-    "OmniStageConfig",
-    "DiTConfig",
-    "DiTCacheConfig",
-    "create_ar_stage_config",
-    "create_dit_stage_config",
-    
-    # All other components are available through their respective modules
-    # processors.*, schedulers.*, executors.*, etc.
-]

vllm_omni/config/__init__.py

@@ -1,50 +0,0 @@
-"""
-Configuration module for vLLM-omni.
-"""
-from typing import Optional
-from pydantic.dataclasses import dataclass
-from pydantic import ConfigDict
-
-from vllm.config import ModelConfig
-from vllm.config import config
-
-import vllm_omni.model_executor.models as me_models
-
-from .stage_config import (
-    OmniStageConfig,
-    DiTConfig,
-    DiTCacheConfig,
-    DiTCacheTensor,
-    create_ar_stage_config,
-    create_dit_stage_config,
-)
-
-
-@config
-@dataclass(config=ConfigDict(arbitrary_types_allowed=True))
-class OmniModelConfig(ModelConfig):
-    """Configuration for Omni models, extending the base ModelConfig."""
-    
-    stage_id: int = 0
-    model_stage: str = "thinker"
-    model_arch: str = "Qwen2_5OmniForConditionalGeneration"
-    engine_output_type: Optional[str] = None
-    
-    @property
-    def registry(self):
-        return me_models.OmniModelRegistry
-    
-    @property
-    def architectures(self) -> list[str]:
-        return [self.model_arch]
-
-
-__all__ = [
-    "OmniModelConfig",
-    "OmniStageConfig",
-    "DiTConfig", 
-    "DiTCacheConfig",
-    "DiTCacheTensor",
-    "create_ar_stage_config",
-    "create_dit_stage_config",
-]

vllm_omni/config/stage_config.py

@@ -1,166 +0,0 @@
-"""
-Stage configuration for vLLM-omni multi-stage processing.
-"""
-
-from dataclasses import dataclass
-from typing import List, Optional, Type, Literal, Any, Dict
-from vllm.config import (
-    VllmConfig,
-    DeviceConfig,
-    LoadConfig,
-    ModelConfig,
-    CompilationConfig,
-)
-from vllm.executor.executor_base import ExecutorBase as Executor
-
-@dataclass
-class DiTCacheTensor:
-    """Configuration for DiT cache tensors."""
-    name: str
-    shape: List[int]
-    dtype: str = "float32"
-    persistent: bool = True
-
-
-@dataclass
-class DiTCacheConfig:
-    """Configuration for DiT caching system."""
-    cache_tensors: List[DiTCacheTensor]
-    max_cache_size: int = 1024 * 1024 * 1024  # 1GB
-    cache_strategy: str = "fifo"  # fifo, lru, lfu
-    enable_optimization: bool = True
-    cache_compression: bool = False
-
-
-@dataclass
-class DiTConfig:
-    """Configuration for DiT (Diffusion Transformer) stages."""
-
-    num_inference_steps: int
-    """Number of diffusion inference steps."""
-
-    model_config: Optional[ModelConfig] = None
-    """Model configuration."""
-    scheduler_config: Optional[Any] = None
-    """Scheduler configuration."""
-    device_config: Optional[DeviceConfig] = None
-    """Device configuration."""
-    load_config: Optional[LoadConfig] = None
-    """Model loading configuration."""
-    compilation_config: Optional[CompilationConfig] = None
-    """Compilation configuration."""
-    dit_cache_config: Optional[DiTCacheConfig] = None
-    """DiT cache configuration."""
-    guidance_scale: float = 7.5
-    use_diffusers: bool = False
-    diffusers_pipeline: Optional[str] = None
-    height: int = 512
-    width: int = 512
-    batch_size: int = 1
-
-
-@dataclass
-class OmniStageConfig:
-    """Configuration for a processing stage in vLLM-omni."""
-
-    stage_id: int
-    engine_type: Literal["AR", "DiT"]
-    model_path: str
-    input_modalities: List[str]
-    output_modalities: List[str]
-    vllm_config: Optional[VllmConfig] = None
-    dit_config: Optional[DiTConfig] = None
-    executor_class: Type[Executor] = None  # Will be set based on engine_type
-    default_stage_args: Optional[Dict[str, Any]] = None
-    stage_output: Optional[Any] = None
-    
-    def __post_init__(self):
-        """Validate configuration after initialization."""
-        if self.engine_type not in ["AR", "DiT"]:
-            raise ValueError(f"Invalid engine_type: {self.engine_type}. Must be 'AR' or 'DiT'")
-        
-        if self.engine_type == "DiT" and self.dit_config is None:
-            raise ValueError("DiT engine requires dit_config")
-        
-        if not self.input_modalities:
-            raise ValueError("input_modalities cannot be empty")
-        
-        if not self.output_modalities:
-            raise ValueError("output_modalities cannot be empty")
-    
-    def get_executor_class(self) -> Type[Executor]:
-        """Get the appropriate executor class for this stage."""
-        if self.executor_class is not None:
-            return self.executor_class
-        
-        if self.engine_type == "AR":
-            from vllm.executor.uniproc_executor import UniProcExecutor
-            return UniProcExecutor
-        elif self.engine_type == "DiT":
-            if self.dit_config and self.dit_config.use_diffusers:
-                from vllm_omni.executor.diffusers_executor import DiffusersPipelineExecutor
-                return DiffusersPipelineExecutor
-            else:
-                from vllm.executor.uniproc_executor import UniProcExecutor
-                return UniProcExecutor
-        
-        raise ValueError(f"No executor class available for engine_type: {self.engine_type}")
-
-
-def create_ar_stage_config(
-    stage_id: int,
-    model_path: str,
-    input_modalities: List[str] = None,
-    output_modalities: List[str] = None,
-    vllm_config: Optional[VllmConfig] = None,
-    executor_class: Optional[Type[Executor]] = None,
-    default_stage_args: Optional[Dict[str, Any]] = None,
-) -> OmniStageConfig:
-    """Create a configuration for an AR (Autoregressive) stage."""
-    if input_modalities is None:
-        input_modalities = ["text"]
-    if output_modalities is None:
-        output_modalities = ["text"]
-    
-    # For now, we'll create minimal configs
-    # vllm_config and executor_class will be handled by the LLM class
-    return OmniStageConfig(
-        stage_id=stage_id,
-        engine_type="AR",
-        model_path=model_path,
-        input_modalities=input_modalities,
-        output_modalities=output_modalities,
-        vllm_config=vllm_config,
-        executor_class=executor_class,
-        default_stage_args=default_stage_args,
-    )
-
-
-def create_dit_stage_config(
-    stage_id: int,
-    model_path: str,
-    input_modalities: Optional[List[str]] = None,
-    output_modalities: Optional[List[str]] = None,
-    dit_config: Optional[DiTConfig] = None,
-    executor_class: Optional[Type[Executor]] = None,
-    default_stage_args: Optional[Dict[str, Any]] = None,
-) -> OmniStageConfig:
-    """Create a configuration for a DiT (Diffusion Transformer) stage."""
-    if input_modalities is None:
-        input_modalities = ["text"]
-    if output_modalities is None:
-        output_modalities = ["image"]
-    
-    if dit_config is None:
-        dit_config = DiTConfig(num_inference_steps=50)
-
-    return OmniStageConfig(
-        stage_id=stage_id,
-        engine_type="DiT",
-        model_path=model_path,
-        input_modalities=input_modalities,
-        output_modalities=output_modalities,
-        dit_config=dit_config,
-        executor_class=executor_class,
-        default_stage_args=default_stage_args,
-    )

vllm_omni/core/dit_cache_manager.py

@@ -1,178 +0,0 @@
-"""
-DiT Cache Manager for vLLM-omni.
-
-This module provides caching functionality for DiT (Diffusion Transformer) models
-to optimize inference performance and memory usage.
-"""
-
-import time
-import torch
-from typing import Dict, List, Optional, Any
-from dataclasses import dataclass
-from ..config import DiTCacheConfig, DiTCacheTensor
-
-
-class DiTCacheManager:
-    """Manages DiT-specific caching for optimized inference."""
-    
-    def __init__(self, config: DiTCacheConfig):
-        self.config = config
-        self.cache_tensors: Dict[str, DiTCacheTensor] = {}
-        self.cache_groups: List[DiTCacheTensor] = config.cache_tensors
-        self.max_cache_size = config.max_cache_size
-        self.current_cache_size = 0
-        self.cache_hits = 0
-        self.cache_misses = 0
-        self.cache_timeout = 3600  # 1 hour default timeout
-    
-    def allocate_cache(self, request_id: str, size: int) -> torch.Tensor:
-        """Allocate cache for a specific request."""
-        # Check if we have enough space
-        if self.current_cache_size + size > self.max_cache_size:
-            self._evict_cache()
-        
-        # Create tensor
-        tensor = torch.zeros(size, dtype=torch.float32)
-        
-        # Store in cache
-        cache_tensor = DiTCacheTensor(
-            name=request_id,
-            tensor=tensor,
-            timestamp=time.time(),
-            access_count=0,
-            size_bytes=size * 4  # Assuming float32
-        )
-        
-        self.cache_tensors[request_id] = cache_tensor
-        self.current_cache_size += cache_tensor.size_bytes
-        
-        return tensor
-    
-    def get_cache(self, request_id: str) -> Optional[torch.Tensor]:
-        """Get cached tensor for a request."""
-        if request_id in self.cache_tensors:
-            cache_tensor = self.cache_tensors[request_id]
-            cache_tensor.access_count += 1
-            self.cache_hits += 1
-            return cache_tensor.tensor
-        else:
-            self.cache_misses += 1
-            return None
-    
-    def store_cache(self, request_id: str, tensor: torch.Tensor) -> None:
-        """Store a tensor in the cache."""
-        if request_id in self.cache_tensors:
-            # Update existing cache
-            old_tensor = self.cache_tensors[request_id]
-            self.current_cache_size -= old_tensor.size_bytes
-            
-            new_size_bytes = tensor.numel() * 4  # Assuming float32
-            self.current_cache_size += new_size_bytes
-            
-            self.cache_tensors[request_id] = DiTCacheTensor(
-                name=request_id,
-                tensor=tensor,
-                timestamp=time.time(),
-                access_count=old_tensor.access_count,
-                size_bytes=new_size_bytes
-            )
-        else:
-            # Create new cache entry
-            new_size_bytes = tensor.numel() * 4
-            if self.current_cache_size + new_size_bytes > self.max_cache_size:
-                self._evict_cache()
-            
-            self.cache_tensors[request_id] = DiTCacheTensor(
-                name=request_id,
-                tensor=tensor,
-                timestamp=time.time(),
-                access_count=0,
-                size_bytes=new_size_bytes
-            )
-            self.current_cache_size += new_size_bytes
-    
-    def release_cache(self, request_id: str) -> None:
-        """Release cache for a request."""
-        if request_id in self.cache_tensors:
-            cache_tensor = self.cache_tensors[request_id]
-            self.current_cache_size -= cache_tensor.size_bytes
-            del self.cache_tensors[request_id]
-    
-    def clear_expired_cache(self) -> None:
-        """Clear expired cache entries."""
-        current_time = time.time()
-        expired_keys = []
-        
-        for request_id, cache_tensor in self.cache_tensors.items():
-            if current_time - cache_tensor.timestamp > self.cache_timeout:
-                expired_keys.append(request_id)
-        
-        for request_id in expired_keys:
-            self.release_cache(request_id)
-    
-    def _evict_cache(self) -> None:
-        """Evict cache entries based on strategy."""
-        if self.config.cache_strategy == "fifo":
-            self._evict_fifo()
-        elif self.config.cache_strategy == "lru":
-            self._evict_lru()
-        elif self.config.cache_strategy == "lfu":
-            self._evict_lfu()
-        else:
-            self._evict_fifo()  # Default to FIFO
-    
-    def _evict_fifo(self) -> None:
-        """Evict cache entries using FIFO strategy."""
-        if not self.cache_tensors:
-            return
-        
-        # Find oldest entry
-        oldest_key = min(self.cache_tensors.keys(), 
-                        key=lambda k: self.cache_tensors[k].timestamp)
-        self.release_cache(oldest_key)
-    
-    def _evict_lru(self) -> None:
-        """Evict cache entries using LRU strategy."""
-        if not self.cache_tensors:
-            return
-        
-        # Find least recently used entry
-        lru_key = min(self.cache_tensors.keys(),
-                     key=lambda k: self.cache_tensors[k].access_count)
-        self.release_cache(lru_key)
-    
-    def _evict_lfu(self) -> None:
-        """Evict cache entries using LFU strategy."""
-        if not self.cache_tensors:
-            return
-        
-        # Find least frequently used entry
-        lfu_key = min(self.cache_tensors.keys(),
-                     key=lambda k: self.cache_tensors[k].access_count)
-        self.release_cache(lfu_key)
-    
-    def get_statistics(self) -> Dict[str, Any]:
-        """Get cache statistics."""
-        total_requests = self.cache_hits + self.cache_misses
-        hit_rate = self.cache_hits / total_requests if total_requests > 0 else 0
-        
-        return {
-            "cache_size": self.current_cache_size,
-            "max_cache_size": self.max_cache_size,
-            "cache_utilization": self.current_cache_size / self.max_cache_size,
-            "cache_hits": self.cache_hits,
-            "cache_misses": self.cache_misses,
-            "hit_rate": hit_rate,
-            "num_cached_tensors": len(self.cache_tensors)
-        }
-    
-    def clear_all_cache(self) -> None:
-        """Clear all cache entries."""
-        self.cache_tensors.clear()
-        self.current_cache_size = 0
-        self.cache_hits = 0
-        self.cache_misses = 0
-    
-    def set_cache_timeout(self, timeout: float) -> None:
-        """Set cache timeout in seconds."""
-        self.cache_timeout = timeout

vllm_omni/core/sched/scheduler.py

@@ -1,40 +0,0 @@
-
-from vllm_omni.request import OmniRequest
-from typing import List
-from threading import Lock, Condition
-from vllm_omni.config import OmniConfig
-
-from vllm.v1.kv_cache_interface import KVCacheConfig
-from vllm.v1.core.sched import SchedulerInterface
-from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry
-from vllm.v1.structured_output import StructuredOutputManager
-
-
-class OmniScheduler(SchedulerInterface):
-    """
-    OmniScheduler: Scheduler for vLLM-omni multimodal processing.
-
-    This scheduler extends vLLM's scheduler to support multimodal and non-autoregressive
-    processing with additional fields and methods specific to vLLM-omni.
-    """
-    def __init__(self,
-        omni_config: OmniConfig,
-        kv_cache_config: KVCacheConfig,
-        structured_output_manager: StructuredOutputManager,
-        mm_registry: MultiModalRegistry = MULTIMODAL_REGISTRY,
-        include_finished_set: bool = False,
-        log_stats: bool = False,
-    ):
-        super().__init__(
-            vllm_config=omni_config.vllm_config,
-            kv_cache_config=kv_cache_config,
-            multimodal_registry=mm_registry,
-            structured_output_manager=structured_output_manager,
-            include_finished_set=include_finished_set,
-            log_stats=log_stats,
-        )
-        self.omni_config = omni_config
-            
-    def schedule(self, requests: List[OmniRequest]) -> List[OmniRequest]:
-        # TODO: Implement scheduling logic
-        pass

vllm_omni/engine/__init__.py

@@ -1,9 +0,0 @@
-"""
-Engine components for vLLM-omni.
-"""
-
-from .output_processor import MultimodalOutputProcessor
-
-__all__ = [
-    "MultimodalOutputProcessor",
-]

vllm_omni/engine/arg_utils.py

@@ -1,51 +0,0 @@
-from typing import Optional
-from dataclasses import dataclass
-from vllm.engine.arg_utils import EngineArgs
-from vllm.utils import FlexibleArgumentParser
-
-from vllm_omni.config import OmniModelConfig
-
-
-@dataclass
-class OmniEngineArgs(EngineArgs):
-    stage_id: int = 0
-    model_stage: str = "thinker"
-    model_arch: str = "Qwen2_5OmniForConditionalGeneration"
-    engine_output_type: Optional[str] = None
-
-    @staticmethod
-    def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
-        """Shared CLI arguments for vLLM engine."""
-        parser.add_argument(
-            "--engine-output-type",
-            type=str,
-            default=EngineArgs.engine_output_type,
-            help=(
-                "Declare EngineCoreOutput.output_type (e.g., 'text', 'image', "
-                "'text+image', 'latent'). This will be written into "
-                "model_config.engine_output_type for schedulers to use."
-            ),
-        )
-        parser.add_argument("--model-stage", type=str, default=OmniEngineArgs.model_stage, 
-        help="Declare model stage (e.g., 'thinker', 'talker', 'token2wav'). This will be written into model_config.model_stage for schedulers to use.")
-        return parser
-
-    def create_model_config(self) -> OmniModelConfig:
-        # First, get the base ModelConfig from the parent class
-        base_config = super().create_model_config()
-        
-        # Create OmniModelConfig by copying all base config attributes
-        # and adding the new omni-specific fields
-        config_dict = base_config.__dict__.copy()
-        
-        # Add the new omni-specific fields
-        config_dict['stage_id'] = self.stage_id
-        config_dict['model_stage'] = self.model_stage
-        config_dict['model_arch'] = self.model_arch
-        config_dict['engine_output_type'] = self.engine_output_type
-        
-        # Create and return the OmniModelConfig instance
-        omni_config = OmniModelConfig(**config_dict)
-        omni_config.hf_config.architectures = omni_config.architectures
-        
-        return omni_config

vllm_omni/engine/diffusion_engine.py

@@ -1,133 +0,0 @@
-"""Diffusion engine adapters for diffusers-backed DiT stages."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from ..config import DiTConfig
-from ..worker.gpu_diffusion_worker import (
-    DiffusionGPUWorker,
-    DiffusionRunnerOutput,
-)
-
-
-class DiffusionEngine:
-    """Base diffusion engine storing shared DiT configuration."""
-
-    def __init__(
-        self,
-        dit_config: DiTConfig,
-        model_path: Optional[str] = None,
-        log_stats: bool = False,
-        multiprocess_mode: bool = False,
-    ) -> None:
-        self.dit_config = dit_config
-
-        model_cfg = getattr(dit_config, "model_config", None)
-        config_model_path = None
-        if model_cfg is not None:
-            if isinstance(model_cfg, dict):
-                config_model_path = (
-                    model_cfg.get("model")
-                    or model_cfg.get("model_path")
-                )
-            else:
-                config_model_path = (
-                    getattr(model_cfg, "model", None)
-                    or getattr(model_cfg, "model_path", None)
-                )
-
-        self.model_path = model_path or config_model_path
-        self.model_config = dit_config.model_config
-        self.cache_config = dit_config.dit_cache_config
-
-        self.height = dit_config.height
-        self.width = dit_config.width
-        self.num_inference_steps = dit_config.num_inference_steps
-        self.guidance_scale = dit_config.guidance_scale
-
-        self.log_stats = log_stats
-        self.multiprocess_mode = multiprocess_mode
-
-    def generate(self, *args, **kwargs):  # pragma: no cover - placeholder
-        raise NotImplementedError("This method should be implemented in subclasses.")
-
-
-class DiffusersPipelineEngine(DiffusionEngine):
-    """Adapter that invokes a diffusers pipeline via DiffusionGPUWorker."""
-
-    def __init__(
-        self,
-        dit_config: DiTConfig,
-        model_path: Optional[str] = None,
-        log_stats: bool = False,
-        multiprocess_mode: bool = False,
-    ) -> None:
-        super().__init__(dit_config, model_path, log_stats, multiprocess_mode)
-
-        resolved_model_path = self.model_path
-        if resolved_model_path is None:
-            raise ValueError("Model path must be provided for DiffusersPipelineEngine.")
-
-        device_cfg = getattr(dit_config, "device_config", None)
-        model_cfg = getattr(dit_config, "model_config", None)
-
-        device = None
-        dtype = None
-
-        if device_cfg:
-            if isinstance(device_cfg, dict):
-                device = device_cfg.get("device")
-                dtype = device_cfg.get("dtype")
-            else:
-                device = getattr(device_cfg, "device", None)
-                dtype = getattr(device_cfg, "dtype", None)
-
-        if dtype is None and model_cfg:
-            if isinstance(model_cfg, dict):
-                dtype = model_cfg.get("dtype")
-            else:
-                dtype = getattr(model_cfg, "dtype", None)
-
-        self.worker = DiffusionGPUWorker(
-            resolved_model_path,
-            pipeline_name=dit_config.diffusers_pipeline,
-            device=device,
-            dtype=dtype,
-        )
-
-    def generate(
-        self,
-        prompt: str,
-        *,
-        height: Optional[int] = None,
-        width: Optional[int] = None,
-        num_inference_steps: Optional[int] = None,
-        guidance_scale: Optional[float] = None,
-        negative_prompt: Optional[str] = None,
-        seed: Optional[int] = None,
-        image: Optional[Any] = None,
-    ) -> DiffusionRunnerOutput:
-        height = int(height) if height is not None else self.height
-        width = int(width) if width is not None else self.width
-        num_steps = (
-            int(num_inference_steps)
-            if num_inference_steps is not None
-            else self.num_inference_steps
-        )
-        guidance = (
-            float(guidance_scale)
-            if guidance_scale is not None
-            else self.guidance_scale
-        )
-
-        return self.worker.generate(
-            prompt=prompt,
-            height=height,
-            width=width,
-            num_inference_steps=num_steps,
-            guidance_scale=guidance,
-            negative_prompt=negative_prompt,
-            seed=seed,
-            image=image,
-        )

vllm_omni/engine/output_processor.py

@@ -1,411 +0,0 @@
-from typing import Dict, Callable, Optional, Any, Union
-
-import torch
-
-from vllm.v1.engine.output_processor import OutputProcessor as VLLMOutputProcessor
-from vllm.v1.engine.output_processor import OutputProcessorOutput, RequestState, RequestOutputCollector
-from vllm.transformers_utils.tokenizer_group import TokenizerGroup
-from vllm.v1.engine import FinishReason
-from vllm.v1.metrics.stats import IterationStats
-from vllm.sampling_params import RequestOutputKind
-from vllm.transformers_utils.tokenizer import AnyTokenizer
-from vllm.v1.engine import EngineCoreRequest
-from vllm.v1.engine.detokenizer import IncrementalDetokenizer
-from vllm.v1.engine.logprobs import LogprobsProcessor
-from vllm.v1.engine.parallel_sampling import ParentRequest
-from vllm.logger import init_logger
-from vllm.v1.engine import EngineCoreOutput
-
-
-logger = init_logger(__name__)
-
-
-class OmniRequestState(RequestState):
-
-    def __init__(
-        self,
-        *args,
-        **kwargs,
-    ):
-        super().__init__(*args, **kwargs)
-        self.mm_type: Optional[str] = None
-        self.mm_accumulated: Optional[torch.Tensor] = None
-
-    @classmethod
-    def from_new_request(
-        cls,
-        tokenizer: AnyTokenizer,
-        request: EngineCoreRequest,
-        prompt: Optional[str],
-        parent_req: Optional[ParentRequest],
-        request_index: int,
-        queue: Optional[Any],
-        log_stats: bool,
-    ) -> "OmniRequestState":
-
-        if sampling_params := request.sampling_params:
-            if not sampling_params.detokenize:
-                tokenizer = None
-            output_kind = sampling_params.output_kind
-            logprobs_processor = LogprobsProcessor.from_new_request(
-                tokenizer=tokenizer,
-                request=request,
-            )
-            detokenizer = IncrementalDetokenizer.from_new_request(
-                tokenizer=tokenizer,
-                request=request,
-            )
-            max_tokens_param = sampling_params.max_tokens
-        else:
-            logprobs_processor = None
-            detokenizer = None
-            max_tokens_param = None
-            assert request.pooling_params is not None
-            output_kind = request.pooling_params.output_kind
-
-        return cls(
-            request_id=request.request_id,
-            parent_req=parent_req,
-            request_index=request_index,
-            lora_name=(request.lora_request.name
-                       if request.lora_request is not None else None),
-            output_kind=output_kind,
-            prompt=prompt,
-            prompt_token_ids=request.prompt_token_ids,
-            logprobs_processor=logprobs_processor,
-            detokenizer=detokenizer,
-            max_tokens_param=max_tokens_param,
-            arrival_time=request.arrival_time,
-            queue=queue,
-            log_stats=log_stats,
-        )
-
-    def add_multimodal_tensor(self, tensor: Optional[torch.Tensor],
-                               mm_type: Optional[str]) -> None:
-        if tensor is None:
-            return
-        try:
-            if mm_type:
-                self.mm_type = (mm_type or "").lower()
-            t = tensor.detach()
-            try:
-                t = t.to("cpu")
-            except Exception:
-                # Best-effort CPU move; keep original device if conversion fails
-                logger.debug("Failed to move multimodal tensor to CPU", exc_info=True)
-            if self.mm_accumulated is None:
-                self.mm_accumulated = t
-            else:
-                self.mm_accumulated = torch.cat([self.mm_accumulated, t], dim=0)
-        except Exception:
-            # Log and continue without crashing the output pipeline
-            logger.exception("Error accumulating multimodal tensor")
-
-    # Override: do not route to pooling-only path; always create completion
-    # outputs, and attach pooling_result into the CompletionOutput.
-    def make_request_output(
-        self,
-        new_token_ids: list[int],
-        pooling_output: Optional[torch.Tensor],
-        finish_reason: Optional[FinishReason],
-        stop_reason: Optional[Union[int, str]],
-        kv_transfer_params: Optional[dict[str, Any]] = None,
-        num_cached_tokens: Optional[int] = None,
-    ) -> Optional[Any]:
-        finished = finish_reason is not None
-        final_only = self.output_kind == RequestOutputKind.FINAL_ONLY
-
-        if not finished and final_only:
-            return None
-
-        if num_cached_tokens is not None:
-            # Keep num_cached_tokens in RequestOutput for compatibility
-            try:
-                self.num_cached_tokens = num_cached_tokens  # type: ignore[attr-defined]
-            except Exception:
-                pass
-
-        request_id = self.request_id
-        output = self._new_completion_output(new_token_ids, finish_reason,
-                                             stop_reason)
-
-        if self.parent_req is None:
-            outputs = [output]
-        else:
-            request_id, outputs, finished = self.parent_req.get_outputs(
-                request_id, output)
-            if not outputs:
-                return None
-
-        return self._new_request_output(request_id, outputs, finished,
-                                        kv_transfer_params)
-
-    def _new_completion_output(
-        self,
-        token_ids: list[int],
-        finish_reason: Optional[FinishReason],
-        stop_reason: Optional[Union[int, str]]
-    ) -> Any:
-        # Reuse base text/logprobs logic, then annotate with pooling_result.
-        base_output = super()._new_completion_output(token_ids, finish_reason,
-                                                     stop_reason)
-        try:
-            if self.mm_accumulated is not None:
-                tensor = self.mm_accumulated
-                try:
-                    tensor = tensor.detach().to("cpu")
-                except Exception:
-                    logger.debug("Failed to move accumulated multimodal tensor to CPU", exc_info=True)
-                # Attach on the completion output for downstream consumers.
-                if not hasattr(base_output, "multimodal_output"):
-                    setattr(base_output, "multimodal_output", {})
-                setattr(base_output, "multimodal_output", {self.mm_type: tensor})
-        except Exception:
-            logger.exception("Error in _new_completion_output")
-        return base_output
-
-
-class MultimodalOutputProcessor(VLLMOutputProcessor):
-    """Handles multimodal output processing by normalizing EngineCoreOutput
-    before delegating to the base vLLM OutputProcessor.
-
-    Strategy:
-    - Route by EngineCoreOutput.output_type when present
-      ("image", "text+image", "latents", "text").
-    - Fallback to pooling/text heuristics when output_type is absent.
-    - Mutate EngineCoreOutput in-place to ensure vLLM's base processor can
-      produce the correct RequestOutput/PoolingRequestOutput.
-    - Allow custom per-modality handlers via register_handler().
-    """
-    def __init__(self, tokenizer: TokenizerGroup, log_stats: bool, engine_core_output_type: Optional[str] = None):
-        super().__init__(tokenizer=tokenizer, log_stats=log_stats)
-        self.output_handlers: Dict[str, Callable[[EngineCoreOutput], None]] = {}
-        self._reqid_to_mm_type: Dict[str, str] = {}
-        self.request_states: dict[str, OmniRequestState] = {}
-        self.engine_core_output_type = engine_core_output_type
-        
-    def register_handler(self, modality: str,
-                         handler: Callable[[EngineCoreOutput], None]) -> None:
-        self.output_handlers[modality.lower()] = handler
-    
-    def add_request(
-        self,
-        request: EngineCoreRequest,
-        prompt: Optional[str],
-        parent_req: Optional[ParentRequest] = None,
-        request_index: int = 0,
-        queue: Optional[RequestOutputCollector] = None,
-    ) -> None:
-        request_id = request.request_id
-        if request_id in self.request_states:
-            raise ValueError(f"Request id {request_id} already running.")
-
-        tokenizer = None if not self.tokenizer else \
-            self.tokenizer.get_lora_tokenizer(request.lora_request)
-
-        req_state = OmniRequestState.from_new_request(tokenizer=tokenizer,
-                                                  request=request,
-                                                  prompt=prompt,
-                                                  parent_req=parent_req,
-                                                  request_index=request_index,
-                                                  queue=queue,
-                                                  log_stats=self.log_stats)
-        self.request_states[request_id] = req_state
-        self.lora_states.add_request(req_state)
-        if parent_req:
-            self.parent_requests[parent_req.request_id] = parent_req
-    
-    def process_outputs(
-        self,
-        engine_core_outputs: list[EngineCoreOutput],
-        engine_core_timestamp: Optional[float] = None,
-        iteration_stats: Optional[IterationStats] = None,
-    ) -> OutputProcessorOutput:
-        self._reqid_to_mm_type.clear()
-        for eco in engine_core_outputs:
-            mm_type = (getattr(eco, "output_type", None) or "").lower()
-            if mm_type:
-                self._reqid_to_mm_type[eco.request_id] = mm_type
-            self._route_and_normalize(eco)
-
-        # Build RequestOutputs without delegating to base, so we can keep ids
-        request_outputs: list[Any] = []
-        reqs_to_abort: list[str] = []
-        for eco in engine_core_outputs:
-            req_id = eco.request_id
-            req_state = self.request_states.get(req_id)
-            if req_state is None:
-                continue
-
-            # 1) Stats
-            self._update_stats_from_output(req_state, eco,
-                                           engine_core_timestamp,
-                                           iteration_stats)
-
-            new_token_ids = eco.new_token_ids
-            pooling_output = eco.pooling_output
-            finish_reason = eco.finish_reason
-            stop_reason = eco.stop_reason
-            kv_transfer_params = eco.kv_transfer_params
-            num_cached_tokens = eco.num_cached_tokens
-            req_state.is_prefilling = False
-
-            # 2) Detokenize and logprobs when text path
-            assert req_state.detokenizer is not None
-            assert req_state.logprobs_processor is not None
-            stop_string = req_state.detokenizer.update(
-                new_token_ids, finish_reason == FinishReason.STOP)
-            if stop_string:
-                finish_reason = FinishReason.STOP
-                stop_reason = stop_string
-            req_state.logprobs_processor.update_from_output(eco)
-
-            # 2.5) Accumulate multimodal tensors in RequestState
-            try:
-                mm_type = (getattr(eco, "output_type", self.engine_core_output_type) or "").lower()
-                if pooling_output is not None and isinstance(req_state, OmniRequestState):
-                    req_state.add_multimodal_tensor(pooling_output, mm_type)
-            except Exception:
-                logger.debug("Failed to accumulate multimodal tensor for request %s", req_id, exc_info=True)
-
-            # 3) Create RequestOutput objects, forcing combined mode to keep ids
-            pooling_for_make = pooling_output
-            if pooling_output is not None and new_token_ids:
-                # Do not consume pooling path now; keep ids and attach mm later
-                pooling_for_make = None
-
-            ro = req_state.make_request_output(new_token_ids, pooling_for_make,
-                                               finish_reason, stop_reason,
-                                               kv_transfer_params,
-                                               num_cached_tokens)
-            if ro:
-                # Attach accumulated multimodal payload if any
-                try:
-                    if isinstance(req_state, OmniRequestState) and req_state.mm_accumulated is not None:
-                        mm_key = req_state.mm_type or "latents"
-                        if not hasattr(ro, "multimodal_output"):
-                            setattr(ro, "multimodal_output", {})
-                        ro.multimodal_output[mm_key] = req_state.mm_accumulated
-                except Exception:
-                    logger.exception("Error attaching multimodal payload in process_outputs")
-                if req_state.queue is not None:
-                    req_state.queue.put(ro)
-                else:
-                    request_outputs.append(ro)
-
-            # 4) Free completed
-            if finish_reason is not None:
-                self.request_states.pop(req_id)
-                parent_req = req_state.parent_req
-                if parent_req and not parent_req.child_requests:
-                    self.parent_requests.pop(parent_req.request_id, None)
-                if not eco.finished:
-                    reqs_to_abort.append(req_id)
-                self._update_stats_from_finished(req_state, finish_reason,
-                                                 iteration_stats)
-                # Cleanup per-request mm state
-                if isinstance(req_state, OmniRequestState):
-                    req_state.mm_accumulated = None
-                    req_state.mm_type = None
-
-        return OutputProcessorOutput(
-            request_outputs=request_outputs,
-            reqs_to_abort=reqs_to_abort,
-        )
-    
-    # ---- routing helpers ----
-    def _route_and_normalize(self, eco: EngineCoreOutput) -> None:
-        output_type = (getattr(eco, "output_type", self.engine_core_output_type) or "").lower()
-
-        # Custom handler first (if registered)
-        if output_type in self.output_handlers:
-            try:
-                self.output_handlers[output_type](eco)
-                # Fall through to default fixups in case the handler left gaps
-            except Exception:
-                logger.exception("Error in custom output handler for %s", output_type)
-
-        if output_type == "image":
-            self._process_image_output(eco)
-        elif output_type in ("text+image", "text,image", "image+text"):
-            self._process_text_image_output(eco)
-        elif output_type in ("latents", "latent"):
-            self._process_latents_output(eco)
-        elif output_type in ("audio", "speech"):
-            self._process_audio_output(eco)
-        elif output_type == "text":
-            self._process_text_output(eco)
-        else:
-            # Fallback heuristic
-            if eco.pooling_output is not None:
-                self._process_pooling_output(eco)
-            else:
-                self._process_text_output(eco)
-
-    # ---- modality processors ----
-    def _process_image_output(self, eco: EngineCoreOutput) -> None:
-        """Ensure image tensors are surfaced via pooling_output for vLLM."""
-        if eco.pooling_output is None:
-            tensor = self._extract_from_multimodal_outputs(
-                eco, keys=("image", "images", "pixel_values", "pixels"))
-            if tensor is not None:
-                eco.pooling_output = tensor
-
-    def _process_text_image_output(self, eco: EngineCoreOutput) -> None:
-        """Allow text+image outputs. Text path stays as new_token_ids;
-        image/latents route via pooling_output."""
-        # Preserve text tokens as-is; ensure pooling_output carries image/latents
-        if eco.pooling_output is None:
-            tensor = self._extract_from_multimodal_outputs(
-                eco, keys=("image", "images", "pixel_values", "pixels",
-                           "latent", "latents", "z"))
-            if tensor is not None:
-                eco.pooling_output = tensor
-
-    def _process_latents_output(self, eco: EngineCoreOutput) -> None:
-        """Ensure latent tensors are surfaced via pooling_output."""
-        if eco.pooling_output is None:
-            tensor = self._extract_from_multimodal_outputs(
-                eco, keys=("latent", "latents", "z", "posterior"))
-            if tensor is not None:
-                eco.pooling_output = tensor
-
-    def _process_audio_output(self, eco: EngineCoreOutput) -> None:
-        """Ensure audio tensors are surfaced via pooling_output."""
-        if eco.pooling_output is None:
-            tensor = self._extract_from_multimodal_outputs(
-                eco, keys=("audio", "audios", "wav", "waveform",
-                           "audio_pcm", "pcm"))
-            if tensor is not None:
-                eco.pooling_output = tensor
-
-    def _process_text_output(self, eco: EngineCoreOutput) -> None:
-        """No-op; base processor will detokenize new_token_ids → text."""
-        return
-
-    def _process_pooling_output(self, eco: EngineCoreOutput) -> None:
-        """Optional sanity checks for pooling tensor."""
-        if eco.pooling_output is None:
-            return
-        if not isinstance(eco.pooling_output, torch.Tensor):
-            # Best-effort: convert to tensor if it's a list/ndarray-like
-            try:
-                eco.pooling_output = torch.as_tensor(eco.pooling_output)
-            except Exception:
-                pass
-
-    def _extract_from_multimodal_outputs(
-        self, eco: EngineCoreOutput, keys: tuple[str, ...]
-    ) -> Optional[torch.Tensor]:
-        mm = getattr(eco, "multimodal_outputs", None)
-        if not isinstance(mm, dict):
-            return None
-        for k in keys:
-            v = mm.get(k)
-            if isinstance(v, torch.Tensor):
-                return v
-        # Try the first tensor in the dict as a fallback
-        for v in mm.values():
-            if isinstance(v, torch.Tensor):
-                return v
-        return None

vllm_omni/entrypoints/api_server.py

@@ -1,146 +0,0 @@
-"""
-API server for vLLM-omni.
-"""
-
-import asyncio
-from typing import Dict, Any, List
-from fastapi import FastAPI, HTTPException
-from fastapi.responses import JSONResponse
-from pydantic import BaseModel
-import uvicorn
-
-from .omni_llm import AsyncOmniLLM
-
-
-class GenerateRequest(BaseModel):
-    """Request model for generation."""
-    prompts: List[str]
-    max_tokens: int = 100
-    temperature: float = 0.7
-    stage_args: List[Dict[str, Any]] = None
-
-
-class GenerateResponse(BaseModel):
-    """Response model for generation."""
-    outputs: List[Dict[str, Any]]
-    stage_outputs: List[Dict[str, Any]] = None
-
-
-app = FastAPI(
-    title="vLLM-omni API",
-    description="Multi-modality models inference and serving",
-    version="0.1.0"
-)
-
-# Global omni_llm instance
-omni_llm: AsyncOmniLLM = None
-
-
-@app.on_event("startup")
-async def startup_event():
-    """Initialize the omni_llm instance on startup."""
-    global omni_llm
-    # This will be set by the run_server function
-    pass
-
-
-@app.post("/generate", response_model=GenerateResponse)
-async def generate(request: GenerateRequest):
-    """Generate text or multimodal content."""
-    try:
-        if omni_llm is None:
-            raise HTTPException(status_code=500, detail="OmniLLM not initialized")
-        
-        # Prepare stage arguments
-        if request.stage_args is None:
-            # Create default stage arguments - one per stage config
-            # For now, we'll process all prompts in the first stage
-            stage_args = [{
-                "prompt": " ".join(request.prompts) if request.prompts else "",
-                "max_tokens": request.max_tokens,
-                "temperature": request.temperature
-            }]
-        else:
-            stage_args = request.stage_args
-        
-        # Generate using omni_llm
-        try:
-            outputs = await omni_llm.generate_async(stage_args)
-        except Exception as e:
-            # Fallback to synchronous generation
-            outputs = omni_llm.generate(stage_args)
-        
-        # Convert outputs to response format
-        response_outputs = []
-        for output in outputs:
-            if hasattr(output, 'outputs') and output.outputs:
-                for out in output.outputs:
-                    response_outputs.append({
-                        "text": getattr(out, 'text', ''),
-                        "finished": getattr(out, 'finish_reason', 'length') != 'length',
-                        "tokens": getattr(out, 'token_ids', [])
-                    })
-            else:
-                response_outputs.append({
-                    "text": "",
-                    "finished": True,
-                    "tokens": []
-                })
-        
-        return GenerateResponse(
-            outputs=response_outputs,
-            stage_outputs=[{"stage": i, "output": "processed"} for i in range(len(stage_args))]
-        )
-    
-    except Exception as e:
-        raise HTTPException(status_code=500, detail=str(e))
-
-
-@app.get("/health")
-async def health_check():
-    """Health check endpoint."""
-    return {"status": "healthy", "service": "vllm-omni"}
-
-
-@app.get("/info")
-async def get_info():
-    """Get information about the service."""
-    if omni_llm is None:
-        return {"error": "OmniLLM not initialized"}
-    
-    return {
-        "service": "vllm-omni",
-        "version": "0.1.0",
-        "num_stages": omni_llm.get_num_stages() if hasattr(omni_llm, 'get_num_stages') else 0,
-        "stage_configs": [
-            {
-                "stage_id": config.stage_id,
-                "engine_type": config.engine_type,
-                "model_path": config.model_path,
-                "input_modalities": config.input_modalities,
-                "output_modalities": config.output_modalities
-            }
-            for config in omni_llm.stage_configs
-        ]
-    }
-
-
-async def run_server(omni_llm_instance: AsyncOmniLLM, host: str = "0.0.0.0", port: int = 8000):
-    """Run the API server."""
-    global omni_llm
-    omni_llm = omni_llm_instance
-    
-    config = uvicorn.Config(
-        app=app,
-        host=host,
-        port=port,
-        log_level="info"
-    )
-    
-    server = uvicorn.Server(config)
-    await server.serve()
-
-
-if __name__ == "__main__":
-    # This is for testing purposes
-    asyncio.run(run_server(None))

vllm_omni/entrypoints/cli/__init__.py

@@ -1,5 +0,0 @@
-"""CLI helpers for vLLM-omni entrypoints."""
-
-from .serve import OmniServeCommand
-
-__all__ = ["OmniServeCommand"]

vllm_omni/entrypoints/cli/main.py

@@ -1,26 +0,0 @@
-"""
-CLI entry point for vLLM-omni that intercepts vLLM commands.
-"""
-
-import sys
-import argparse
-from typing import List, Optional
-from vllm_omni.entrypoints.cli.serve import OmniServeCommand
-
-
-def main():
-    """Main CLI entry point that intercepts vLLM commands."""
-    # Check if --omni flag is present
-    if "--omni" in sys.argv:
-        # Remove --omni flag and "serve" command, process with vLLM-omni
-        omni_args = [arg for arg in sys.argv[1:] if arg not in ["--omni", "serve"]]
-        omni_serve = OmniServeCommand()
-        omni_serve.run(omni_args)
-    else:
-        # Forward to original vLLM CLI
-        from vllm.entrypoints.cli.main import main as vllm_main
-        vllm_main()
-
-
-if __name__ == "__main__":
-    main()

vllm_omni/entrypoints/cli/serve.py

@@ -1,208 +0,0 @@
-"""
-Omni serve command for vLLM-omni.
-"""
-
-import argparse
-import asyncio
-from typing import List, Optional
-
-from ..omni_llm import AsyncOmniLLM
-from vllm_omni.config import (
-    DiTConfig,
-    create_ar_stage_config,
-    create_dit_stage_config,
-)
-
-
-class OmniServeCommand:
-    """Command handler for vLLM-omni serve command."""
-    
-    def __init__(self):
-        self.parser = self._create_parser()
-    
-    def _create_parser(self) -> argparse.ArgumentParser:
-        """Create argument parser for omni serve command."""
-        parser = argparse.ArgumentParser(
-            description="vLLM-omni: Multi-modality models inference and serving"
-        )
-        
-        # Model arguments - make it optional with default
-        parser.add_argument(
-            "model",
-            nargs="?",
-            default="Qwen/Qwen3-0.6B",
-            help="Path to the model or model name (default: Qwen/Qwen3-0.6B)"
-        )
-        
-        # Server arguments
-        parser.add_argument(
-            "--port",
-            type=int,
-            default=8000,
-            help="Port to run the server on"
-        )
-        
-        parser.add_argument(
-            "--host",
-            type=str,
-            default="0.0.0.0",
-            help="Host to run the server on"
-        )
-        
-        # Stage configuration arguments
-        parser.add_argument(
-            "--ar-stage",
-            type=str,
-            help="AR stage model path"
-        )
-        
-        parser.add_argument(
-            "--dit-stage",
-            type=str,
-            help="DiT stage model path"
-        )
-        
-        parser.add_argument(
-            "--dit-steps",
-            type=int,
-            default=50,
-            help="Number of DiT inference steps"
-        )
-        
-        parser.add_argument(
-            "--dit-guidance-scale",
-            type=float,
-            default=7.5,
-            help="DiT guidance scale"
-        )
-        
-        parser.add_argument(
-            "--use-diffusers",
-            action="store_true",
-            help="Use diffusers pipeline for DiT stage"
-        )
-        
-        # Other arguments
-        parser.add_argument(
-            "--log-stats",
-            action="store_true",
-            help="Enable logging statistics"
-        )
-        
-        return parser
-    
-    def run(self, args: List[str]) -> None:
-        """Run the omni serve command."""
-        parsed_args = self.parser.parse_args(args)
-        
-        # Create stage configurations
-        stage_configs = self._create_stage_configs(parsed_args)
-        
-        # Create AsyncOmniLLM instance
-        omni_llm = AsyncOmniLLM(
-            stage_configs=stage_configs,
-            log_stats=parsed_args.log_stats
-        )
-        
-        # Start the server
-        asyncio.run(self._start_server(omni_llm, parsed_args))
-    
-    def _create_stage_configs(self, args) -> List:
-        """Create stage configurations based on arguments."""
-        stage_configs = []
-        stage_id = 0
-        
-        # Add AR stage - use main model if ar_stage not specified
-        ar_model = args.ar_stage if args.ar_stage else args.model
-        ar_config = create_ar_stage_config(
-            stage_id=stage_id,
-            model_path=ar_model,
-            input_modalities=["text"],
-            output_modalities=["text"]
-        )
-        stage_configs.append(ar_config)
-        stage_id += 1
-        
-        # Add DiT stage if specified
-        if args.dit_stage:
-            dit_model = args.dit_stage
-        elif args.use_diffusers:
-            # Use a default DiT model if diffusers is enabled
-            dit_model = "stabilityai/stable-diffusion-2-1"
-        else:
-            dit_model = None
-            
-        if dit_model:
-            dit_config = DiTConfig(
-                num_inference_steps=args.dit_steps,
-                guidance_scale=args.dit_guidance_scale,
-                use_diffusers=args.use_diffusers,
-                diffusers_pipeline="auto" if args.use_diffusers else None,
-            )
-            
-            dit_stage_config = create_dit_stage_config(
-                stage_id=stage_id,
-                model_path=dit_model,
-                input_modalities=["text"],
-                output_modalities=["image"],
-                dit_config=dit_config
-            )
-            stage_configs.append(dit_stage_config)
-            stage_id += 1
-        
-        # If no specific stages are specified, use the main model
-        if not stage_configs:
-            # Try to detect if it's a multimodal model
-            if "omni" in args.model.lower() or "multimodal" in args.model.lower():
-                # Assume it's a multimodal model that can handle both AR and DiT
-                ar_config = create_ar_stage_config(
-                    stage_id=0,
-                    model_path=args.model,
-                    input_modalities=["text"],
-                    output_modalities=["text"]
-                )
-                stage_configs.append(ar_config)
-                
-                dit_config = DiTConfig(
-                    num_inference_steps=args.dit_steps,
-                    guidance_scale=args.dit_guidance_scale,
-                    use_diffusers=args.use_diffusers,
-                    diffusers_pipeline="auto" if args.use_diffusers else None,
-                )
-                
-                dit_stage_config = create_dit_stage_config(
-                    stage_id=1,
-                    model_path=args.model,
-                    input_modalities=["text"],
-                    output_modalities=["image"],
-                    dit_config=dit_config
-                )
-                stage_configs.append(dit_stage_config)
-            else:
-                # Default to AR stage
-                ar_config = create_ar_stage_config(
-                    stage_id=0,
-                    model_path=args.model,
-                    input_modalities=["text"],
-                    output_modalities=["text"]
-                )
-                stage_configs.append(ar_config)
-        
-        return stage_configs
-    
-    async def _start_server(self, omni_llm: AsyncOmniLLM, args) -> None:
-        """Start the API server."""
-        try:
-            # Import here to avoid circular imports
-            from vllm_omni.entrypoints.api_server import run_server
-            
-            await run_server(
-                omni_llm_instance=omni_llm,
-                host=args.host,
-                port=args.port
-            )
-        except KeyboardInterrupt:
-            print("\nShutting down server...")
-        except Exception as e:
-            print(f"Error starting server: {e}")
-            raise

vllm_omni/entrypoints/omni_llm.py

@@ -1,157 +0,0 @@
-from typing import Union, Sequence, Optional, Any
-import cloudpickle
-from pydantic import ValidationError
-
-from vllm.inputs import PromptType
-from vllm.sampling_params import SamplingParams
-from vllm.entrypoints.llm import LLM
-
-from vllm.v1.engine.llm_engine import LLMEngine
-from vllm.engine.arg_utils import HfOverrides
-from vllm.usage.usage_lib import UsageContext
-from vllm.config import CompilationConfig, is_init_field
-from vllm.utils import Counter
-from vllm.logger import init_logger
-import vllm.envs as envs
-
-from vllm_omni.entrypoints.utils import load_stage_configs_from_model
-from vllm_omni.entrypoints.omni_stage import OmniStage
-from vllm_omni.engine.arg_utils import OmniEngineArgs
-from vllm_omni.engine.output_processor import MultimodalOutputProcessor
-from vllm_omni.engine.processor import OmniProcessor
-from vllm_omni.outputs import OmniRequestOutput
-
-
-logger = init_logger(__name__)
-
-
-class OmniLLM:
-    def __init__(self, model: str, stage_configs = None, log_stats: bool = False, **kwargs):
-        if stage_configs is None:
-            self.initialize_stage_configs(model)
-        else:
-            self.stage_configs = stage_configs
-        
-        self.stage_list = []
-        self.initialize_stages(model)
-        
-    def initialize_stage_configs(self, model: str):
-        self.stage_configs = load_stage_configs_from_model(model)
-    
-    def initialize_stages(self, model: str):
-        for stage_config in self.stage_configs:
-            stage = OmniStage(stage_config)
-            stage_llm = OmniStageLLM(model=model, **stage_config.engine_args)
-            stage.set_engine(stage_llm)
-            self.stage_list.append(stage)
-    
-    def generate(
-        self,
-        prompts: Union[PromptType, Sequence[PromptType]],
-        sampling_params_list: Optional[Union[SamplingParams,
-                                        Sequence[SamplingParams]]] = None,
-    ) -> list[OmniRequestOutput]:
-        """Generate text outputs for the given prompts."""
-        final_outputs: list[OmniRequestOutput] = []
-        if len(sampling_params_list) != len(self.stage_list):
-            raise ValueError(f"Expected {len(self.stage_list)} sampling params, got {len(sampling_params_list)}")
-        for stage_id, stage in enumerate(self.stage_list):
-            if stage_id > 0:
-                engine_inputs = stage.process_engine_inputs(self.stage_list, prompts)
-            else:
-                engine_inputs = prompts
-            engine_outputs = self._run_generation(stage, sampling_params_list[stage_id], engine_inputs)
-            stage.set_engine_outputs(engine_outputs)
-            if hasattr(stage, 'final_output') and stage.final_output:
-                final_outputs.append(OmniRequestOutput(
-                    stage_id=stage_id, 
-                    final_output_type=stage.final_output_type, 
-                    request_output=engine_outputs))
-        return final_outputs
-    
-    def _run_generation(self, stage: OmniStage, sampling_params: SamplingParams, prompts: Union[PromptType, Sequence[PromptType]]):
-        engine_outputs = []
-        for ro in stage.engine.generate(prompts, sampling_params):
-            engine_outputs.append(ro)
-        return engine_outputs
-
-
-class OmniStageLLM(LLM): 
-    def __init__(self, 
-                model: str, 
-                compilation_config: Optional[Union[int, dict[str, Any],
-                                           CompilationConfig]] = None,
-                hf_overrides: Optional[HfOverrides] = None,
-                **kwargs):
-        """LLM constructor."""
-        if "disable_log_stats" not in kwargs:
-            kwargs["disable_log_stats"] = True
-
-        if "worker_cls" in kwargs:
-            worker_cls = kwargs["worker_cls"]
-            # if the worker_cls is not qualified string name,
-            # we serialize it using cloudpickle to avoid pickling issues
-            if isinstance(worker_cls, type):
-                kwargs["worker_cls"] = cloudpickle.dumps(worker_cls)
-
-        if "kv_transfer_config" in kwargs and isinstance(
-                kwargs["kv_transfer_config"], dict):
-            from vllm.config import KVTransferConfig
-            raw_config_dict = kwargs["kv_transfer_config"]
-            try:
-                kwargs["kv_transfer_config"] = KVTransferConfig(
-                    **raw_config_dict)
-            except ValidationError as e:
-                logger.error(
-                    "Failed to convert 'kv_transfer_config' dict to "
-                    "KVTransferConfig object. Dict: %s. Error: %s",
-                    raw_config_dict, e)
-                # Consider re-raising a more specific vLLM error or ValueError
-                # to provide better context to the user.
-                raise ValueError(
-                    f"Invalid 'kv_transfer_config' provided: {e}") from e
-
-        if hf_overrides is None:
-            hf_overrides = {}
-
-        if compilation_config is not None:
-            if isinstance(compilation_config, int):
-                compilation_config_instance = CompilationConfig(
-                    level=compilation_config)
-            elif isinstance(compilation_config, dict):
-                predicate = lambda x: is_init_field(CompilationConfig, x[0])
-                compilation_config_instance = CompilationConfig(
-                    **dict(filter(predicate, compilation_config.items())))
-            else:
-                compilation_config_instance = compilation_config
-        else:
-            compilation_config_instance = CompilationConfig()
-
-        engine_args = OmniEngineArgs(
-            model=model,
-            hf_overrides=hf_overrides,
-            compilation_config=compilation_config_instance,
-            **kwargs,
-        )
-
-        # Create the Engine (autoselects V0 vs V1)
-        self.llm_engine = LLMEngine.from_engine_args(
-            engine_args=engine_args, usage_context=UsageContext.LLM_CLASS)
-        self.llm_engine.output_processor = MultimodalOutputProcessor(tokenizer=self.llm_engine.tokenizer, 
-                                                                    log_stats=self.llm_engine.log_stats)
-        self.llm_engine.processor = OmniProcessor(vllm_config=self.llm_engine.vllm_config,
-                                                  tokenizer=self.llm_engine.tokenizer)
-        self.engine_class = type(self.llm_engine)
-
-        self.request_counter = Counter()
-        self.default_sampling_params: Union[dict[str, Any], None] = None
-
-        if envs.VLLM_USE_V1:
-            supported_tasks = self.llm_engine \
-                .get_supported_tasks()  # type: ignore
-        else:
-            supported_tasks = self.llm_engine.model_config.supported_tasks
-
-        logger.info("Supported_tasks: %s", supported_tasks)
-
-        self.supported_tasks = supported_tasks

vllm_omni/entrypoints/omni_stage.py

@@ -1,80 +0,0 @@
-"""
-Stage manager for orchestrating multiple engines in vLLM-omni.
-"""
-
-import importlib
-from typing import List, Union
-from vllm.v1.engine.llm_engine import LLMEngine
-from vllm.v1.engine.async_llm import AsyncLLM
-from vllm.inputs import TextPrompt
-from vllm.v1.engine import EngineCoreOutput
-
-from vllm_omni.inputs.data import OmniTokensPrompt
-
-
-class OmniStage:
-    def __init__(self, stage_config):
-        self.stage_config = stage_config
-        self.engine = None
-        self.async_engine = None
-        self.stage_id = stage_config.stage_id
-        self.engine_args = stage_config.engine_args
-        self.model_stage = stage_config.engine_args.model_stage
-        if hasattr(stage_config, 'engine_input_source'):
-            self.engine_input_source = stage_config.engine_input_source
-        else:
-            self.engine_input_source = []
-        self.engine_output_type = stage_config.engine_args.engine_output_type
-        self.engine_outputs = None
-        if hasattr(stage_config, 'custom_process_input_func'):
-            # Import the module specified in the config (already a full module path)
-            module_path, func_name = stage_config.custom_process_input_func.rsplit('.', 1)
-            module = importlib.import_module(module_path)
-            self.custom_process_input_func = getattr(module, func_name)
-        else:
-            self.custom_process_input_func = None
-
-        if hasattr(stage_config, 'final_output'):
-            self.final_output = stage_config.final_output
-        else:
-            self.final_output = False
-
-        if hasattr(stage_config, 'final_output_type'):
-            self.final_output_type = stage_config.final_output_type
-        else:
-            self.final_output_type = None
-
-    def set_engine(self, engine: LLMEngine) -> None:
-        """Initialize the engine for the stage."""
-        self.engine = engine
-
-    def set_async_engine(self, async_engine: AsyncLLM) -> None:
-        """Initialize the async engine for the stage."""
-        self.async_engine = async_engine
-    
-    def set_engine_outputs(self, engine_outputs: EngineCoreOutput) -> None:
-        """Set the engine output for the stage."""
-        self.engine_outputs = engine_outputs
-    
-    def process_engine_inputs(self, stage_list, prompt: Union[OmniTokensPrompt, TextPrompt] = None) -> List[Union[OmniTokensPrompt, TextPrompt]]:
-        """Process the engine input for the stage."""
-        if self.custom_process_input_func is None:
-            engine_inputs = []
-            if len(self.engine_input_source) == 0:
-                raise ValueError("engine_input_source is empty")
-            source_stage_id = self.engine_input_source[0]
-            source_outputs = stage_list[source_stage_id].engine_outputs
-            multi_modal_data = {source_output.request_id: 
-            prompt.get('multi_modal_data', None) for source_output, prompt in zip(source_outputs, prompt)}
-
-            for source_output in source_outputs:
-                engine_input = OmniTokensPrompt(
-                    prompt_token_ids = source_output.outputs[0].token_ids,
-                    multi_modal_data=multi_modal_data[source_output.request_id] if multi_modal_data else None,
-                )
-                engine_inputs.append(engine_input)
-            return engine_inputs
-
-        else:
-            engine_input_source = self.engine_input_source
-            return self.custom_process_input_func(stage_list, engine_input_source, prompt)

vllm_omni/entrypoints/utils.py

@@ -1,25 +0,0 @@
-import os
-from pathlib import Path
-from omegaconf import OmegaConf
-from vllm.transformers_utils.config import get_config
-
-# Get the project root directory (2 levels up from this file)
-PROJECT_ROOT = Path(__file__).parent.parent.parent
-
-
-def load_stage_configs_from_model(model: str):
-    """Load stage configs from model."""
-    hf_config = get_config(model, trust_remote_code=True)
-    model_type = hf_config.model_type
-    stage_config_file = f"vllm_omni/model_executor/stage_configs/{model_type}.yaml"
-    stage_config_path = PROJECT_ROOT / stage_config_file
-    if not os.path.exists(stage_config_path):
-        raise FileNotFoundError(f"Stage config file {stage_config_path} not found")
-    stage_configs = load_stage_configs_from_yaml(config_path=str(stage_config_path))
-    return stage_configs
-
-
-def load_stage_configs_from_yaml(config_path: str):
-    """Load stage configs from yaml file."""
-    config_data = OmegaConf.load(config_path)
-    return config_data.stage_args

vllm_omni/inputs/data.py

@@ -1,12 +0,0 @@
-from vllm.inputs.data import TokensPrompt
-from typing import Any, NotRequired
-import torch
-
-
-class OmniTokensPrompt(TokensPrompt):
-    prompt_embeds: NotRequired[torch.Tensor]
-    """The embeddings of the prompt."""
-
-    # New: optional additional information dictionary
-    # Values may be torch.Tensor or list
-    additional_information: NotRequired[dict[str, Any]]

vllm_omni/model_executor/layers/mrope.py

@@ -1,699 +0,0 @@
-import itertools
-from typing import Optional, Union
-
-import numpy as np
-import torch
-from transformers import PretrainedConfig
-
-from vllm.model_executor.layers.rotary_embedding.base import RotaryEmbedding
-from vllm.model_executor.layers.rotary_embedding.common import apply_rotary_emb_dispatch
-
-def _apply_rotary_emb(
-    x: torch.Tensor,
-    cos: torch.Tensor,
-    sin: torch.Tensor,
-    is_neox_style: bool,
-) -> torch.Tensor:
-    """
-    Args:
-        x: [num_tokens, num_heads, head_size]
-        cos: [num_tokens, head_size // 2]
-        sin: [num_tokens, head_size // 2]
-        is_neox_style: Whether to use the Neox-style or GPT-J-style rotary
-            positional embeddings.
-    """
-    cos = cos.unsqueeze(-2).to(x.dtype)
-    sin = sin.unsqueeze(-2).to(x.dtype)
-    if is_neox_style:
-        x1, x2 = torch.chunk(x, 2, dim=-1)
-    else:
-        x1 = x[..., ::2]
-        x2 = x[..., 1::2]
-    o1 = x1 * cos - x2 * sin
-    o2 = x2 * cos + x1 * sin
-    if is_neox_style:
-        return torch.cat((o1, o2), dim=-1)
-    else:
-        return torch.stack((o1, o2), dim=-1).flatten(-2)
-
-class MRotaryEmbedding(RotaryEmbedding):
-    """Rotary Embedding with Multimodal Sections."""
-
-    def __init__(
-        self,
-        head_size: int,
-        rotary_dim: int,
-        max_position_embeddings: int,
-        base: float,
-        is_neox_style: bool,
-        dtype: torch.dtype,
-        mrope_section: Optional[list[int]] = None,
-    ) -> None:
-        # In Qwen2.5-VL, the maximum index value is related to the duration of
-        # the input video. We enlarge max_position_embeddings to 4 times to get
-        # a larger the cos and sin cache.
-        self.cache_max_position_num = max_position_embeddings * 4
-        super().__init__(head_size, rotary_dim, self.cache_max_position_num,
-                         base, is_neox_style, dtype)
-
-        self.mrope_section = mrope_section
-        if self.mrope_section:
-            print("Warning: mrope_section check is disabled in Qwen2.5-Omni, this may cause errors, and should be restored in the future.")
-            # assert sum(self.mrope_section) == rotary_dim // 2
-
-    def forward(
-        self,
-        positions: torch.Tensor,
-        query: torch.Tensor,
-        key: Optional[torch.Tensor] = None,
-    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
-        """PyTorch-native implementation equivalent to forward().
-
-        Args:
-            positions:
-                [num_tokens,] (text only) or
-                [3, num_tokens] (T/H/W positions with multimodal inputs)
-            query: [num_tokens, num_heads * head_size]
-            key: [num_tokens, num_kv_heads * head_size]
-        """
-        assert positions.ndim == 1 or positions.ndim == 2
-        assert key is not None
-
-        num_tokens = positions.shape[-1]
-        cos_sin = self.cos_sin_cache[positions]
-        cos, sin = cos_sin.chunk(2, dim=-1)
-        if positions.ndim == 2:
-            assert self.mrope_section
-
-            cos = torch.cat([
-                m[i]
-                for i, m in enumerate(cos.split(self.mrope_section, dim=-1))
-            ],
-                            dim=-1)
-            sin = torch.cat([
-                m[i]
-                for i, m in enumerate(sin.split(self.mrope_section, dim=-1))
-            ],
-                            dim=-1)
-
-        query_shape = query.shape
-        query = query.view(num_tokens, -1, self.head_size)
-        query_rot = query[..., :self.rotary_dim]
-        query_pass = query[..., self.rotary_dim:]
-        # query_rot = apply_rotary_emb_dispatch(query_rot, cos, sin,
-        #                                       self.is_neox_style)
-        query_rot = _apply_rotary_emb(query_rot, cos, sin,
-                                              self.is_neox_style)
-        query = torch.cat((query_rot, query_pass), dim=-1).reshape(query_shape)
-
-        key_shape = key.shape
-        key = key.view(num_tokens, -1, self.head_size)
-        key_rot = key[..., :self.rotary_dim]
-        key_pass = key[..., self.rotary_dim:]
-        # key_rot = apply_rotary_emb_dispatch(key_rot, cos, sin,
-        #                                     self.is_neox_style)
-        key_rot = _apply_rotary_emb(key_rot, cos, sin,
-                                              self.is_neox_style)
-        key = torch.cat((key_rot, key_pass), dim=-1).reshape(key_shape)
-        return query, key
-
-    @classmethod
-    def get_input_positions(
-        cls,
-        input_tokens: list[int],
-        hf_config: PretrainedConfig,
-        image_grid_thw: Optional[Union[list[list[int]], torch.Tensor]],
-        video_grid_thw: Optional[Union[list[list[int]], torch.Tensor]],
-        second_per_grid_ts: Optional[list[float]],
-        context_len: int = 0,
-        seq_len: Optional[int] = None,
-        audio_feature_lengths: Optional[torch.Tensor] = None,
-        use_audio_in_video: bool = False,
-    ) -> tuple[list[list[int]], int]:
-        """Get mrope input positions and delta value."""
-
-        image_grid_thw = [] if image_grid_thw is None else image_grid_thw
-        video_grid_thw = [] if video_grid_thw is None else video_grid_thw
-        second_per_grid_ts = [] if second_per_grid_ts is None else \
-            second_per_grid_ts
-
-        llm_positions, mrope_position_delta = \
-            cls.get_input_positions_tensor(
-                input_tokens=input_tokens,
-                hf_config=hf_config,
-                image_grid_thw=image_grid_thw,
-                video_grid_thw=video_grid_thw,
-                second_per_grid_ts=second_per_grid_ts,
-                context_len=context_len,
-                seq_len=seq_len,
-                audio_feature_lengths=audio_feature_lengths,
-                use_audio_in_video=use_audio_in_video,
-            )
-
-        return llm_positions.tolist(), mrope_position_delta
-
-    @classmethod
-    def get_input_positions_tensor(
-        cls,
-        input_tokens: list[int],
-        hf_config: PretrainedConfig,
-        image_grid_thw: Union[list[list[int]], torch.Tensor],
-        video_grid_thw: Union[list[list[int]], torch.Tensor],
-        second_per_grid_ts: list[float],
-        context_len: int = 0,
-        seq_len: Optional[int] = None,
-        audio_feature_lengths: Optional[torch.Tensor] = None,
-        use_audio_in_video: bool = False,
-    ) -> tuple[torch.Tensor, int]:
-        from vllm.transformers_utils.config import thinker_uses_mrope
-        if thinker_uses_mrope(hf_config):
-            return cls._omni_get_input_positions_tensor(
-                input_tokens=input_tokens,
-                hf_config=hf_config,
-                image_grid_thw=image_grid_thw,
-                video_grid_thw=video_grid_thw,
-                second_per_grid_ts=second_per_grid_ts,
-                context_len=context_len,
-                seq_len=seq_len,
-                audio_feature_lengths=audio_feature_lengths,
-                use_audio_in_video=use_audio_in_video,
-            )
-        elif hf_config.model_type in ["glm4v", "glm4v_moe"]:
-            return cls._glm4v_get_input_positions_tensor(
-                input_tokens=input_tokens,
-                hf_config=hf_config,
-                image_grid_thw=image_grid_thw,
-                video_grid_thw=video_grid_thw,
-                context_len=context_len,
-                seq_len=seq_len,
-            )
-        else:
-            return cls._vl_get_input_positions_tensor(
-                input_tokens=input_tokens,
-                hf_config=hf_config,
-                image_grid_thw=image_grid_thw,
-                video_grid_thw=video_grid_thw,
-                second_per_grid_ts=second_per_grid_ts,
-                context_len=context_len,
-                seq_len=seq_len,
-            )
-
-    @classmethod
-    def _glm4v_get_input_positions_tensor(
-        cls,
-        input_tokens: list[int],
-        hf_config: PretrainedConfig,
-        image_grid_thw: Union[list[list[int]], torch.Tensor],
-        video_grid_thw: Union[list[list[int]], torch.Tensor],
-        context_len: int = 0,
-        seq_len: Optional[int] = None,
-    ) -> tuple[torch.Tensor, int]:
-        """Get mrope input positions and delta value for GLM4V."""
-
-        image_token_id = hf_config.image_token_id
-        video_start_token_id = hf_config.video_start_token_id
-        video_end_token_id = hf_config.video_end_token_id
-        spatial_merge_size = hf_config.vision_config.spatial_merge_size
-        llm_pos_ids_list: list = []
-
-        if not (image_grid_thw is None and video_grid_thw is None):
-            if isinstance(image_grid_thw, torch.Tensor):
-                image_grid_thw = image_grid_thw.tolist()
-
-            input_token_type: list[str] = []
-            video_check_flg = False
-            for token in input_tokens:
-                if token == video_start_token_id:
-                    video_check_flg = True
-                elif token == video_end_token_id:
-                    video_check_flg = False
-
-                if (token == image_token_id) and (video_check_flg is False):
-                    input_token_type.append("image")
-                elif (token == image_token_id) and (video_check_flg is True):
-                    input_token_type.append("video")
-                else:
-                    input_token_type.append("text")
-
-            input_type_group: list[tuple[str, int, int]] = []
-            for key, group_iter in itertools.groupby(
-                    enumerate(input_token_type), lambda x: x[1]):
-                group_list = list(group_iter)
-                start_index = group_list[0][0]
-                end_index = group_list[-1][0] + 1
-                input_type_group.append((key, start_index, end_index))
-
-            video_frame_num = 1
-            mm_data_idx = 0
-            for modality_type, start_idx, end_idx in input_type_group:
-                st_idx = llm_pos_ids_list[-1].max() + 1 if len(
-                    llm_pos_ids_list) > 0 else 0
-                if modality_type == "image":
-                    t, h, w = (
-                        image_grid_thw[mm_data_idx][0],
-                        image_grid_thw[mm_data_idx][1],
-                        image_grid_thw[mm_data_idx][2],
-                    )
-                    llm_grid_t, llm_grid_h, llm_grid_w = \
-                        t, h // spatial_merge_size, w // spatial_merge_size
-
-                    t_index = torch.arange(llm_grid_t).view(-1, 1).expand(
-                        -1, llm_grid_h * llm_grid_w).flatten()
-                    h_index = torch.arange(llm_grid_h).view(1, -1, 1).expand(
-                        llm_grid_t, -1, llm_grid_w).flatten()
-                    w_index = torch.arange(llm_grid_w).view(1, 1, -1).expand(
-                        llm_grid_t, llm_grid_h, -1).flatten()
-                    llm_pos_ids_list.append(
-                        torch.stack([t_index, h_index, w_index]) + st_idx)
-                    mm_data_idx += 1
-
-                elif modality_type == "video":
-                    t, h, w = (
-                        video_frame_num,
-                        image_grid_thw[mm_data_idx][1],
-                        image_grid_thw[mm_data_idx][2],
-                    )
-                    llm_grid_t, llm_grid_h, llm_grid_w = \
-                        t, h // spatial_merge_size, w // spatial_merge_size
-
-                    for t_idx in range(llm_grid_t):
-                        t_index = torch.tensor(t_idx).view(-1, 1).expand(
-                            -1, llm_grid_h * llm_grid_w).flatten()
-                        h_index = torch.arange(llm_grid_h).view(
-                            1, -1, 1).expand(1, -1, llm_grid_w).flatten()
-                        w_index = torch.arange(llm_grid_w).view(
-                            1, 1, -1).expand(1, llm_grid_h, -1).flatten()
-                        llm_pos_ids_list.append(
-                            torch.stack([t_index, h_index, w_index]) + st_idx)
-
-                    mm_data_idx += 1
-                    video_frame_num += 1
-
-                else:
-                    text_len = end_idx - start_idx
-                    llm_pos_ids_list.append(
-                        torch.arange(text_len).view(1, -1).expand(3, -1) +
-                        st_idx)
-                    video_frame_num = 1
-
-        else:
-            text_len = len(input_tokens)
-            llm_pos_ids_list.append(
-                torch.arange(text_len).view(1, -1).expand(3, -1))
-
-        llm_positions = torch.cat(llm_pos_ids_list, dim=1).reshape(3, -1)
-        llm_positions = llm_positions[:, context_len:seq_len]
-        mrope_position_delta = (llm_positions.max() + 1 -
-                                len(input_tokens)).item()
-        return llm_positions, mrope_position_delta
-
-    @classmethod
-    def _vl_get_input_positions_tensor(
-        cls,
-        input_tokens: list[int],
-        hf_config: PretrainedConfig,
-        image_grid_thw: Union[list[list[int]], torch.Tensor],
-        video_grid_thw: Union[list[list[int]], torch.Tensor],
-        second_per_grid_ts: list[float],
-        context_len: int = 0,
-        seq_len: Optional[int] = None,
-    ) -> tuple[torch.Tensor, int]:
-        """Get mrope input positions and delta value."""
-
-        image_token_id = hf_config.image_token_id
-        video_token_id = hf_config.video_token_id
-        vision_start_token_id = hf_config.vision_start_token_id
-        spatial_merge_size = hf_config.vision_config.spatial_merge_size
-        tokens_per_second = getattr(hf_config.vision_config,
-                                    "tokens_per_second", 1.0)
-
-        input_tokens_tensor = torch.tensor(input_tokens)
-        vision_start_indices = torch.argwhere(
-            input_tokens_tensor == vision_start_token_id).squeeze(1)
-        vision_tokens = input_tokens_tensor[vision_start_indices + 1]
-        image_nums = (vision_tokens == image_token_id).sum()
-        video_nums = (vision_tokens == video_token_id).sum()
-        llm_pos_ids_list: list = []
-
-        st = 0
-        remain_images, remain_videos = image_nums, video_nums
-
-        image_index, video_index = 0, 0
-        for _ in range(image_nums + video_nums):
-            video_second_per_grid_t = 0.0
-            if image_token_id in input_tokens and remain_images > 0:
-                ed_image = input_tokens.index(image_token_id, st)
-            else:
-                ed_image = len(input_tokens) + 1
-            if video_token_id in input_tokens and remain_videos > 0:
-                ed_video = input_tokens.index(video_token_id, st)
-            else:
-                ed_video = len(input_tokens) + 1
-            if ed_image < ed_video:
-                t, h, w = (
-                    image_grid_thw[image_index][0],
-                    image_grid_thw[image_index][1],
-                    image_grid_thw[image_index][2],
-                )
-                image_index += 1
-                remain_images -= 1
-                ed = ed_image
-            else:
-                t, h, w = (
-                    video_grid_thw[video_index][0],
-                    video_grid_thw[video_index][1],
-                    video_grid_thw[video_index][2],
-                )
-                video_second_per_grid_t = 1.0
-                if second_per_grid_ts:
-                    video_second_per_grid_t = second_per_grid_ts[video_index]
-                video_index += 1
-                remain_videos -= 1
-                ed = ed_video
-
-            llm_grid_t, llm_grid_h, llm_grid_w = \
-                t, h // spatial_merge_size, w // spatial_merge_size
-            text_len = ed - st
-
-            st_idx = llm_pos_ids_list[-1].max() + 1 if len(
-                llm_pos_ids_list) > 0 else 0
-            llm_pos_ids_list.append(
-                torch.arange(text_len).view(1, -1).expand(3, -1) + st_idx)
-
-            t_index = (torch.arange(llm_grid_t).view(-1, 1).expand(
-                -1, llm_grid_h * llm_grid_w) * video_second_per_grid_t *
-                       tokens_per_second).long().flatten()
-
-            h_index = torch.arange(llm_grid_h).view(1, -1, 1).expand(
-                llm_grid_t, -1, llm_grid_w).flatten()
-            w_index = torch.arange(llm_grid_w).view(1, 1, -1).expand(
-                llm_grid_t, llm_grid_h, -1).flatten()
-            llm_pos_ids_list.append(
-                torch.stack([t_index, h_index, w_index]) + text_len + st_idx)
-            st = ed + llm_grid_t * llm_grid_h * llm_grid_w
-
-        if st < len(input_tokens):
-            st_idx = llm_pos_ids_list[-1].max() + 1 if len(
-                llm_pos_ids_list) > 0 else 0
-            text_len = len(input_tokens) - st
-            llm_pos_ids_list.append(
-                torch.arange(text_len).view(1, -1).expand(3, -1) + st_idx)
-
-        llm_positions = torch.cat(llm_pos_ids_list, dim=1).reshape(3, -1)
-        mrope_position_delta = (llm_positions.max() + 1 -
-                                len(input_tokens)).item()
-        llm_positions = llm_positions[:, context_len:seq_len]
-
-        return llm_positions, mrope_position_delta
-
-    @classmethod
-    def _omni_get_input_positions_tensor(
-        cls,
-        input_tokens: list[int],
-        hf_config: PretrainedConfig,
-        image_grid_thw: Union[list[list[int]], torch.Tensor],
-        video_grid_thw: Union[list[list[int]], torch.Tensor],
-        second_per_grid_ts: Optional[list[float]] = None,
-        context_len: int = 0,
-        seq_len: Optional[int] = None,
-        audio_feature_lengths: Optional[torch.Tensor] = None,
-        use_audio_in_video: bool = False,
-    ) -> tuple[torch.Tensor, int]:
-        """Get mrope input positions and delta value (Qwen2.5-Omni version).
-
-        Differences from MRotaryEmbedding:
-            1. Add audio support (and related `audio_feature_lengths`).
-            2. Add `use_audio_in_video` option to read audio from video inputs.
-                In this case, audio and vision position ids will be split into
-                chunks and interleaved.
-
-        Example:
-
-            (V_i are vision position ids, A_i are audio position ids)
-
-            |V_1 ...    V_n|A_1 ...   A_n|V_n+1 ... V_2n|A_n+1 ... A_2n|...
-            |vision chunk 1|audio chunk 1|vision chunk 2|audio chunk 2 |...
-        """
-
-        # TODO(fyabc): refactor and share more code with
-        #  _vl_get_input_positions_tensor.
-
-        thinker_config = hf_config.thinker_config
-        audio_token_id = thinker_config.audio_token_index
-        image_token_id = thinker_config.image_token_index
-        video_token_id = thinker_config.video_token_index
-        audio_start_token_id = thinker_config.audio_start_token_id
-        audio_end_token_id = thinker_config.audio_end_token_id
-        vision_start_token_id = thinker_config.vision_start_token_id
-        vision_end_token_id = thinker_config.vision_end_token_id
-        seconds_per_chunk = thinker_config.seconds_per_chunk
-        spatial_merge_size = thinker_config.vision_config.spatial_merge_size
-        tokens_per_second = getattr(thinker_config.vision_config,
-                                    "tokens_per_second", 25)
-
-        if isinstance(image_grid_thw, list):
-            image_grid_thw = torch.tensor(image_grid_thw)
-        if isinstance(video_grid_thw, list):
-            video_grid_thw = torch.tensor(video_grid_thw)
-
-        src_item = input_tokens
-        audio_seqlens = audio_feature_lengths
-        if not second_per_grid_ts:
-            second_per_grid_ts = [1] * video_grid_thw.shape[0]
-        audio_idx = 0
-        video_idx = 0
-        image_idx = 0
-        new_src_item: list[int] = []
-        llm_pos_ids_list: list[torch.Tensor] = []
-
-        idx = 0
-        while idx < len(src_item):
-            new_src_item_len = len(new_src_item)
-            start_idx = llm_pos_ids_list[-1].max() + 1 if len(
-                llm_pos_ids_list) > 0 else 0
-            if src_item[idx] not in [
-                    audio_token_id, video_token_id, image_token_id
-            ]:
-                if use_audio_in_video and idx > 0:
-                    if src_item[idx] == vision_end_token_id and \
-                        src_item[idx - 1] == audio_end_token_id:
-                        # processing the <|audio_eos|> before <|vision_eos|>
-                        start_idx -= 1
-                    elif src_item[idx] == audio_start_token_id and \
-                        src_item[idx - 1] == vision_start_token_id:
-                        # processing the <|audio_bos|> after <|vision_eos|>
-                        start_idx -= 1
-                new_src_item.append(src_item[idx])
-                llm_pos_ids = torch.tensor([start_idx],
-                                           dtype=torch.long).expand(3, -1)
-                llm_pos_ids_list.append(llm_pos_ids)
-            elif src_item[idx] == audio_token_id:
-                assert audio_seqlens is not None
-                audio_seqlen = audio_seqlens[audio_idx]
-                place_num = (((audio_seqlen - 1) // 2 + 1 - 2) // 2 + 1)
-                new_src_item.extend([audio_token_id] * place_num)
-                llm_pos_ids = torch.arange(place_num).expand(3, -1) + start_idx
-                llm_pos_ids_list.append(llm_pos_ids)
-                audio_idx += 1
-            elif src_item[idx] == image_token_id:
-                grid_t = image_grid_thw[image_idx][0]
-                grid_hs = image_grid_thw[:, 1]
-                grid_ws = image_grid_thw[:, 2]
-                t_index = (torch.arange(grid_t) * 1 * tokens_per_second).long()
-                llm_pos_ids = cls._get_llm_pos_ids_for_vision(
-                    start_idx, image_idx, spatial_merge_size, t_index, grid_hs,
-                    grid_ws)
-                llm_pos_ids_list.append(llm_pos_ids)
-                vision_seqlen = image_grid_thw[image_idx].prod() // (
-                    spatial_merge_size**2)
-                new_src_item.extend([image_token_id] * vision_seqlen)
-                image_idx += 1
-            elif src_item[idx] == video_token_id and not use_audio_in_video:
-                grid_t = video_grid_thw[video_idx][0]
-                grid_hs = video_grid_thw[:, 1]
-                grid_ws = video_grid_thw[:, 2]
-                t_index = (torch.arange(grid_t) *
-                           second_per_grid_ts[video_idx] *
-                           tokens_per_second).long()
-                llm_pos_ids = cls._get_llm_pos_ids_for_vision(
-                    start_idx, video_idx, spatial_merge_size, t_index, grid_hs,
-                    grid_ws)
-                llm_pos_ids_list.append(llm_pos_ids)
-                vision_seqlen = video_grid_thw[video_idx].prod() // (
-                    spatial_merge_size**2)
-                new_src_item.extend([video_token_id] * vision_seqlen)
-                video_idx += 1
-            else:
-                # read audio from video
-                assert audio_seqlens is not None
-                audio_seqlen = audio_seqlens[audio_idx]
-                vision_seqlen = video_grid_thw[video_idx].prod() // (
-                    spatial_merge_size**2)
-                grid_t = video_grid_thw[video_idx][0]
-                grid_h = video_grid_thw[video_idx][1]
-                grid_w = video_grid_thw[video_idx][2]
-                grid_hs = video_grid_thw[:, 1]
-                grid_ws = video_grid_thw[:, 2]
-                t_ntoken_per_chunk = int(tokens_per_second * seconds_per_chunk)
-                t_index = (torch.arange(grid_t) *
-                           second_per_grid_ts[video_idx] *
-                           tokens_per_second).long()
-                t_index_split_chunk = cls._split_list_into_ranges(
-                    t_index, t_ntoken_per_chunk)
-                place_num = (((audio_seqlen - 1) // 2 + 1 - 2) // 2 + 1) + 2
-                pure_audio_len = place_num - 2
-                added_audio_len = 0
-                audio_llm_pos_ids_list: list[torch.Tensor] = []
-                for t_chunk in t_index_split_chunk:
-                    vision_ntoken_per_chunk = len(
-                        t_chunk) * grid_h * grid_w // (spatial_merge_size**2)
-                    new_src_item.extend([video_token_id] *
-                                        vision_ntoken_per_chunk)
-                    vision_llm_pos_ids_list = cls._get_llm_pos_ids_for_vision(
-                        start_idx, video_idx, spatial_merge_size, t_chunk,
-                        grid_hs, grid_ws).split(1, dim=1)
-                    llm_pos_ids_list.extend(vision_llm_pos_ids_list)
-                    new_src_item.extend(
-                        min(t_ntoken_per_chunk, pure_audio_len -
-                            added_audio_len) * [audio_token_id])
-                    audio_start_idx = start_idx if len(
-                        audio_llm_pos_ids_list
-                    ) == 0 else audio_llm_pos_ids_list[-1][0].item() + 1
-                    if min(t_ntoken_per_chunk,
-                           pure_audio_len - added_audio_len) > 0:
-                        audio_llm_pos_ids_list = (torch.arange(
-                            min(t_ntoken_per_chunk, pure_audio_len -
-                                added_audio_len)).expand(3, -1) +
-                                                  audio_start_idx).split(1,
-                                                                         dim=1)
-                    else:
-                        audio_llm_pos_ids_list = []
-                    added_audio_len += min(t_ntoken_per_chunk,
-                                           pure_audio_len - added_audio_len)
-                    llm_pos_ids_list.extend(audio_llm_pos_ids_list)
-                if added_audio_len < pure_audio_len:
-                    new_src_item.extend(
-                        (pure_audio_len - added_audio_len) * [audio_token_id])
-                    audio_llm_pos_ids_list = (
-                        torch.arange(pure_audio_len - added_audio_len).expand(
-                            3, -1) + llm_pos_ids_list[-1].max() + 1).split(
-                                1, dim=1)
-                    llm_pos_ids_list.extend(audio_llm_pos_ids_list)
-                audio_idx += 1
-                video_idx += 1
-            # move to the next token
-            idx += len(new_src_item) - new_src_item_len
-
-        llm_positions = torch.cat(llm_pos_ids_list, dim=1)
-        mrope_position_delta = torch.cat(llm_pos_ids_list,
-                                         dim=1).max() + 1 - len(src_item)
-        llm_positions = llm_positions[:, context_len:seq_len]
-
-        return llm_positions, mrope_position_delta
-
-    @staticmethod
-    def _get_llm_pos_ids_for_vision(
-        start_idx: int,
-        vision_idx: int,
-        spatial_merge_size: int,
-        t_index: list[int],
-        grid_hs: torch.Tensor,
-        grid_ws: torch.Tensor,
-    ) -> torch.Tensor:
-        llm_pos_ids_list = []
-        llm_grid_h = grid_hs[vision_idx] // spatial_merge_size
-        llm_grid_w = grid_ws[vision_idx] // spatial_merge_size
-        h_index = (torch.arange(llm_grid_h).view(1, -1, 1).expand(
-            len(t_index), -1, llm_grid_w).flatten())
-        w_index = (torch.arange(llm_grid_w).view(1, 1, -1).expand(
-            len(t_index), llm_grid_h, -1).flatten())
-        t_index_tensor = torch.Tensor(t_index).to(llm_grid_h.device).view(
-            -1, 1).expand(-1, llm_grid_h * llm_grid_w).long().flatten()
-        _llm_pos_ids = torch.stack([t_index_tensor, h_index, w_index])
-        llm_pos_ids_list.append(_llm_pos_ids + start_idx)
-        llm_pos_ids = torch.cat(llm_pos_ids_list, dim=1)
-        return llm_pos_ids
-
-    @staticmethod
-    def _split_list_into_ranges(lst: torch.Tensor,
-                                interval: int) -> list[list[int]]:
-        ranges: list[list[int]] = [[]
-                                   for _ in range((max(lst) // interval) + 1)]
-        for num in lst:
-            index = num // interval
-            ranges[index].append(num)
-        return ranges
-
-    @staticmethod
-    def get_next_input_positions(
-        mrope_position_delta: int,
-        context_len: int,
-        seq_len: int,
-    ) -> list[list[int]]:
-        return [
-            list(
-                range(context_len + mrope_position_delta,
-                      seq_len + mrope_position_delta)) for _ in range(3)
-        ]
-
-    @staticmethod
-    def get_next_input_positions_tensor(out: np.ndarray, out_offset: int,
-                                        mrope_position_delta: int,
-                                        context_len: int, num_new_tokens: int):
-
-        values = np.arange(mrope_position_delta + context_len,
-                           mrope_position_delta + context_len + num_new_tokens,
-                           dtype=out.dtype)
-        out[:, out_offset:out_offset + num_new_tokens] = values
-
-    @classmethod
-    def omni_get_updates_use_audio_in_video(
-        cls,
-        thinker_config: PretrainedConfig,
-        audio_len: int,
-        video_grid_thw: Union[list[int], torch.Tensor],
-        video_second_per_grid_t: float,
-    ) -> list[int]:
-        """Get video prompt updates when `use_audio_in_video` is True.
-
-        In this case, audio and vision update ids will be split into
-        chunks and interleaved (details in `_omni_get_input_positions_tensor`).
-
-        <|video_bos|><|VIDEO|><|video_eos|> =>
-        <|video_bos|><|audio_bos|>(... chunks ...)<|audio_eos|><|video_eos|>
-        """
-
-        audio_token_id = thinker_config.audio_token_index
-        video_token_id = thinker_config.video_token_index
-        audio_start_token_id = thinker_config.audio_start_token_id
-        audio_end_token_id = thinker_config.audio_end_token_id
-        seconds_per_chunk = thinker_config.seconds_per_chunk
-        spatial_merge_size = thinker_config.vision_config.spatial_merge_size
-        tokens_per_second = getattr(thinker_config.vision_config,
-                                    "tokens_per_second", 25)
-
-        grid_t = video_grid_thw[0]
-        grid_h = video_grid_thw[1]
-        grid_w = video_grid_thw[2]
-        t_ntoken_per_chunk = int(tokens_per_second * seconds_per_chunk)
-        t_index = (torch.arange(grid_t) * video_second_per_grid_t *
-                   tokens_per_second).long()
-        t_index_split_chunk = cls._split_list_into_ranges(
-            t_index, t_ntoken_per_chunk)
-
-        updates = [audio_start_token_id]
-        added_audio_len = 0
-        for t_chunk in t_index_split_chunk:
-            vision_ntoken_per_chunk = len(t_chunk) * grid_h * grid_w // (
-                spatial_merge_size**2)
-            updates.extend([video_token_id] * vision_ntoken_per_chunk)
-
-            audio_chunk_size = min(t_ntoken_per_chunk,
-                                   audio_len - added_audio_len)
-            updates.extend(audio_chunk_size * [audio_token_id])
-            added_audio_len += audio_chunk_size
-        if added_audio_len < audio_len:
-            updates.extend((audio_len - added_audio_len) * [audio_token_id])
-        updates.extend([audio_end_token_id])
-
-        return updates

vllm_omni/model_executor/models/qwen2_5_omni.py

@@ -1,710 +0,0 @@
-from functools import cached_property
-from typing import Iterable, List, Optional, Set, Tuple, Union, NamedTuple, Dict
-
-import os
-import glob
-import numpy as np
-import torch
-import torch.nn as nn
-from transformers.models.qwen2_5_omni.configuration_qwen2_5_omni import (
-    Qwen2_5OmniConfig, Qwen2_5OmniThinkerConfig, Qwen2_5OmniTalkerConfig)
-
-from vllm.config import VllmConfig
-from vllm.logger import init_logger
-from vllm.model_executor.layers.linear import ColumnParallelLinear
-from vllm.model_executor.layers.sampler import SamplerOutput, get_sampler
-from vllm.model_executor.sampling_metadata import SamplingMetadata
-from vllm.model_executor.models.qwen2_5_omni_thinker import (
-    Qwen2_5OmniConditionalGenerationMixin,
-    Qwen2_5OmniThinkerMultiModalProcessor,
-    Qwen2_5OmniThinkerProcessingInfo,
-    Qwen2_5OmniThinkerDummyInputsBuilder)
-# from vllm.model_executor.models.qwen2_code2wav_dit import Qwen2Code2wav
-from vllm.multimodal import MULTIMODAL_REGISTRY
-from vllm.sequence import IntermediateTensors
-
-from vllm.model_executor.models.interfaces import SupportsMultiModal, SupportsPP
-from vllm.model_executor.models.utils import (AutoWeightsLoader, WeightsMapper,
-                    init_vllm_registered_model,
-                    maybe_prefix)
-from vllm.model_executor.model_loader.weight_utils import download_weights_from_hf
-from vllm_omni.model_executor.models.utils import add_prefix_to_loaded_weights
-
-TALKER_CODEC_EOS_TOKEN_ID = 8294
-TALKER_CODEC_BOS_TOKEN_ID = 8293
-
-class OmniOutput(NamedTuple):
-    """Output from the merged Omni model containing both text and audio."""
-    text_hidden_states: torch.Tensor
-    multimodal_outputs: dict = {}
-    intermediate_tensors: Optional[IntermediateTensors] = None
-
-logger = init_logger(__name__)
-
-@MULTIMODAL_REGISTRY.register_processor(
-    Qwen2_5OmniThinkerMultiModalProcessor,
-    info=Qwen2_5OmniThinkerProcessingInfo,
-    dummy_inputs=Qwen2_5OmniThinkerDummyInputsBuilder,
-)
-class Qwen2_5OmniForConditionalGeneration(nn.Module, SupportsMultiModal,
-                                         SupportsPP,
-                                         Qwen2_5OmniConditionalGenerationMixin):
-
-    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
-        super().__init__()
-        self.have_multimodal_outputs = True 
-        config: Qwen2_5OmniConfig = vllm_config.model_config.hf_config
-        quant_config = vllm_config.quant_config
-        multimodal_config = vllm_config.model_config.multimodal_config
-        # keep vllm_config for later submodule init
-        self.vllm_config = vllm_config
-        
-        # Initialize thinker components
-        thinker_config: Qwen2_5OmniThinkerConfig = config.thinker_config
-        self.thinker_config = thinker_config
-        self.multimodal_config = multimodal_config
-        
-        # Initialize talker components
-        talker_config: Qwen2_5OmniTalkerConfig = config.talker_config
-        self.talker_config = talker_config
-        
-
-
-        self.model_stage = vllm_config.model_config.model_stage
-        if self.model_stage=="thinker":
-            # Initialize thinker model (multimodal processing)
-            self.thinker = init_vllm_registered_model(
-                vllm_config=vllm_config,
-                prefix=maybe_prefix(prefix, "thinker"),
-                hf_config=thinker_config,
-                # Use registry architecture key
-                architectures=["Qwen2_5OmniThinkerModel"],
-                )
-            self.model = self.thinker
-            self.talker = None
-            self.token2wav = None
-            
-        elif self.model_stage=="talker":
-            self.thinker = None
-            # Initialize talker model wrapper (handles projection + LM)
-            self.talker = init_vllm_registered_model(
-                vllm_config=vllm_config,
-                prefix=maybe_prefix(prefix, "talker"),
-                hf_config=talker_config,
-                # Use registry architecture key
-                architectures=["Qwen2_5OmniTalkerModel"],
-            )
-            self.talker.init_multi_modal(thinker_config)
-            self.model=self.talker
-            self.token2wav = None
-            
-        elif self.model_stage=="code2wav":
-            self.thinker = None
-            self.talker = None
-            # Initialize token2wav (code->mel->wav) like thinker/talker
-            self.token2wav_config = getattr(config, 'token2wav_config', None)
-            self.token2wav = None
-            if self.token2wav_config is not None:
-                self.token2wav = init_vllm_registered_model(
-                    vllm_config=vllm_config,
-                    prefix=maybe_prefix(prefix, "token2wav"),
-                    hf_config=self.token2wav_config,
-                    architectures=["Qwen2_5OmniToken2WavModel"],
-                )
-            # voice resources (loaded on demand)
-            self._token2wav_conds: Dict[str, torch.Tensor] = {}
-            self._token2wav_ref_mels: Dict[str, torch.Tensor] = {}
-            self.model = self.token2wav
-        else:
-            raise ValueError("Invalid model stage")
-        
-        # Set up intermediate tensors
-        self.make_empty_intermediate_tensors = (
-            self.thinker.make_empty_intermediate_tensors) if self.model_stage=="thinker" else lambda: None
-
-    # -------------------- Device utilities --------------------
-    @staticmethod
-    def _module_device(module: nn.Module) -> torch.device:
-        try:
-            return next(module.parameters()).device
-        except StopIteration:
-            # No parameters; fall back to buffers or cpu
-            for _, buf in module.named_buffers(recurse=True):
-                return buf.device
-            return torch.device("cpu")
-
-    def move_submodules_to_devices(
-        self,
-        *,
-        thinker_device: Optional[Union[str, torch.device]] = None,
-        talker_device: Optional[Union[str, torch.device]] = None,
-        token2wav_device: Optional[Union[str, torch.device]] = None,
-    ) -> None:
-        """Optionally move thinker/talker/token2wav to different devices.
-
-        Example:
-            model.move_submodules_to_devices(
-                thinker_device='cuda:0',
-                talker_device='cuda:1',
-                token2wav_device='cpu',
-            )
-        """
-        if thinker_device is not None and self.thinker is not None:
-            self.thinker.to(thinker_device)
-        if talker_device is not None and self.talker is not None:
-            self.talker.to(talker_device)
-        if token2wav_device is not None and self.token2wav is not None:
-            self.token2wav.to(token2wav_device)
-
-    @cached_property
-    def sampler(self):
-        if hasattr(self.model, "sampler"):
-            return self.model.sampler
-        return get_sampler()
-
-    def get_input_embeddings(
-        self,
-        input_ids: torch.Tensor,
-        multimodal_embeddings=None,
-    ) -> torch.Tensor:
-        if self.model_stage == "code2wav":
-            return torch.zeros_like(input_ids).reshape(-1, 1).repeat(1, self.vllm_config.model_config.get_hidden_size())
-        return self.model.get_input_embeddings(
-            input_ids, multimodal_embeddings)
-       
-
-    def get_multimodal_embeddings(self, **kwargs):
-        # Delegate to thinker model for multimodal processing
-        return self.model.get_multimodal_embeddings(**kwargs)
-    
-    def last_index_of(self, list, value):
-        return len(list) - 1 - list[::-1].index(value)
-
-    def forward(
-        self,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-        inputs_embeds: Optional[torch.Tensor] = None,
-        generate_audio: bool = True,
-        voice_type: str = "Chelsie",
-        codec: Optional[torch.Tensor] = None,
-        sampling_metadata: Optional[SamplingMetadata] = None,
-        logits_index: Optional[int] = None,
-        sampler = None,
-        additional_information: Optional[dict[str, object]] = None,
-        **kwargs: object,
-    ) -> Union[torch.Tensor, IntermediateTensors, OmniOutput]:
-        """
-        Workflow:
-        1) Thinker: multimodal understanding → text hidden states.
-        2) If audio requested and codec not provided, use talker to derive codec.
-        3) If audio requested (or codec provided), use token2wav to synthesize waveform.
-        4) Return text hidden states (and audio when applicable).
-        """        
-        if self.model_stage=="thinker":
-            # Normalize to batched inputs if caller provides 1D/2D unbatched tensors
-            added_batch_dim = False
-            if input_ids is not None and input_ids.ndim == 1:
-                input_ids = input_ids.unsqueeze(0)
-                added_batch_dim = True
-            if positions is not None and positions.ndim == 1:
-                positions = positions.unsqueeze(0)
-                added_batch_dim = True
-            if inputs_embeds is not None and inputs_embeds.ndim == 2:
-                inputs_embeds = inputs_embeds.unsqueeze(0)
-                added_batch_dim = True
-            thinker_dev = self._module_device(self.thinker)
-            
-            #if input_ids is None, set it to a zero tensor, in the length of the same as the embedding seq length
-            if input_ids is None:
-                input_ids = torch.zeros(inputs_embeds.shape[1], dtype=torch.long, device=thinker_dev).unsqueeze(0) #(1, 0)
-                added_batch_dim = True
-
-            # 1) Thinker (ensure inputs on thinker's device)
-            if input_ids is not None and input_ids.device != thinker_dev:
-                input_ids = input_ids.to(thinker_dev)
-            if positions is not None and positions.device != thinker_dev:
-                positions = positions.to(thinker_dev)
-            if inputs_embeds is not None and inputs_embeds.device != thinker_dev:
-                inputs_embeds = inputs_embeds.to(thinker_dev)
-            # Run thinker
-            thinker_output = self.thinker(
-                input_ids=input_ids,
-                positions=positions[0],
-                intermediate_tensors=intermediate_tensors,
-                inputs_embeds=inputs_embeds,
-                **kwargs,
-            )
-
-            if isinstance(thinker_output, tuple):
-                embeds, text_hidden_states = thinker_output
-            else:
-                text_hidden_states = thinker_output
-
-            # Text-only path
-            return OmniOutput(
-                text_hidden_states=text_hidden_states.squeeze(0) if added_batch_dim else text_hidden_states,
-                multimodal_outputs=None
-            )
-
-        # 2) Talker (if codec not provided)
-        if self.model_stage=="talker":
-            
-            if input_ids is None and additional_information is None:
-                input_ids = torch.zeros(inputs_embeds.shape[0], dtype=torch.long, device=inputs_embeds.device)
-                additional_information = {}
-                self.thinker_reply_part = torch.zeros_like(inputs_embeds)
-                is_profile = True
-            else:
-                is_profile = False
-
-            if input_ids is not None and additional_information is not None and not is_profile:
-                # read from additional_information dict
-                thinker_result = None
-                if additional_information is not None and isinstance(additional_information, dict):
-                    thinker_result = additional_information.get("thinker_result")
-                    prompt_embeds = additional_information.get("prompt_embeds")
-                    prompt_token_ids = additional_information.get("prompt_token_ids")
-                    thinker_output_token_ids = additional_information.get("thinker_output_token_ids")
-                else:
-                    thinker_result = torch.zeros_like(inputs_embeds)
-                    prompt_embeds = torch.zeros_like(inputs_embeds)
-                    prompt_token_ids = torch.zeros(inputs_embeds.shape[0], dtype=torch.int64, device=inputs_embeds.device)
-                    thinker_output_token_ids = torch.zeros(inputs_embeds.shape[0], dtype=torch.int64, device=inputs_embeds.device)
-                    
-                if thinker_result is None:
-                    thinker_result = torch.zeros_like(inputs_embeds)
-                self.thinker_reply_part = thinker_result.squeeze(0)
-                if self.thinker_reply_part.shape[1] > 1:
-                    self.thinker_reply_part = self.thinker_reply_part[1:, :]
-                input_ids, inputs_embeds = self._thinker_to_talker_prefill(
-                    voice_type=voice_type,
-                    output_prompt_embeds=thinker_result,
-                    output_token_ids = thinker_output_token_ids,
-                    thinker_prompt_embeds=prompt_embeds, 
-                    prompt_token_ids = prompt_token_ids,
-                )
-            elif not is_profile:
-                input_ids, inputs_embeds = \
-                self._thinker_to_talker_decode_one_step(
-                    output_prompt_embeds=self.thinker_reply_part[:1] if self.thinker_reply_part.shape[0]>=1 \
-                        else torch.zeros(1, self.thinker_reply_part.shape[1]).to(self._module_device(self.model)).to(torch.bfloat16)+(-1.25*2**(-123)),
-                    output_token_ids=input_ids,
-                )
-
-                if self.thinker_reply_part.shape[0] >=1:
-                    self.thinker_reply_part = self.thinker_reply_part[1:, :]
-
-            with torch.inference_mode():
-                talker_hidden = self.talker(
-                    input_ids=input_ids,
-                    positions=positions[0],
-                    inputs_embeds=inputs_embeds,
-
-                )
-            
-            return OmniOutput(
-                text_hidden_states=talker_hidden,
-                multimodal_outputs=None
-            )
-
-        if self.model_stage=="code2wav":
-            code = input_ids if input_ids is not None else torch.zeros(inputs_embeds.shape[0], dtype=torch.long, device=inputs_embeds.device)
-            audio_tensor = self.generate_audio(code[:-1] if code[-1]==TALKER_CODEC_EOS_TOKEN_ID else code, voice_type)
-            return OmniOutput(
-                text_hidden_states = None,
-                multimodal_outputs = {
-                    "audio": audio_tensor
-                }
-            )
-
-        return OmniOutput(
-            text_hidden_states=torch.cat(
-                [
-                    torch.zeros([inputs_embeds.shape[0],self.talker.config.hidden_size], dtype=torch.bfloat16).to(self._module_device(self.model)),
-                    self.talker.thinker_to_talker_proj(self.talker.get_input_embeddings(torch.tensor([TALKER_CODEC_BOS_TOKEN_ID,TALKER_CODEC_EOS_TOKEN_ID]).to(torch.bfloat16).to(self._module_device(self.model))))[0]
-                ],
-                dim=0),
-            multimodal_outputs=None
-        )
-    
-    def generate_audio(self, code, voice_type):
-        token2wav_dev = self._module_device(self.token2wav)
-        if isinstance(code, torch.Tensor):
-            code_tensor = code.to(dtype=torch.long, device=token2wav_dev)
-        else:
-            code_tensor = torch.as_tensor(code, dtype=torch.long, device=token2wav_dev)
-        if code_tensor.ndim == 2 and code_tensor.shape[0] == 1:
-            code_tensor = code_tensor.squeeze(0)
-
-        audio_tensor = self._codec_to_audio(code_tensor, voice_type)
-
-        return audio_tensor
-
-    def _load_talker_embedding(
-            self,
-    ) -> torch.nn.Embedding:
-        return self.talker.language_model.model.embed_tokens
-
-    def _init_special_tokens_embeddings(
-        self,
-    ):
-        # talker embeddings
-        self.talker_embedding = self._load_talker_embedding()
-
-        # embed_text_bos_token
-        self.tts_text_spk_token_ids = {
-            # M02: Male voice with standard Mandarin and a slight northern accent
-            'm02': 151870,
-            'Ethan': 151870,
-
-            # F030: Your anime-styled virtual girlfriend
-            'f030': 151872,
-            'Chelsie': 151872,
-        }
-        self.default_tts_text_spk_type = list(
-            self.tts_text_spk_token_ids.keys())[0]
-        self.tts_text_spk_token_ids['prefix_caching'] = 151870
-
-        talker_hf_config = self.talker_config
-        if hasattr(talker_hf_config, 'talker_config'):
-            talker_hf_config = talker_hf_config.talker_config
-
-        self.embed_text_bos_token = self.thinker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_text_start_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        self.embed_text_spk_tokens = {
-            key:
-            self.thinker_embedding(
-                torch.tensor(
-                    [value],
-                    dtype=torch.long,
-                    device=self._module_device(self.talker),
-                ))
-            for key, value in self.tts_text_spk_token_ids.items()
-        }
-        self.embed_text_eos_token = self.thinker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_text_end_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        self.embed_text_pad_token = self.thinker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_text_pad_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        self.embed_codec_bos_token = self.talker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_codec_start_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        self.embed_codec_eos_token = self.talker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_codec_end_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        self.embed_codec_pad_token = self.talker_embedding(
-            torch.tensor(
-                [talker_hf_config.tts_codec_pad_token_id],
-                dtype=torch.long,
-                device=self._module_device(self.talker),
-            ))
-        return set(["thinker_embedding.weight", "talker_embedding.weight"])
-
-    def _get_embed_text_spk_token(self, voice_type: str):
-        if voice_type not in self.embed_text_spk_tokens:
-            return self.embed_text_bos_token
-        return self.embed_text_spk_tokens[voice_type]
-
-    def _get_text_spk_token_id(self, voice_type: str):
-        talker_hf_config = self.talker_config
-        if hasattr(talker_hf_config, 'talker_config'):
-            talker_hf_config = talker_hf_config.talker_config
-
-        if voice_type not in self.tts_text_spk_token_ids:
-            return talker_hf_config.tts_text_start_token_id
-        return self.tts_text_spk_token_ids[voice_type]
-
-    def _thinker_to_talker_prefill(
-        self,
-        voice_type: str,
-        output_prompt_embeds,
-        output_token_ids,
-        thinker_prompt_embeds,
-        prompt_token_ids
-    ):
-
-        talker_hf_config = self.talker_config
-        if hasattr(talker_hf_config, 'talker_config'):
-            talker_hf_config = talker_hf_config.talker_config
-
-        # if len(output.outputs[0].token_ids) == 2:
-            # issue request
-        prompt_embeds = torch.cat([
-            thinker_prompt_embeds,
-            self._get_embed_text_spk_token(voice_type) +
-            self.embed_codec_pad_token,
-            output_prompt_embeds[:1] + self.embed_codec_bos_token,
-        ],
-                                    dim=0)
-
-        prompt_token_ids_processed = prompt_token_ids + [
-            talker_hf_config.tts_codec_pad_token_id,
-            output_token_ids[0],
-
-        ]
-        input_tokens_len = len(prompt_token_ids_processed)
-        # the code below is from model runner in Qwen, may need to further discuss later
-        if input_tokens_len > 2:
-            prompt_token_ids_processed = (
-                [self.talker_config.tts_codec_mask_token_id] *
-                (input_tokens_len - 2) + [
-                    self.talker_config.tts_codec_pad_token_id,
-                    self.talker_config.tts_codec_start_token_id
-                ])
-        else:
-            prompt_token_ids_processed = [
-                self.talker_config.tts_codec_pad_token_id,
-                self.talker_config.tts_codec_start_token_id,
-            ][-input_tokens_len:]
-        if isinstance(prompt_token_ids_processed,list):
-            prompt_token_ids_processed = torch.Tensor(prompt_token_ids_processed).to(torch.int64).to(self._module_device(self.talker))
-        return prompt_token_ids_processed, prompt_embeds
-            
-    def _thinker_to_talker_decode_one_step(
-        self,
-        output_prompt_embeds,
-        output_token_ids,
-    ):
-        processed_output_token_embeds = output_prompt_embeds + self.talker.get_input_embeddings(output_token_ids) #for decode
-        return output_token_ids, processed_output_token_embeds
-
-    def compute_logits(
-        self,
-        hidden_states: Union[torch.Tensor, OmniOutput],
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[torch.Tensor]:
-        # Handle OmniOutput type
-        if isinstance(hidden_states, OmniOutput):
-            hidden_states = hidden_states.text_hidden_states
-        
-        # Use thinker model for logits computation
-        return self.model.compute_logits(hidden_states, sampling_metadata)
-
-    def sample(
-        self,
-        logits: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[SamplerOutput]:
-        # Use thinker model for sampling
-        return self.model.sample(logits, sampling_metadata)
-
-    def generate_speech(self, text_tokens: torch.Tensor, voice_type: str = "default"):
-        """
-        Generate speech from text tokens using the talker and token2wav models.
-        This method is kept for backward compatibility and direct speech generation.
-        
-        Args:
-            text_tokens: Text tokens from thinker model
-            voice_type: Voice type for speech generation
-            
-        Returns:
-            Audio tensor
-        """
-        # Generate codec tokens using talker model
-        talker_output = self.talker(
-            input_ids=None,
-            positions=None,
-            inputs_embeds=text_tokens
-        )
-        
-        # Convert talker output to codec tokens
-        codec_tokens = self._convert_to_codec_tokens(talker_output)
-        
-        # Generate audio using token2wav model
-        return self._codec_to_audio(codec_tokens, voice_type=voice_type)
-
-
-    def _convert_to_codec_tokens(self, talker_output: torch.Tensor, sampling_metadata: SamplingMetadata) -> torch.Tensor:
-        """
-        Reference (HF): use the talker's codec head to obtain logits, suppress BOS,
-        then greedily select the next codec token for the current step.
-        """
-        with torch.inference_mode():
-            logits = self.talker.compute_logits(talker_output, None)
-            if logits is None:
-                return torch.zeros((talker_output.size(0), 0), dtype=torch.long, device=talker_output.device)
-
-            # Suppress only codec_bos, consistent with HF generate's suppress_tokens behavior
-            bos_id = None
-            if hasattr(self, 'talker_config') and hasattr(self.talker_config, 'tts_codec_start_token_id'):
-                bos_id = int(getattr(self.talker_config, 'tts_codec_start_token_id'))
-            if bos_id is not None:
-                logits[..., bos_id] = -1e9
-
-            # Take the distribution at the last step and select greedily
-            next_id = self.talker.sample(logits, sampling_metadata).sampled_token_ids
-            return next_id.to(dtype=torch.long)
-
-    def _init_token2wav_model(self):
-        """Initialize speaker resources if provided; model is constructed in __init__."""
-        if self.token2wav is None or self.token2wav_config is None:
-            return
-        device = 'cuda' if torch.cuda.is_available() else 'cpu'
-        # optional speaker resources
-        conds = getattr(self.token2wav_config, 'conds', None)
-        ref_mels = getattr(self.token2wav_config, 'ref_mels', None)
-        if isinstance(conds, dict) and isinstance(ref_mels, dict):
-            self._token2wav_conds = {k: torch.as_tensor(v, device=device) for k, v in conds.items()}
-            self._token2wav_ref_mels = {k: torch.as_tensor(v, device=device) for k, v in ref_mels.items()}
-        # legacy: load from directory if provided
-        model_path = self.vllm_config.model_config.model
-        if isinstance(model_path, str) and os.path.isdir(model_path):
-            spk_pt = os.path.join(model_path, 'spk_dict.pt')
-            if os.path.exists(spk_pt):
-                data = torch.load(spk_pt, map_location=device)
-                for key, value in data.items():
-                    self._token2wav_conds[key] = value["cond"].to(device)
-                    self._token2wav_ref_mels[key] = value["ref_mel"].to(device)
-            else:
-                # legacy npy inputs
-                for f in sorted(glob.glob(os.path.join(model_path, 'inputs', '*spk_emb.npy'))):
-                    key = os.path.basename(f).split('_')[0].lower()
-                    self._token2wav_conds[key] = torch.as_tensor(np.load(f), device=device)
-                for f in sorted(glob.glob(os.path.join(model_path, 'inputs', '*ref_mel.npy'))):
-                    key = os.path.basename(f).split('_')[0].lower()
-                    self._token2wav_ref_mels[key] = torch.as_tensor(np.load(f), device=device)
-
-    def _codec_to_audio(self, codec_tokens: torch.Tensor, voice_type: str = "default") -> Optional[torch.Tensor]:
-        if self.token2wav is None:
-            self._init_token2wav_model()
-        if self.token2wav is None:
-            return None
-        # Normalize voice type
-        voice = (voice_type or 'default')
-        # Resolve cond / ref_mel if provided
-        cond = None
-        ref_mel = None
-        if voice in self._token2wav_conds and voice in self._token2wav_ref_mels:
-            cond = self._token2wav_conds[voice]
-            ref_mel = self._token2wav_ref_mels[voice]
-        # Fallback: create dummy cond/ref_mel if not provided
-        token2wav_dev = self._module_device(self.token2wav)
-        if cond is None:
-            cond = torch.zeros((1, self.token2wav_config.dit_config.enc_emb_dim), device=token2wav_dev, dtype=torch.float32)
-        if ref_mel is None:
-            ref_mel = torch.zeros((1, 300, self.token2wav_config.dit_config.mel_dim), device=token2wav_dev, dtype=torch.float32)
-
-        # Ensure codec is (1, T) long tensor on correct device
-        if isinstance(codec_tokens, torch.Tensor):
-            codec = codec_tokens.to(dtype=torch.long, device=token2wav_dev)
-            if codec.ndim == 1:
-                codec = codec.unsqueeze(0)
-        else:
-            codec = torch.as_tensor(codec_tokens, dtype=torch.long, device=token2wav_dev).unsqueeze(0)
-
-        # Streaming with chunked process and boundary alignment (rely on token2wav.process_chunk)
-        factor = getattr(self.token2wav.token2wav.factor, 'factor', 2)
-        chunk_size = 48
-        mel_dim = getattr(self.token2wav.token2wav.code2wav_dit_model, 'mel_dim', self.token2wav_config.dit_config.mel_dim)
-        total_mel = int(codec.shape[1] * factor)
-        steps = 10
-
-        # Prepare initial noise for the whole sequence
-        y_all = torch.randn((1, total_mel, mel_dim), dtype=ref_mel.dtype, device=token2wav_dev)
-
-        logger.info(f"Currently, we do not use the chunked process, we only use the token2wav.process_chunk for the whole sequence.\
-                    The stream mode will be implemented in the future.")
-
-        chunk_ends = []
-        for i in range(codec.shape[1]):
-            chunk_code_length = i * 2 - 24
-            finished = i==(codec.shape[1]-1)
-            if (chunk_code_length > 0 and
-                chunk_code_length % chunk_size == 0) or finished:
-                chunk_ends.append(i)
-
-        # Number of chunks in mel domain
-        prev_generated = None
-        wav_chunks: list = []
-        prev_id = 0
-
-        with torch.inference_mode():
-            for n,i in enumerate([0]):
-                finished = (i == codec.shape[1] - 1)
-                _, audio_chunk = self.token2wav.process_chunk(
-                    conditioning=cond,
-                    reference_mel=ref_mel,
-                    codec_all=codec,
-                    y_all=y_all,
-                    i=n,
-                    steps=steps,
-                    prev_generated=prev_generated if prev_generated is not None else [],
-                    finished=True,
-                )
-                prev_generated = audio_chunk
-                wav_chunks.append(audio_chunk.detach().cpu().numpy())
-                prev_id = i
-
-        if len(wav_chunks) == 0:
-            return torch.zeros(0, device=token2wav_dev)
-
-        waveform = np.concatenate(wav_chunks)
-        return torch.as_tensor(waveform, device=token2wav_dev)
-
-    def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]) -> Set[str]:
-        """Load weights for all components of the omni model."""
-        loaded_weights = set()
-        thinker_weights = []
-        talker_weights = []
-        token2wav_weights = []
-        for k, v in weights:
-            if k.startswith('thinker.'):
-                thinker_weights.append((k, v))
-            elif k.startswith('talker.'):
-                talker_weights.append((k, v))
-            elif k.startswith('token2wav.'):
-                token2wav_weights.append((k, v))
-            else:
-                raise ValueError(f"Unknown weight prefix: {k}")
-
-        # Load thinker weights
-        if self.thinker:
-            if thinker_weights:
-                thinker_loaded = self.thinker.load_weights(thinker_weights)
-            else:
-                thinker_loaded = set([k for k,v in thinker_weights])
-            thinker_loaded = add_prefix_to_loaded_weights(thinker_loaded, 'thinker')
-            loaded_weights.update(thinker_loaded)
-
-        
-        # Load talker weights
-        if talker_weights and self.talker is not None:
-            # Map talker weights to appropriate components
-            if self.thinker is None:
-                thinker_embedding_weights = [w for n, w in thinker_weights if n == "thinker.model.embed_tokens.weight"]
-                if thinker_embedding_weights:
-                    self.thinker_embedding = nn.Embedding(thinker_embedding_weights[0].shape[0], thinker_embedding_weights[0].shape[1])
-                    self.thinker_embedding.weight = nn.Parameter(thinker_embedding_weights[0].to(self._module_device(self.talker)))
-            talker_loaded = self.talker.load_weights(talker_weights)
-            talker_loaded = add_prefix_to_loaded_weights(talker_loaded, 'talker')
-            loaded_weights.update(talker_loaded)
-            loaded_weights.update(self._init_special_tokens_embeddings())
-            
-        
-        # Load token2wav weights (if any)
-        if token2wav_weights and self.token2wav is not None:
-            self._init_token2wav_model()
-            t2w_loaded = self.token2wav.load_weights(token2wav_weights, os.path.join(self.vllm_config.model_config.model, "spk_dict.pt"))
-            t2w_loaded = add_prefix_to_loaded_weights(t2w_loaded, 'token2wav')
-            loaded_weights.update(t2w_loaded)
-        
-        return loaded_weights

vllm_omni/model_executor/models/qwen2_5_omni_talker.py

@@ -1,239 +0,0 @@
-from functools import cached_property
-from typing import Iterable, List, Optional, Set, Tuple, Union
-
-import torch
-import torch.nn as nn
-from transformers.models.qwen2_5_omni.configuration_qwen2_5_omni import Qwen2_5OmniTalkerConfig
-from transformers.models.qwen2_5_omni.modeling_qwen2_5_omni import Qwen2_5OmniAudioEncoder
-
-from vllm.attention import AttentionMetadata
-from vllm.config import VllmConfig
-from vllm.forward_context import ForwardContext, get_forward_context
-from vllm.model_executor.layers.linear import ColumnParallelLinear
-from vllm.model_executor.layers.sampler import SamplerOutput, get_sampler
-from vllm.model_executor.sampling_metadata import SamplingMetadata
-from vllm.model_executor.models.qwen2_5_omni_thinker import (
-    Qwen2_5OmniConditionalGenerationMixin,
-    Qwen2_5OmniThinkerMultiModalProcessor,
-    Qwen2_5OmniThinkerProcessingInfo,
-    Qwen2_5OmniThinkerDummyInputsBuilder)
-from vllm.multimodal import MULTIMODAL_REGISTRY
-from vllm.logger import init_logger
-from vllm.sequence import IntermediateTensors
-
-from vllm.model_executor.models.interfaces import SupportsMultiModal, SupportsPP, MultiModalEmbeddings
-from vllm.model_executor.models.utils import (AutoWeightsLoader, WeightsMapper,
-                    init_vllm_registered_model,
-                    maybe_prefix,
-                    merge_multimodal_embeddings)
-from vllm.model_executor.models.qwen2_5_vl import Qwen2_5_VisionTransformer
-
-
-@MULTIMODAL_REGISTRY.register_processor(
-    Qwen2_5OmniThinkerMultiModalProcessor,
-    info=Qwen2_5OmniThinkerProcessingInfo,
-    dummy_inputs=Qwen2_5OmniThinkerDummyInputsBuilder,
-)
-class Qwen2_5OmniTalkerForConditionalGeneration(nn.Module, SupportsMultiModal,
-                                                SupportsPP,
-                                                Qwen2_5OmniConditionalGenerationMixin):
-    logger = init_logger(__name__)
-    # Align to thinker-style static mapper for clarity
-    hf_to_vllm_mapper = WeightsMapper(
-        orig_to_new_prefix={
-            # text LM head/body in talker
-            "talker.codec_head.": "language_model.lm_head.",
-            "talker.model.": "language_model.model.",
-            # projection weights
-            "talker.thinker_to_talker_proj.": "thinker_to_talker_proj.",
-            # fallback root
-            "talker.": "",
-        })
-
-    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
-        super().__init__()
-        config: Qwen2_5OmniTalkerConfig = vllm_config.model_config.hf_config
-        quant_config = vllm_config.quant_config
-
-        self.prefix = prefix
-        self.quant_config = quant_config
-
-        if hasattr(config, "talker_config"):
-            self.config = config.talker_config
-            vllm_config.model_config.hf_text_config = \
-                    vllm_config.model_config.hf_config.talker_config
-        else:
-            self.config = config
-
-        self.thinker_to_talker_proj = ColumnParallelLinear(
-            self.config.embedding_size,
-            self.config.hidden_size,
-            bias=True,
-            gather_output=True,
-            skip_bias_add=False,
-            quant_config=quant_config,
-        )
-        self.language_model = init_vllm_registered_model(
-            vllm_config=vllm_config,
-            prefix=maybe_prefix(prefix, "language_model"),
-            hf_config=getattr(self.config, 'text_config', self.config),
-            architectures=["Qwen2ForCausalLM_old"],
-        )
-        self.make_empty_intermediate_tensors = (
-            self.language_model.make_empty_intermediate_tensors)
-    
-    def init_multi_modal(self, thinker_config):
-        self.audio_tower = Qwen2_5OmniAudioEncoder(thinker_config.audio_config)
-        self.visual = Qwen2_5_VisionTransformer(
-            vision_config=thinker_config.vision_config,
-            norm_eps=getattr(thinker_config.text_config, "rms_norm_eps", 1e-6),
-            quant_config=self.quant_config,
-            prefix=maybe_prefix(self.prefix, "visual"),
-        )
-
-    def get_language_model(self) -> torch.nn.Module:
-        return self.language_model
-
-    @cached_property
-    def sampler(self):
-        if hasattr(self.language_model, "sampler"):
-            return self.language_model.sampler
-
-        return get_sampler()
-
-    def get_input_embeddings(
-        self,
-        input_ids: torch.Tensor,
-        multimodal_embeddings: Optional[MultiModalEmbeddings] = None,
-    ) -> torch.Tensor:
-        inputs_embeds = self.language_model.get_input_embeddings(input_ids)
-        if multimodal_embeddings is not None \
-            and len(multimodal_embeddings) != 0:
-
-            # TODO (ywang96): support overlapping modalitiy embeddings so that
-            # `use_audio_in_video` will work on V1.
-            inputs_embeds = merge_multimodal_embeddings(
-                input_ids, inputs_embeds, multimodal_embeddings, [
-                    self.config.image_token_index,
-                    self.config.video_token_index,
-                    self.config.audio_token_index
-                ])
-        return inputs_embeds
-
-    def forward(
-        self,
-        input_ids: torch.Tensor = None,
-        positions: torch.Tensor = None,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-        inputs_embeds: Optional[torch.Tensor] = None,
-        **kwargs: object,
-    ) -> Union[torch.Tensor, IntermediateTensors]:
-        assert input_ids is not None or inputs_embeds is not None, "input_ids or inputs_embeds must be provided"
-        forward_context: ForwardContext = get_forward_context()
-        attn_metadata: AttentionMetadata = forward_context.attn_metadata
-
-        if intermediate_tensors is not None:
-            inputs_embeds = None
-        elif inputs_embeds is None:
-            # for profile_run:
-            inputs_embeds = self.get_input_embeddings(input_ids)
-
-        input_ids = None
-
-        # projection
-        inputs_embeds, _ = self.thinker_to_talker_proj(inputs_embeds)
-
-        hidden_states = self.language_model.model(input_ids,
-                                                  positions,
-                                                  intermediate_tensors,
-                                                  inputs_embeds=inputs_embeds)
-        return hidden_states
-
-    def compute_logits(
-        self,
-        hidden_states: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[torch.Tensor]:
-        return self.language_model.compute_logits(hidden_states,
-                                                  sampling_metadata)
-
-    def sample(
-        self,
-        logits: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[SamplerOutput]:
-        return self.language_model.sample(logits, sampling_metadata)
-
-    def load_weights(self, weights: Iterable[Tuple[str,
-                                                   torch.Tensor]]) -> Set[str]:
-        loader = AutoWeightsLoader(
-            self,
-            skip_prefixes=['thinker.', 'token2wav.'],
-        )
-        loaded = loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
-        # Log load summary
-        try:
-            total_bytes = 0
-            for name, param in self.named_parameters():
-                if param is not None and param.data is not None:
-                    total_bytes += param.data.numel() * param.data.element_size()
-            device = next(self.parameters()).device
-            self.logger.info(
-                "[Model Loaded] name=%s, success=%s, size=%.2f MB, device=%s",
-                self.__class__.__name__, True, total_bytes / (1024**2), str(device))
-        except Exception:
-            pass
-        multi_model_weights = set()
-        for name, param in self.visual.named_parameters():
-            multi_model_weights.add("visual."+name)
-        for name, param in self.audio_tower.named_parameters():
-            multi_model_weights.add("audio_tower."+name)
-        loaded.update(multi_model_weights)
-        return loaded
-    
-    def _parse_and_validate_multimodal_inputs(self, **kwargs: object) -> dict:
-        mm_input_by_modality = {}
-
-        # Preserve the order of modalities if there are multiple of them
-        # from the order of kwargs.
-        for input_key in kwargs:
-            if input_key in ("pixel_values", "image_embeds"
-                             ) and "image" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "image"] = self._parse_and_validate_image_input(**kwargs)
-            if input_key in ("pixel_values_videos", "video_embeds"
-                             ) and "video" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "video"] = self._parse_and_validate_video_input(**kwargs)
-            if input_key in ("input_audio_features"
-                             ) and "audio" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "audio"] = self._parse_and_validate_audio_input(**kwargs)
-        return mm_input_by_modality
-
-    def get_multimodal_embeddings(self,
-                                  **kwargs: object) -> MultiModalEmbeddings:
-
-        mm_input_by_modality = self._parse_and_validate_multimodal_inputs(
-            **kwargs)
-        if not mm_input_by_modality:
-            return []
-
-        # The result multimodal_embeddings is tuple of tensors, with each
-        # tensor correspoending to a multimodal data item (image or video).
-        multimodal_embeddings: tuple[torch.Tensor, ...] = ()
-
-        # NOTE: It is important to iterate over the keys in this dictionary
-        # to preserve the order of the modalities.
-        for modality in mm_input_by_modality:
-            multimodal_input = mm_input_by_modality[modality]
-            if modality == "image":
-                vision_embeddings = self._process_image_input(multimodal_input)
-                multimodal_embeddings += vision_embeddings
-            if modality == "video":
-                video_embeddings = self._process_video_input(multimodal_input)
-                multimodal_embeddings += video_embeddings
-            if modality == "audio":
-                audio_embeddings = self._process_audio_input(multimodal_input)
-                multimodal_embeddings += audio_embeddings
-        return multimodal_embeddings

vllm_omni/model_executor/models/qwen2_5_omni_thinker.py

@@ -1,889 +0,0 @@
-from collections.abc import Iterable, Mapping, Sequence
-from copy import copy
-from functools import partial
-from typing import Any, Optional, Union
-
-import torch
-import torch.nn as nn
-from transformers.feature_extraction_utils import BatchFeature
-from transformers.models.qwen2_5_omni.configuration_qwen2_5_omni import (
-    Qwen2_5OmniConfig, Qwen2_5OmniThinkerConfig)
-from transformers.models.qwen2_5_omni.modeling_qwen2_5_omni import (
-    Qwen2_5OmniAudioEncoder)
-from transformers.models.qwen2_5_omni.processing_qwen2_5_omni import (
-    Qwen2_5OmniProcessor)
-from transformers.models.whisper import WhisperFeatureExtractor
-
-from vllm.config import VllmConfig
-from vllm.logger import init_logger
-
-from vllm.model_executor.models.qwen2_5_vl import (
-    Qwen2_5_VisionTransformer, Qwen2_5_VLImageEmbeddingInputs,
-    Qwen2_5_VLImageInputs, Qwen2_5_VLImagePixelInputs,
-    Qwen2_5_VLProcessingInfo, Qwen2_5_VLVideoEmbeddingInputs,
-    Qwen2_5_VLVideoInputs, Qwen2_5_VLVideoPixelInputs)
-from vllm.model_executor.models.qwen2_audio import (
-    Qwen2AudioInputs, Qwen2AudioProcessingInfo,
-    _get_feat_extract_output_lengths)
-from vllm.model_executor.models.qwen2_vl import Qwen2VLMultiModalDataParser
-from vllm.model_executor.sampling_metadata import SamplingMetadata
-from vllm.multimodal import MULTIMODAL_REGISTRY
-from vllm.multimodal.inputs import (ImageItem, ModalityData,
-                                    MultiModalDataDict, MultiModalFieldConfig,
-                                    MultiModalKwargs, NestedTensors)
-from vllm.multimodal.parse import (AudioProcessorItems, DictEmbeddingItems,
-                                   ModalityDataItems, MultiModalDataItems,
-                                   MultiModalDataParser)
-from vllm.multimodal.processing import (BaseMultiModalProcessor,
-                                        PlaceholderFeaturesInfo,
-                                        PromptReplacement, PromptUpdate)
-from vllm.multimodal.profiling import BaseDummyInputsBuilder
-from vllm.sequence import IntermediateTensors
-from vllm.transformers_utils.tokenizer import decode_tokens, encode_tokens
-
-from vllm.model_executor.models.interfaces import MultiModalEmbeddings, SupportsMultiModal, SupportsPP
-from vllm.model_executor.models.utils import (AutoWeightsLoader, WeightsMapper,
-                    init_vllm_registered_model, maybe_prefix,
-                    merge_multimodal_embeddings)
-
-from vllm_omni.model_executor.layers.mrope import MRotaryEmbedding
-
-try:
-    import flash_attn
-except (ImportError, ModuleNotFoundError):
-    flash_attn = None
-
-logger = init_logger(__name__)
-
-
-def _qwen2_5_omni_thinker_field_config(hf_inputs: Mapping[str, torch.Tensor]):
-    audio_feature_lengths = hf_inputs.get("audio_feature_lengths",
-                                          torch.empty((0, )))
-
-    image_grid_thw = hf_inputs.get("image_grid_thw", torch.empty((0, 3)))
-    image_grid_sizes = image_grid_thw.prod(-1)
-
-    video_grid_thw = hf_inputs.get("video_grid_thw", torch.empty((0, 3)))
-    video_grid_sizes = video_grid_thw.prod(-1)
-
-    return dict(
-        input_audio_features=MultiModalFieldConfig.flat_from_sizes(
-            "audio", audio_feature_lengths, dim=1),
-        feature_attention_mask=MultiModalFieldConfig.batched("audio"),
-        audio_feature_lengths=MultiModalFieldConfig.batched("audio"),
-        pixel_values=MultiModalFieldConfig.flat_from_sizes(
-            "image", image_grid_sizes),
-        image_embeds=MultiModalFieldConfig.flat_from_sizes(
-            "image", image_grid_sizes),
-        image_grid_thw=MultiModalFieldConfig.batched("image"),
-        pixel_values_videos=MultiModalFieldConfig.flat_from_sizes(
-            "video", video_grid_sizes),
-        video_embeds=MultiModalFieldConfig.flat_from_sizes(
-            "video", video_grid_sizes),
-        video_grid_thw=MultiModalFieldConfig.batched("video"),
-        second_per_grid_ts=MultiModalFieldConfig.batched("video"),
-    )
-
-
-class Qwen2_5OmniThinkerMultiModalDataParser(Qwen2VLMultiModalDataParser):
-
-    def _parse_audio_data(
-        self,
-        data: Union[dict[str, torch.Tensor], ModalityData[ImageItem]],
-    ) -> ModalityDataItems[Any, Any]:
-        if isinstance(data, dict):
-            return DictEmbeddingItems(
-                data,
-                modality="audio",
-                required_fields={
-                    "input_audio_features", "audio_feature_lengths"
-                },
-                fields_factory=_qwen2_5_omni_thinker_field_config,
-            )
-
-        return super()._parse_audio_data(data)
-
-
-class Qwen2_5OmniThinkerProcessingInfo(Qwen2AudioProcessingInfo,
-                                       Qwen2_5_VLProcessingInfo):
-
-    def get_hf_config(self):
-        return self.ctx.get_hf_config(Qwen2_5OmniConfig).thinker_config
-
-    def get_hf_processor(self, **kwargs: object) -> Qwen2_5OmniProcessor:
-        return self.ctx.get_hf_processor(
-            Qwen2_5OmniProcessor,
-            use_fast=kwargs.pop("use_fast", True),
-            **kwargs,
-        )
-
-    def get_feature_extractor(self, **kwargs: object):
-        hf_processor = self.get_hf_processor(**kwargs)
-        feature_extractor = hf_processor.feature_extractor  # type: ignore
-        assert isinstance(feature_extractor, WhisperFeatureExtractor)
-        return feature_extractor
-
-    def get_supported_mm_limits(self) -> Mapping[str, Optional[int]]:
-        return {"audio": None, "image": None, "video": None}
-
-
-class Qwen2_5OmniThinkerDummyInputsBuilder(
-        BaseDummyInputsBuilder[Qwen2_5OmniThinkerProcessingInfo]):
-
-    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
-        num_audios = mm_counts.get("audio", 0)
-        num_images = mm_counts.get("image", 0)
-        num_videos = mm_counts.get("video", 0)
-
-        hf_processor = self.info.get_hf_processor()
-
-        audio_token: str = hf_processor.audio_token
-        image_token: str = hf_processor.image_token
-        video_token: str = hf_processor.video_token
-
-        return (audio_token * num_audios + image_token * num_images +
-                video_token * num_videos)
-
-    def get_dummy_mm_data(
-        self,
-        seq_len: int,
-        mm_counts: Mapping[str, int],
-    ) -> MultiModalDataDict:
-        num_audios = mm_counts.get("audio", 0)
-        num_images = mm_counts.get("image", 0)
-        num_videos = mm_counts.get("video", 0)
-
-        feature_extractor = self.info.get_feature_extractor()
-
-        target_audio_length = min(
-            feature_extractor.chunk_length,
-            30,
-        ) * feature_extractor.sampling_rate
-        target_width, target_height = \
-            self.info.get_image_size_with_most_features()
-        target_num_frames = \
-            self.info.get_num_frames_with_most_features(seq_len, mm_counts)
-
-        mm_data = {
-            "audio":
-            self._get_dummy_audios(length=target_audio_length,
-                                   num_audios=num_audios),
-            "image":
-            self._get_dummy_images(width=target_width,
-                                   height=target_height,
-                                   num_images=num_images),
-            "video":
-            self._get_dummy_videos(width=target_width,
-                                   height=target_height,
-                                   num_frames=target_num_frames,
-                                   num_videos=num_videos),
-        }
-
-        return mm_data
-
-
-class Qwen2_5OmniThinkerMultiModalProcessor(
-        BaseMultiModalProcessor[Qwen2_5OmniThinkerProcessingInfo]):
-
-    def _get_data_parser(self) -> MultiModalDataParser:
-        feature_extractor = self.info.get_feature_extractor()
-        return Qwen2_5OmniThinkerMultiModalDataParser(
-            target_sr=feature_extractor.sampling_rate)
-
-    def _call_hf_processor(
-        self,
-        prompt: str,
-        mm_data: Mapping[str, object],
-        mm_kwargs: Mapping[str, object],
-        tok_kwargs: Mapping[str, object],
-    ) -> BatchFeature:
-        mm_data = dict(mm_data)
-        audios = mm_data.pop("audios", [])
-
-        # NOTE: WhisperFeatureExtractor cannot handle empty list of audios
-        if audios:
-            # NOTE: Qwen2.5-Omni processor accept "audio"
-            mm_data["audio"] = audios
-            mm_kwargs = dict(**mm_kwargs, )
-
-        hf_inputs = super()._call_hf_processor(
-            prompt=prompt,
-            mm_data=mm_data,
-            mm_kwargs=mm_kwargs,
-            tok_kwargs=tok_kwargs,
-        )
-
-        input_features = hf_inputs.pop('input_features', None)
-        feature_attention_mask = hf_inputs.get('feature_attention_mask', None)
-        if ('input_audio_features' not in hf_inputs
-                and input_features is not None):
-            if feature_attention_mask is not None:
-                input_features = input_features.permute(
-                    0, 2, 1)[feature_attention_mask.bool()].permute(1, 0)
-            hf_inputs['input_audio_features'] = input_features
-        if ('audio_feature_lengths' not in hf_inputs
-                and feature_attention_mask is not None):
-            hf_inputs['audio_feature_lengths'] = feature_attention_mask.sum(-1)
-        return hf_inputs
-
-    def _get_mm_fields_config(
-        self,
-        hf_inputs: BatchFeature,
-        hf_processor_mm_kwargs: Mapping[str, object],
-    ) -> Mapping[str, MultiModalFieldConfig]:
-        return _qwen2_5_omni_thinker_field_config(hf_inputs)
-
-    def _maybe_apply_prompt_updates(
-        self,
-        mm_items: MultiModalDataItems,
-        hf_processor_mm_kwargs: Mapping[str, object],
-        prompt_ids: list[int],
-        mm_kwargs: MultiModalKwargs,
-        is_update_applied: bool,
-    ) -> tuple[list[int], str, Mapping[str, list[PlaceholderFeaturesInfo]]]:
-        """
-        Qwen2.5-Omni reimplements this function to handle `use_audio_in_video`.
-        """
-        unbound_prompt_updates = self._get_prompt_updates(
-            mm_items,
-            hf_processor_mm_kwargs,
-            mm_kwargs,
-        )
-        mm_prompt_updates = self._bind_and_group_updates(
-            unbound_prompt_updates)
-
-        mm_item_counts = mm_items.get_all_counts()
-        self._validate_mm_kwargs(mm_kwargs, mm_item_counts)
-
-        use_audio_in_video = hf_processor_mm_kwargs.get(
-            "use_audio_in_video", False)
-
-        if is_update_applied:
-            mm_placeholders = self._find_mm_placeholders(
-                mm_prompt_updates,
-                prompt_ids,
-                mm_item_counts,
-            )
-            self._validate_mm_placeholders(
-                mm_placeholders,
-                mm_item_counts,
-                use_audio_in_video=use_audio_in_video)
-
-            tokenizer = self.info.get_tokenizer()
-            prompt = decode_tokens(tokenizer, prompt_ids)
-        else:
-            (
-                prompt_ids,
-                prompt,
-                mm_placeholders,
-            ) = self._apply_prompt_updates(
-                prompt_ids,
-                mm_prompt_updates,
-                mm_item_counts,
-            )
-            self._validate_mm_placeholders(
-                mm_placeholders,
-                mm_item_counts,
-                use_audio_in_video=use_audio_in_video)
-
-        tokenizer = self.info.get_tokenizer()
-        prompt = decode_tokens(tokenizer, prompt_ids)
-
-        if use_audio_in_video:
-            mm_kwargs["use_audio_in_video"] = True
-
-        return prompt_ids, prompt, mm_placeholders
-
-    def _get_prompt_updates(
-        self,
-        mm_items: MultiModalDataItems,
-        hf_processor_mm_kwargs: Mapping[str, Any],
-        out_mm_kwargs: MultiModalKwargs,
-    ) -> Sequence[PromptUpdate]:
-        processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
-        tokenizer = self.info.get_tokenizer()
-        image_processor = self.info.get_image_processor(
-            **hf_processor_mm_kwargs)
-        vocab = tokenizer.get_vocab()
-
-        audio_token = processor.audio_token
-        image_token = processor.image_token
-        video_token = processor.video_token
-        audio_token_id = vocab[audio_token]
-        image_token_id = vocab[image_token]
-        video_token_id = vocab[video_token]
-
-        audio_feature_lengths = out_mm_kwargs.get("audio_feature_lengths")
-        feature_attention_mask = out_mm_kwargs.get("feature_attention_mask")
-        if audio_feature_lengths is None and feature_attention_mask is None:
-            audio_output_lengths = []
-        elif audio_feature_lengths is not None:
-            _, audio_output_lens = _get_feat_extract_output_lengths(
-                audio_feature_lengths)
-            audio_output_lengths = audio_output_lens.tolist()
-        elif feature_attention_mask is not None:
-            assert isinstance(feature_attention_mask, torch.Tensor)
-            _, audio_output_lens = _get_feat_extract_output_lengths(
-                feature_attention_mask.sum(-1))
-            audio_output_lengths = audio_output_lens.tolist()
-
-        # number of audios read from video.
-        audio_in_video_item_idx = 0
-
-        def get_replacement_qwen2_audio(item_idx: int):
-            item_idx += audio_in_video_item_idx
-
-            num_features = audio_output_lengths[item_idx]
-            if num_features == 0:
-                audios = mm_items.get_items("audio", AudioProcessorItems)
-                audio = audios.get(item_idx)
-                raise ValueError(
-                    f"The audio {audio} (len={len(audio)}) is too short "
-                    "to be represented inside the model")
-
-            return [audio_token_id] * num_features
-
-        def get_replacement_qwen2_vision(item_idx: int, modality: str):
-            grid_thw = out_mm_kwargs[f"{modality}_grid_thw"][item_idx]
-            assert isinstance(grid_thw, torch.Tensor)
-            merge_length = image_processor.merge_size**2
-
-            token_id = image_token_id if modality == "image" else video_token_id
-            return [token_id] * (int(grid_thw.prod()) // merge_length)
-
-        use_audio_in_video = hf_processor_mm_kwargs.get(
-            "use_audio_in_video", False)
-        thinker_config = self.info.get_hf_config()
-
-        def get_replacement_qwen2_use_audio_in_video(item_idx: int):
-            nonlocal audio_in_video_item_idx
-
-            audio_num_features = audio_output_lengths[audio_in_video_item_idx +
-                                                      item_idx]
-            video_grid_thw = out_mm_kwargs["video_grid_thw"][item_idx]
-
-            audio_in_video_item_idx += 1
-
-            second_per_grid_ts = hf_processor_mm_kwargs.get(
-                "second_per_grid_ts", None)
-            if second_per_grid_ts:
-                video_second_per_grid_t = second_per_grid_ts[item_idx]
-            else:
-                video_second_per_grid_t = 1.0
-
-            return MRotaryEmbedding.omni_get_updates_use_audio_in_video(
-                thinker_config=thinker_config,
-                audio_len=audio_num_features,
-                video_grid_thw=video_grid_thw,
-                video_second_per_grid_t=video_second_per_grid_t,
-            )
-
-        video_replacement_fn = (
-            get_replacement_qwen2_use_audio_in_video if use_audio_in_video else
-            partial(get_replacement_qwen2_vision, modality="video"))
-
-        return [
-            PromptReplacement(
-                modality="audio",
-                target=audio_token,
-                replacement=get_replacement_qwen2_audio,
-            ),
-            PromptReplacement(
-                modality="image",
-                target=image_token,
-                replacement=partial(get_replacement_qwen2_vision,
-                                    modality="image"),
-            ),
-            PromptReplacement(
-                modality="video",
-                target=video_token,
-                replacement=video_replacement_fn,
-            ),
-        ]
-
-    def _apply_hf_processor_main(
-        self,
-        prompt: Union[str, list[int]],
-        mm_items: MultiModalDataItems,
-        hf_processor_mm_kwargs: Mapping[str, object],
-        tokenization_kwargs: Mapping[str, object],
-        *,
-        enable_hf_prompt_update: bool,
-    ) -> tuple[list[int], BatchFeature, bool]:
-        """
-        Qwen2.5-Omni reimplements this function to handle text only.
-        """
-        if isinstance(prompt, str):
-            if enable_hf_prompt_update:
-                return self._apply_hf_processor_text_mm(
-                    prompt_text=prompt,
-                    mm_items=mm_items,
-                    hf_processor_mm_kwargs=hf_processor_mm_kwargs,
-                    tokenization_kwargs=tokenization_kwargs,
-                )
-            tokenizer = self.info.get_tokenizer()
-            prompt_ids = encode_tokens(tokenizer, prompt)
-        else:
-            prompt_ids = self._apply_hf_processor_tokens_only(prompt)
-
-        mm_processed_data = self._apply_hf_processor_mm_only(
-            mm_items=mm_items,
-            hf_processor_mm_kwargs=hf_processor_mm_kwargs,
-            tokenization_kwargs=tokenization_kwargs,
-        )
-
-        return prompt_ids, mm_processed_data, False
-
-    def _apply_hf_processor_mm_only(
-        self,
-        mm_items: MultiModalDataItems,
-        hf_processor_mm_kwargs: Mapping[str, object],
-        tokenization_kwargs: Mapping[str, object],
-    ) -> BatchFeature:
-        """
-        Qwen2.5-Omni reimplements this function to handle `use_audio_in_video`.
-        """
-        mm_counts = mm_items.get_all_counts()
-
-        use_audio_in_video = hf_processor_mm_kwargs.get(
-            "use_audio_in_video", False)
-        if use_audio_in_video and "video" in mm_counts:
-            assert "audio" in mm_counts
-            mm_counts["audio"] -= mm_counts["video"]
-
-        _, mm_processed_data, _ = self._apply_hf_processor_text_mm(
-            prompt_text=self.dummy_inputs.get_dummy_text(mm_counts),
-            mm_items=mm_items,
-            hf_processor_mm_kwargs=hf_processor_mm_kwargs,
-            tokenization_kwargs=tokenization_kwargs,
-        )
-
-        return mm_processed_data
-
-    def _validate_mm_placeholders(
-        self,
-        mm_placeholders: Mapping[str, list[PlaceholderFeaturesInfo]],
-        mm_item_counts: Mapping[str, int],
-        use_audio_in_video: bool = False,
-    ) -> None:
-        if use_audio_in_video:
-            mm_item_counts = copy(mm_item_counts)
-            if "video" in mm_item_counts:
-                assert "audio" in mm_item_counts
-                mm_item_counts["audio"] -= mm_item_counts["video"]
-        super()._validate_mm_placeholders(mm_placeholders, mm_item_counts)
-
-
-class Qwen2_5OmniConditionalGenerationMixin:
-
-    def _validate_and_reshape_mm_tensor(self,
-                                        mm_input: object,
-                                        name: str,
-                                        dim: int = 0) -> torch.Tensor:
-        if not isinstance(mm_input, (torch.Tensor, list)):
-            raise ValueError(f"Incorrect type of {name}. "
-                             f"Got type: {type(mm_input)}")
-        if isinstance(mm_input, torch.Tensor):
-            return torch.concat(list(mm_input), dim=dim)
-        else:
-            return torch.concat(mm_input, dim=dim)
-
-    def _parse_and_validate_audio_input(
-            self, **kwargs: object) -> Optional[Qwen2AudioInputs]:
-        input_audio_features = kwargs.pop('input_audio_features', None)
-        audio_feature_lengths = kwargs.pop('audio_feature_lengths', None)
-        feature_attention_mask = kwargs.pop('feature_attention_mask', None)
-        if input_audio_features is None:
-            return None
-        input_audio_features = self._validate_and_reshape_mm_tensor(
-            input_audio_features, 'input_audio_features', dim=1)
-        if feature_attention_mask is not None:
-            feature_attention_mask = self._validate_and_reshape_mm_tensor(
-                feature_attention_mask, 'feature_attention_mask')
-        if not isinstance(input_audio_features, (torch.Tensor, list)):
-            raise ValueError("Incorrect type of audio input features. "
-                             f"Got type: {type(input_audio_features)}")
-        return Qwen2AudioInputs(input_features=input_audio_features,
-                                audio_feature_lengths=audio_feature_lengths,
-                                feature_attention_mask=feature_attention_mask)
-
-    def _parse_and_validate_image_input(
-        self,
-        **kwargs: dict[str, Any],
-    ) -> Optional[Qwen2_5_VLImageInputs]:
-        pixel_values = kwargs.pop("pixel_values", None)
-        image_embeds = kwargs.pop("image_embeds", None)
-        image_grid_thw = kwargs.pop("image_grid_thw", None)
-
-        if pixel_values is None and image_embeds is None:
-            return None
-
-        if pixel_values is not None:
-            pixel_values = self._validate_and_reshape_mm_tensor(
-                pixel_values, "image pixel values")
-            image_grid_thw = self._validate_and_reshape_mm_tensor(
-                image_grid_thw, "image grid_thw")
-
-            if not isinstance(pixel_values, (torch.Tensor, list)):
-                raise ValueError("Incorrect type of image pixel values. "
-                                 f"Got type: {type(pixel_values)}")
-
-            return Qwen2_5_VLImagePixelInputs(type="pixel_values",
-                                              pixel_values=pixel_values,
-                                              image_grid_thw=image_grid_thw)
-
-        if image_embeds is not None:
-            image_embeds = self._validate_and_reshape_mm_tensor(
-                image_embeds, "image embeds")
-            image_grid_thw = self._validate_and_reshape_mm_tensor(
-                image_grid_thw, "image grid_thw")
-
-            if not isinstance(image_embeds, torch.Tensor):
-                raise ValueError("Incorrect type of image embeddings. "
-                                 f"Got type: {type(image_embeds)}")
-            return Qwen2_5_VLImageEmbeddingInputs(
-                type="image_embeds",
-                image_embeds=image_embeds,
-                image_grid_thw=image_grid_thw)
-
-    def _parse_and_validate_video_input(
-        self,
-        **kwargs: dict[str, Any],
-    ) -> Optional[Qwen2_5_VLVideoInputs]:
-        pixel_values_videos = kwargs.pop("pixel_values_videos", None)
-        video_embeds = kwargs.pop("video_embeds", None)
-        video_grid_thw = kwargs.pop("video_grid_thw", None)
-
-        if pixel_values_videos is None and video_embeds is None:
-            return None
-
-        if pixel_values_videos is not None:
-            pixel_values_videos = self._validate_and_reshape_mm_tensor(
-                pixel_values_videos, "video pixel values")
-            video_grid_thw = self._validate_and_reshape_mm_tensor(
-                video_grid_thw, "video grid_thw")
-
-            return Qwen2_5_VLVideoPixelInputs(
-                type="pixel_values_videos",
-                pixel_values_videos=pixel_values_videos,
-                video_grid_thw=video_grid_thw,
-            )
-
-        if video_embeds is not None:
-            video_embeds = self._validate_and_reshape_mm_tensor(
-                video_embeds, "video embeds")
-            video_grid_thw = self._validate_and_reshape_mm_tensor(
-                video_grid_thw, "video grid_thw")
-
-            if not isinstance(video_embeds, torch.Tensor):
-                raise ValueError("Incorrect type of video embeddings. "
-                                 f"Got type: {type(video_embeds)}")
-            return Qwen2_5_VLVideoEmbeddingInputs(
-                type="video_embeds",
-                video_embeds=video_embeds,
-                video_grid_thw=video_grid_thw)
-
-    def _process_audio_input(
-        self,
-        audio_input: Qwen2AudioInputs,
-        audio_hashes: list[str] = None,
-        cached_audio_features: torch.Tensor = None,
-    ) -> torch.Tensor:
-
-        input_features = audio_input["input_features"]
-        audio_feature_lengths = audio_input["audio_feature_lengths"]
-        if input_features.ndim == 3:
-            assert input_features.shape[0] == 1
-            input_features = input_features.squeeze(0)
-        if audio_feature_lengths.ndim == 2:
-            assert audio_feature_lengths.shape[
-                0] == 1 or audio_feature_lengths.shape[1] == 1
-            if audio_feature_lengths.shape[0] == 1:
-                audio_feature_lengths = audio_feature_lengths.squeeze(0)
-            else:
-                audio_feature_lengths = audio_feature_lengths.squeeze(1)
-
-        audio_feat_lengths, audio_output_lengths = (
-            self.audio_tower._get_feat_extract_output_lengths(
-                audio_feature_lengths))
-
-        audio_outputs = self.audio_tower(
-            input_features.to(self.audio_tower.dtype),
-            feature_lens=audio_feature_lengths,
-            aftercnn_lens=audio_feat_lengths,
-        )
-        audio_features = audio_outputs.last_hidden_state
-        return audio_features.split(audio_output_lengths.tolist())
-
-    def _process_image_input(
-            self,
-            image_input: Qwen2_5_VLImageInputs) -> tuple[torch.Tensor, ...]:
-        if image_input["type"] == "image_embeds":
-            return image_input["image_embeds"].type(self.visual.dtype)
-
-        grid_thw = image_input["image_grid_thw"]
-        assert grid_thw.ndim == 2
-
-        pixel_values = image_input["pixel_values"].type(self.visual.dtype)
-        image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
-        # Split concatenated embeddings for each image item.
-        merge_size = self.visual.spatial_merge_size
-        sizes = grid_thw.prod(-1) // merge_size // merge_size
-
-        return image_embeds.split(sizes.tolist())
-
-    def _process_video_input(
-            self,
-            video_input: Qwen2_5_VLVideoInputs,
-            video_hashes: list[str] = None,
-            cached_video_embeds: torch.Tensor = None) -> torch.Tensor:
-        if video_input["type"] == "video_embeds":
-            return video_input["video_embeds"].type(self.visual.dtype)
-
-        grid_thw = video_input["video_grid_thw"]
-        assert grid_thw.ndim == 2
-
-        pixel_values_videos = video_input["pixel_values_videos"].type(
-            self.visual.dtype)
-        video_embeds = self.visual(pixel_values_videos, grid_thw=grid_thw)
-        # Split concatenated embeddings for each video item.
-        merge_size = self.visual.spatial_merge_size
-        sizes = grid_thw.prod(-1) // merge_size // merge_size
-
-        return video_embeds.split(sizes.tolist())
-
-
-@MULTIMODAL_REGISTRY.register_processor(
-    Qwen2_5OmniThinkerMultiModalProcessor,
-    info=Qwen2_5OmniThinkerProcessingInfo,
-    dummy_inputs=Qwen2_5OmniThinkerDummyInputsBuilder,
-)
-class Qwen2_5OmniThinkerForConditionalGeneration(
-        nn.Module, SupportsMultiModal, SupportsPP,
-        Qwen2_5OmniConditionalGenerationMixin):
-    hf_to_vllm_mapper = WeightsMapper(
-        orig_to_new_prefix={
-            "thinker.lm_head.": "language_model.lm_head.",
-            "thinker.model.": "language_model.model.",
-            "thinker.": "",
-        })
-
-    @classmethod
-    def get_placeholder_str(cls, modality: str, i: int) -> Optional[str]:
-        if modality.startswith("image"):
-            return "<|vision_start|><|IMAGE|><|vision_end|>"
-        if modality.startswith("video"):
-            return "<|vision_start|><|VIDEO|><|vision_end|>"
-        if modality.startswith("audio"):
-            return f"Audio {i}: <|audio_bos|><|AUDIO|><|audio_eos|>"
-
-        raise ValueError("Only image, video or audio modality is supported")
-
-    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
-        super().__init__()
-        thinker_config: Qwen2_5OmniThinkerConfig = (
-            vllm_config.model_config.hf_config)
-        quant_config = vllm_config.quant_config
-        multimodal_config = vllm_config.model_config.multimodal_config
-        self.config = thinker_config
-        self.multimodal_config = multimodal_config
-
-        # force "use_flash_attention_2=True" to audio tower to align
-        # the results.
-        if flash_attn is not None:
-            audio_config = thinker_config.audio_config
-            audio_config._attn_implementation_autoset = True
-            audio_config._attn_implementation = "flash_attention_2"
-        else:
-            logger.warning(
-                "flash_attn is not available, the model may not yield the "
-                "exactly same result as the transformers implementation "
-                "in the audio tower part.")
-
-        self.audio_tower = Qwen2_5OmniAudioEncoder(thinker_config.audio_config)
-        self.visual = Qwen2_5_VisionTransformer(
-            vision_config=thinker_config.vision_config,
-            norm_eps=getattr(thinker_config.text_config, "rms_norm_eps", 1e-6),
-            quant_config=quant_config,
-            prefix=maybe_prefix(prefix, "visual"),
-        )
-        self.quant_config = quant_config
-        self.language_model = init_vllm_registered_model(
-            vllm_config=vllm_config,
-            prefix=maybe_prefix(prefix, "language_model"),
-            hf_config=thinker_config.text_config,
-            architectures=["Qwen2ForCausalLM"],
-        )
-
-        self.make_empty_intermediate_tensors = (
-            self.language_model.make_empty_intermediate_tensors)
-
-    def _parse_and_validate_multimodal_inputs(self, **kwargs: object) -> dict:
-        mm_input_by_modality = {}
-
-        # Preserve the order of modalities if there are multiple of them
-        # from the order of kwargs.
-        for input_key in kwargs:
-            if input_key in ("pixel_values", "image_embeds"
-                             ) and "image" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "image"] = self._parse_and_validate_image_input(**kwargs)
-            if input_key in ("pixel_values_videos", "video_embeds"
-                             ) and "video" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "video"] = self._parse_and_validate_video_input(**kwargs)
-            if input_key in ("input_audio_features"
-                             ) and "audio" not in mm_input_by_modality:
-                mm_input_by_modality[
-                    "audio"] = self._parse_and_validate_audio_input(**kwargs)
-        return mm_input_by_modality
-
-    def get_language_model(self) -> torch.nn.Module:
-        return self.language_model
-
-    def get_multimodal_embeddings(self,
-                                  **kwargs: object) -> MultiModalEmbeddings:
-
-        mm_input_by_modality = self._parse_and_validate_multimodal_inputs(
-            **kwargs)
-        if not mm_input_by_modality:
-            return []
-
-        # The result multimodal_embeddings is tuple of tensors, with each
-        # tensor correspoending to a multimodal data item (image or video).
-        multimodal_embeddings: tuple[torch.Tensor, ...] = ()
-
-        # NOTE: It is important to iterate over the keys in this dictionary
-        # to preserve the order of the modalities.
-        for modality in mm_input_by_modality:
-            multimodal_input = mm_input_by_modality[modality]
-            if modality == "image":
-                vision_embeddings = self._process_image_input(multimodal_input)
-                multimodal_embeddings += vision_embeddings
-            if modality == "video":
-                video_embeddings = self._process_video_input(multimodal_input)
-                multimodal_embeddings += video_embeddings
-            if modality == "audio":
-                audio_embeddings = self._process_audio_input(multimodal_input)
-                multimodal_embeddings += audio_embeddings
-        return multimodal_embeddings
-
-    def get_input_embeddings(
-        self,
-        input_ids: torch.Tensor,
-        multimodal_embeddings: Optional[MultiModalEmbeddings] = None,
-    ) -> torch.Tensor:
-        inputs_embeds = self.language_model.get_input_embeddings(input_ids)
-        if multimodal_embeddings is not None \
-            and len(multimodal_embeddings) != 0:
-
-            # TODO (ywang96): support overlapping modalitiy embeddings so that
-            # `use_audio_in_video` will work on V1.
-            inputs_embeds = merge_multimodal_embeddings(
-                input_ids, inputs_embeds, multimodal_embeddings, [
-                    self.config.image_token_index,
-                    self.config.video_token_index,
-                    self.config.audio_token_index
-                ])
-        return inputs_embeds
-
-    def get_multimodal_embeddings_v0(
-            self, **kwargs: object) -> Optional[NestedTensors]:
-        audio_input = self._parse_and_validate_audio_input(**kwargs)
-        image_input = self._parse_and_validate_image_input(**kwargs)
-        video_input = self._parse_and_validate_video_input(**kwargs)
-
-        if audio_input is None and image_input is None and video_input is None:
-            return None
-
-        multimodal_embeddings: list[tuple[NestedTensors, str]] = []
-
-        if audio_input is not None:
-            audio_embeds = self._process_audio_input(audio_input)
-            multimodal_embeddings.append((audio_embeds, "audio"))
-        if image_input is not None:
-            image_embeds = self._process_image_input(image_input)
-            multimodal_embeddings.append((image_embeds, "image"))
-        if video_input is not None:
-            video_embeds = self._process_video_input(video_input)
-            multimodal_embeddings.append((video_embeds, "video"))
-        return multimodal_embeddings
-
-    def get_input_embeddings_v0(
-        self,
-        input_ids: torch.Tensor,
-        multimodal_embeddings: Optional[NestedTensors] = None,
-    ) -> torch.Tensor:
-        inputs_embeds = self.language_model.get_input_embeddings(input_ids)
-        if multimodal_embeddings is None or len(multimodal_embeddings) == 0:
-            return inputs_embeds
-
-        for embeddings, modality in multimodal_embeddings:
-            if modality == "audio":
-                placeholder_token_id = self.config.audio_token_index
-            if modality == "image":
-                placeholder_token_id = self.config.image_token_index
-            if modality == "video":
-                placeholder_token_id = self.config.video_token_index
-            inputs_embeds = merge_multimodal_embeddings(
-                input_ids, inputs_embeds, embeddings, placeholder_token_id)
-        return inputs_embeds
-
-    def forward(
-        self,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-        inputs_embeds: Optional[torch.Tensor] = None,
-        **kwargs: object,
-    ) -> Union[torch.Tensor, IntermediateTensors]:
-        if intermediate_tensors is not None:
-            inputs_embeds = None
-
-        # NOTE: In v1, inputs_embeds is always generated at model runner, this
-        # condition is for v0 compatibility.
-        elif inputs_embeds is None:
-            multimodal_embeddings = self.get_multimodal_embeddings(**kwargs)
-            inputs_embeds = self.get_input_embeddings(
-                input_ids, multimodal_embeddings)
-            text_inputs_embeds = self.get_input_embeddings(input_ids, [(torch.zeros_like(embeddings), modality) for embeddings, modality in multimodal_embeddings] if multimodal_embeddings is not None else None)
-            input_ids = None
-        else:
-            text_inputs_embeds = inputs_embeds
-
-        hidden_states = self.language_model.model(input_ids,
-                                                  positions,
-                                                  intermediate_tensors,
-                                                  inputs_embeds=inputs_embeds)
-        return text_inputs_embeds, hidden_states.unsqueeze(0) #(1, S, D)
-
-    def compute_logits(
-        self,
-        hidden_states: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[torch.Tensor]:
-        return self.language_model.compute_logits(hidden_states,
-                                                  sampling_metadata)
-
-    def load_weights(self, weights: Iterable[tuple[str,
-                                                   torch.Tensor]]) -> set[str]:
-        loader = AutoWeightsLoader(
-            self,
-            skip_prefixes=["talker.", "token2wav."],
-        )
-        loaded_weights = loader.load_weights(weights,
-                                             mapper=self.hf_to_vllm_mapper)
-        # Log load summary
-        try:
-            total_bytes = 0
-            for name, param in self.named_parameters():
-                if param is not None and param.data is not None:
-                    total_bytes += param.data.numel() * param.data.element_size()
-            device = next(self.parameters()).device
-            logger.info(
-                "[Model Loaded] name=%s, success=%s, size=%.2f MB, device=%s",
-                self.__class__.__name__, True, total_bytes / (1024**2), str(device))
-        except Exception:
-            pass
-        return loaded_weights
-

vllm_omni/model_executor/models/qwen2_old.py

@@ -1,539 +0,0 @@
-from typing import Iterable, Optional, Set, Tuple, Union
-
-import torch
-from torch import nn
-from transformers import Qwen2Config
-
-from vllm.attention import Attention, AttentionType
-from vllm.compilation.decorators import support_torch_compile
-from vllm.config import CacheConfig, VllmConfig
-from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
-from vllm.logger import init_logger
-from vllm.model_executor.layers.activation import SiluAndMul
-from vllm.model_executor.layers.layernorm import RMSNorm
-from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
-                                               QKVParallelLinear,
-                                               RowParallelLinear)
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
-from vllm.model_executor.layers.pooler import Pooler, PoolingType
-from vllm.model_executor.layers.quantization import QuantizationConfig
-from vllm.model_executor.layers.rotary_embedding import get_rope
-from vllm.model_executor.layers.sampler import SamplerOutput, get_sampler
-from vllm.model_executor.layers.vocab_parallel_embedding import (
-    ParallelLMHead, VocabParallelEmbedding)
-from vllm.model_executor.model_loader.weight_utils import (
-    default_weight_loader, maybe_remap_kv_scale_name)
-from vllm.v1.pool.metadata import PoolingMetadata
-from vllm.model_executor.sampling_metadata import SamplingMetadata
-from vllm.sequence import IntermediateTensors, PoolerOutput
-
-from vllm.model_executor.models.interfaces import SupportsLoRA, SupportsPP
-from vllm.model_executor.models.utils import (AutoWeightsLoader, PPMissingLayer, WeightsMapper,
-                    is_pp_missing_parameter,
-                    make_empty_intermediate_tensors_factory, make_layers,
-                    maybe_prefix)
-
-logger = init_logger(__name__)
-
-
-class Qwen2MLP(nn.Module):
-
-    def __init__(
-        self,
-        hidden_size: int,
-        intermediate_size: int,
-        hidden_act: str,
-        quant_config: Optional[QuantizationConfig] = None,
-        prefix: str = "",
-    ) -> None:
-        super().__init__()
-        self.gate_up_proj = MergedColumnParallelLinear(
-            hidden_size,
-            [intermediate_size] * 2,
-            bias=False,
-            quant_config=quant_config,
-            prefix=f"{prefix}.gate_up_proj",
-        )
-        self.down_proj = RowParallelLinear(
-            intermediate_size,
-            hidden_size,
-            bias=False,
-            quant_config=quant_config,
-            prefix=f"{prefix}.down_proj",
-        )
-        if hidden_act != "silu":
-            raise ValueError(f"Unsupported activation: {hidden_act}. "
-                             "Only silu is supported for now.")
-        self.act_fn = SiluAndMul()
-
-    def forward(self, x):
-        gate_up, _ = self.gate_up_proj(x)
-        x = self.act_fn(gate_up)
-        x, _ = self.down_proj(x)
-        return x
-
-
-class Qwen2Attention(nn.Module):
-
-    def __init__(self,
-                 hidden_size: int,
-                 num_heads: int,
-                 num_kv_heads: int,
-                 max_position: int = 4096 * 32,
-                 rope_theta: float = 10000,
-                 head_dim: Optional[int] = None,
-                 cache_config: Optional[CacheConfig] = None,
-                 quant_config: Optional[QuantizationConfig] = None,
-                 rope_scaling: Optional[Tuple] = None,
-                 prefix: str = "",
-                 attn_type: str = AttentionType.DECODER) -> None:
-        super().__init__()
-        self.hidden_size = hidden_size
-        tp_size = get_tensor_model_parallel_world_size()
-        self.total_num_heads = num_heads
-        assert self.total_num_heads % tp_size == 0
-        self.num_heads = self.total_num_heads // tp_size
-        self.total_num_kv_heads = num_kv_heads
-        if self.total_num_kv_heads >= tp_size:
-            # Number of KV heads is greater than TP size, so we partition
-            # the KV heads across multiple tensor parallel GPUs.
-            assert self.total_num_kv_heads % tp_size == 0
-        else:
-            # Number of KV heads is less than TP size, so we replicate
-            # the KV heads across multiple tensor parallel GPUs.
-            assert tp_size % self.total_num_kv_heads == 0
-        self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
-        self.head_dim = head_dim or (hidden_size // self.total_num_heads)
-        self.q_size = self.num_heads * self.head_dim
-        self.kv_size = self.num_kv_heads * self.head_dim
-        self.scaling = self.head_dim**-0.5
-        self.rope_theta = rope_theta
-
-        self.qkv_proj = QKVParallelLinear(
-            hidden_size,
-            self.head_dim,
-            self.total_num_heads,
-            self.total_num_kv_heads,
-            bias=True,
-            quant_config=quant_config,
-            prefix=f"{prefix}.qkv_proj",
-        )
-        self.o_proj = RowParallelLinear(
-            self.total_num_heads * self.head_dim,
-            hidden_size,
-            bias=False,
-            quant_config=quant_config,
-            prefix=f"{prefix}.o_proj",
-        )
-
-        self.rotary_emb = get_rope(
-            self.head_dim,
-            rotary_dim=self.head_dim,
-            max_position=max_position,
-            base=self.rope_theta,
-            rope_scaling=rope_scaling,
-        )
-        self.attn = Attention(self.num_heads,
-                              self.head_dim,
-                              self.scaling,
-                              num_kv_heads=self.num_kv_heads,
-                              cache_config=cache_config,
-                              quant_config=quant_config,
-                              prefix=f"{prefix}.attn",
-                              attn_type=attn_type)
-
-    def forward(
-        self,
-        positions: torch.Tensor,
-        hidden_states: torch.Tensor,
-    ) -> torch.Tensor:
-        qkv, _ = self.qkv_proj(hidden_states)
-        q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
-        q, k = self.rotary_emb(positions, q, k)
-        attn_output = self.attn(q, k, v)
-        output, _ = self.o_proj(attn_output)
-        return output
-
-
-class Qwen2DecoderLayer(nn.Module):
-
-    def __init__(
-        self,
-        config: Qwen2Config,
-        cache_config: Optional[CacheConfig] = None,
-        quant_config: Optional[QuantizationConfig] = None,
-        prefix: str = "",
-    ) -> None:
-        super().__init__()
-        self.hidden_size = config.hidden_size
-        # Requires transformers > 4.32.0
-        rope_theta = getattr(config, "rope_theta", 1000000)
-        rope_scaling = getattr(config, "rope_scaling", None)
-
-        # By default, Qwen2 uses causal attention as it is a decoder-only model.
-        # You can override the HF config with `is_causal=False` to enable
-        # bidirectional attention, which is used in some embedding models
-        # (e.g. Alibaba-NLP/gte-Qwen2-7B-instruct)
-        if getattr(config, "is_causal", True):
-            attn_type = AttentionType.DECODER
-        else:
-            attn_type = AttentionType.ENCODER_ONLY
-
-        self.self_attn = Qwen2Attention(
-            hidden_size=self.hidden_size,
-            num_heads=config.num_attention_heads,
-            max_position=config.max_position_embeddings,
-            num_kv_heads=config.num_key_value_heads,
-            rope_theta=rope_theta,
-            cache_config=cache_config,
-            quant_config=quant_config,
-            rope_scaling=rope_scaling,
-            head_dim=getattr(config, "head_dim", None),
-            prefix=f"{prefix}.self_attn",
-            attn_type=attn_type,
-        )
-        self.mlp = Qwen2MLP(
-            hidden_size=self.hidden_size,
-            intermediate_size=config.intermediate_size,
-            hidden_act=config.hidden_act,
-            quant_config=quant_config,
-            prefix=f"{prefix}.mlp",
-        )
-        self.input_layernorm = RMSNorm(config.hidden_size,
-                                       eps=config.rms_norm_eps)
-        self.post_attention_layernorm = RMSNorm(config.hidden_size,
-                                                eps=config.rms_norm_eps)
-
-    def forward(
-        self,
-        positions: torch.Tensor,
-        hidden_states: torch.Tensor,
-        residual: Optional[torch.Tensor],
-    ) -> Tuple[torch.Tensor, torch.Tensor]:
-        # Self Attention
-        if residual is None:
-            residual = hidden_states
-            hidden_states = self.input_layernorm(hidden_states)
-        else:
-            hidden_states, residual = self.input_layernorm(
-                hidden_states, residual)
-        hidden_states = self.self_attn(
-            positions=positions,
-            hidden_states=hidden_states,
-        )
-
-        # Fully Connected
-        hidden_states, residual = self.post_attention_layernorm(
-            hidden_states, residual)
-        hidden_states = self.mlp(hidden_states)
-        return hidden_states, residual
-
-
-@support_torch_compile(
-    dynamic_arg_dims={
-        "input_ids": 0,
-        # positions is of shape (3, seq_len) if mrope is enabled for qwen2-vl,
-        # otherwise (seq_len, ).
-        "positions": -1,
-        "intermediate_tensors": 0,
-        "inputs_embeds": 0,
-    })
-class Qwen2Model(nn.Module):
-
-    def __init__(self,
-                 *,
-                 vllm_config: VllmConfig,
-                 prefix: str = "",
-                 decoder_layer_type: type[nn.Module] = Qwen2DecoderLayer):
-        super().__init__()
-
-        config = vllm_config.model_config.hf_config
-        cache_config = vllm_config.cache_config
-        quant_config = vllm_config.quant_config
-
-        # TODO (@robertgshaw2): see if this can be moved out
-        if (cache_config.sliding_window is not None
-                and hasattr(config, "max_window_layers")):
-            raise ValueError("Sliding window for some but all layers is not "
-                             "supported. This model uses sliding window "
-                             "but `max_window_layers` = {} is less than "
-                             "`num_hidden_layers` = {}. Please open an issue "
-                             "to discuss this feature.".format(
-                                 config.max_window_layers,
-                                 config.num_hidden_layers,
-                             ))
-
-        self.config = config
-        self.quant_config = quant_config
-        self.vocab_size = config.vocab_size
-
-        if get_pp_group().is_first_rank or (config.tie_word_embeddings
-                                            and get_pp_group().is_last_rank):
-            self.embed_tokens = VocabParallelEmbedding(
-                config.vocab_size,
-                getattr(config, "embedding_size", config.hidden_size),
-                quant_config=quant_config,
-                prefix=f"{prefix}.embed_tokens",
-            )
-        else:
-            self.embed_tokens = PPMissingLayer()
-
-        # Use the provided decoder layer type or default to Qwen2DecoderLayer
-        decoder_layer_type = decoder_layer_type or Qwen2DecoderLayer
-        self.start_layer, self.end_layer, self.layers = make_layers(
-            config.num_hidden_layers,
-            lambda prefix: decoder_layer_type(config=config,
-                                              cache_config=cache_config,
-                                              quant_config=quant_config,
-                                              prefix=prefix),
-            prefix=f"{prefix}.layers",
-        )
-
-        self.make_empty_intermediate_tensors = (
-            make_empty_intermediate_tensors_factory(
-                ["hidden_states", "residual"], config.hidden_size))
-        if get_pp_group().is_last_rank:
-            self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
-        else:
-            self.norm = PPMissingLayer()
-
-    def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
-        return self.embed_tokens(input_ids)
-
-    def forward(
-        self,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-        inputs_embeds: Optional[torch.Tensor] = None,
-    ) -> Union[torch.Tensor, IntermediateTensors]:
-        if get_pp_group().is_first_rank:
-            if inputs_embeds is not None:
-                hidden_states = inputs_embeds
-            else:
-                hidden_states = self.get_input_embeddings(input_ids)
-            residual = None
-        else:
-            assert intermediate_tensors is not None
-            hidden_states = intermediate_tensors["hidden_states"]
-            residual = intermediate_tensors["residual"]
-        for layer in self.layers[self.start_layer:self.end_layer]:
-            hidden_states, residual = layer(
-                positions,
-                hidden_states,
-                residual,
-            )
-        if not get_pp_group().is_last_rank:
-            return IntermediateTensors({
-                "hidden_states": hidden_states,
-                "residual": residual
-            })
-        hidden_states, _ = self.norm(hidden_states, residual)
-        return hidden_states
-
-    def load_weights(self, weights: Iterable[Tuple[str,
-                                                   torch.Tensor]]) -> Set[str]:
-        stacked_params_mapping = [
-            # (param_name, shard_name, shard_id)
-            ("qkv_proj", "q_proj", "q"),
-            ("qkv_proj", "k_proj", "k"),
-            ("qkv_proj", "v_proj", "v"),
-            ("gate_up_proj", "gate_proj", 0),
-            ("gate_up_proj", "up_proj", 1),
-        ]
-        params_dict = dict(self.named_parameters(remove_duplicate=False))
-        loaded_params: Set[str] = set()
-        for name, loaded_weight in weights:
-            if "rotary_emb.inv_freq" in name:
-                continue
-            if (self.quant_config is not None and
-                (scale_name := self.quant_config.get_cache_scale(name))):
-                # Loading kv cache quantization scales
-                param = params_dict[scale_name]
-                weight_loader = getattr(param, "weight_loader",
-                                        default_weight_loader)
-                loaded_weight = (loaded_weight if loaded_weight.dim() == 0 else
-                                 loaded_weight[0])
-                weight_loader(param, loaded_weight)
-                loaded_params.add(scale_name)
-                continue
-            for (param_name, weight_name, shard_id) in stacked_params_mapping:
-                if weight_name not in name:
-                    continue
-                name = name.replace(weight_name, param_name)
-                # Skip loading extra bias for GPTQ models.
-                if name.endswith(".bias") and name not in params_dict:
-                    continue
-                if is_pp_missing_parameter(name, self):
-                    continue
-                param = params_dict[name]
-                weight_loader = param.weight_loader
-                weight_loader(param, loaded_weight, shard_id)
-                break
-            else:
-                # Skip loading extra bias for GPTQ models.
-                if name.endswith(".bias") and name not in params_dict:
-                    continue
-                # Remapping the name of FP8 kv-scale.
-                name = maybe_remap_kv_scale_name(name, params_dict)
-                if name is None:
-                    continue
-                if is_pp_missing_parameter(name, self):
-                    continue
-                param = params_dict[name]
-                weight_loader = getattr(param, "weight_loader",
-                                        default_weight_loader)
-                weight_loader(param, loaded_weight)
-            loaded_params.add(name)
-        return loaded_params
-
-
-class Qwen2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
-    packed_modules_mapping = {
-        "qkv_proj": [
-            "q_proj",
-            "k_proj",
-            "v_proj",
-        ],
-        "gate_up_proj": [
-            "gate_proj",
-            "up_proj",
-        ],
-    }
-
-    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
-        super().__init__()
-        config = vllm_config.model_config.hf_config
-        quant_config = vllm_config.quant_config
-        lora_config = vllm_config.lora_config
-
-        self.config = config
-        self.lora_config = lora_config
-
-        self.quant_config = quant_config
-        self.model = Qwen2Model(vllm_config=vllm_config,
-                                prefix=maybe_prefix(prefix, "model"))
-
-        if get_pp_group().is_last_rank:
-            if config.tie_word_embeddings:
-                self.lm_head = self.model.embed_tokens
-            else:
-                self.lm_head = ParallelLMHead(config.vocab_size,
-                                              config.hidden_size,
-                                              quant_config=quant_config,
-                                              prefix=maybe_prefix(
-                                                  prefix, "lm_head"))
-        else:
-            self.lm_head = PPMissingLayer()
-
-        self.logits_processor = LogitsProcessor(config.vocab_size)
-        self.sampler = get_sampler()
-
-        self.make_empty_intermediate_tensors = (
-            self.model.make_empty_intermediate_tensors)
-
-    def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
-        return self.model.get_input_embeddings(input_ids)
-
-    def forward(
-        self,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-        inputs_embeds: Optional[torch.Tensor] = None,
-    ) -> Union[torch.Tensor, IntermediateTensors]:
-        hidden_states = self.model(input_ids, positions, intermediate_tensors,
-                                   inputs_embeds)
-        return hidden_states
-
-    def compute_logits(
-        self,
-        hidden_states: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[torch.Tensor]:
-        logits = self.logits_processor(self.lm_head, hidden_states,
-                                       sampling_metadata)
-        return logits
-
-    def sample(
-        self,
-        logits: torch.Tensor,
-        sampling_metadata: SamplingMetadata,
-    ) -> Optional[SamplerOutput]:
-        next_tokens = self.sampler(logits, sampling_metadata)
-        return next_tokens
-
-    def load_weights(self, weights: Iterable[Tuple[str,
-                                                   torch.Tensor]]) -> Set[str]:
-        loader = AutoWeightsLoader(
-            self,
-            skip_prefixes=(["lm_head."]
-                           if self.config.tie_word_embeddings else None),
-        )
-        return loader.load_weights(weights)
-
-
-class Qwen2EmbeddingModel(nn.Module, SupportsLoRA, SupportsPP):
-    packed_modules_mapping = {
-        "qkv_proj": [
-            "q_proj",
-            "k_proj",
-            "v_proj",
-        ],
-        "gate_up_proj": [
-            "gate_proj",
-            "up_proj",
-        ],
-    }
-
-    hf_to_vllm_mapper = WeightsMapper(orig_to_new_prefix={"model.": ""})
-
-    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
-        super().__init__()
-        config = vllm_config.model_config.hf_config
-        quant_config = vllm_config.quant_config
-        lora_config = vllm_config.lora_config
-        pooler_config = vllm_config.model_config.pooler_config
-
-        self.config = config
-        self.lora_config = lora_config
-
-        self.quant_config = quant_config
-        self.model = Qwen2Model(vllm_config=vllm_config,
-                                prefix=maybe_prefix(prefix, "model"))
-
-        # TODO: Replace this model class with as_embedding_model(
-        # Qwen2ForCausalLM) after changing the default pooling method
-        if pooler_config.pooling_type is None:
-            logger.warning(
-                "This embedding model will default to last-token pooling in "
-                "an upcoming version. To avoid breaking changes, you should "
-                "pass `--override-pooler-config '{\"pooling_type\": \"MEAN\"}'`"
-                " explicitly.")
-
-        self._pooler = Pooler.from_config_with_defaults(
-            pooler_config,
-            pooling_type=PoolingType.MEAN,
-            normalize=True,
-            softmax=False)
-
-    def forward(
-        self,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> torch.Tensor:
-        return self.model(input_ids, positions, intermediate_tensors)
-
-    def pooler(
-        self,
-        hidden_states: torch.Tensor,
-        pooling_metadata: PoolingMetadata,
-    ) -> Optional[PoolerOutput]:
-        return self._pooler(hidden_states, pooling_metadata)
-
-    def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
-        weights = self.hf_to_vllm_mapper.apply(weights)
-        weights = ((name, data) for name, data in weights
-                   if not name.startswith("lm_head."))
-        self.model.load_weights(weights)

vllm_omni/model_executor/models/utils.py

@@ -1,8 +0,0 @@
-from vllm.model_executor.models.utils import maybe_prefix
-
-
-def add_prefix_to_loaded_weights(weights: set[str], prefix: str) -> set[str]:
-    """
-    Add a prefix to the names of the loaded weights.
-    """
-    return {maybe_prefix(prefix, name) for name in weights}

vllm_omni/model_executor/stage_configs/qwen2_5_omni.yaml

@@ -1,40 +0,0 @@
-# stage config for running qwen2.5-omni with architecture of OmniLLM.
-stage_args:
-  - stage_id: 0
-    engine_args:
-      model_stage: thinker
-      model_arch: Qwen2_5OmniForConditionalGeneration
-      worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
-      scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
-      gpu_memory_utilization: 0.32
-      enforce_eager: true  # need to discuss
-      trust_remote_code: true
-      engine_output_type: latent  # change the param name,such as pooling_output
-    final_output: true
-    final_output_type: text
-  - stage_id: 1
-    engine_args:
-      model_stage: talker
-      model_arch: Qwen2_5OmniForConditionalGeneration
-      worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
-      scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
-      gpu_memory_utilization: 0.32
-      enforce_eager: true
-      trust_remote_code: true
-      engine_output_type: latent
-    engine_input_source: [0]
-    custom_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen2_5_omni.thinker2talker
-  - stage_id: 2
-    engine_args:
-      model_stage: code2wav
-      model_arch: Qwen2_5OmniForConditionalGeneration
-      worker_cls: vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker
-      scheduler_cls: vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler
-      gpu_memory_utilization: 0.3
-      enforce_eager: true
-      trust_remote_code: true
-      enable_prefix_caching: false
-      engine_output_type: audio
-    engine_input_source: [1]
-    final_output: true
-    final_output_type: audio

vllm_omni/model_executor/stage_input_processors/qwen2_5_omni.py

@@ -1,40 +0,0 @@
-from typing import Union
-import torch
-
-from vllm.inputs import TextPrompt
-
-from vllm_omni.inputs.data import OmniTokensPrompt
-
-def thinker2talker(stage_list, engine_input_source, prompt: Union[OmniTokensPrompt, TextPrompt] = None):
-    if not engine_input_source:
-      raise ValueError("engine_input_source cannot be empty")
-    source_stage_id = engine_input_source[0]
-    if source_stage_id >= len(stage_list):
-      raise IndexError(f"Invalid stage_id: {source_stage_id}")
-    if stage_list[source_stage_id].engine_outputs is None:
-        raise RuntimeError(f"Stage {source_stage_id} has no outputs yet")
-    thinker_outputs = stage_list[source_stage_id].engine_outputs
-    talker_inputs = []
-    multi_modal_data = {thinker_output.request_id: 
-            prompt.get('multi_modal_data', None) for thinker_output, prompt in zip(thinker_outputs, prompt)}
-    
-    for i, thinker_output in enumerate(thinker_outputs):
-        output = thinker_output.outputs[0]
-        prompt_token_ids = thinker_output.prompt_token_ids
-        thinker_output_ids = output.token_ids
-        prompt_token_ids_len = len(prompt_token_ids)
-        thinker_hidden_states = output.multimodal_output["latent"].clone().detach().cuda()
-        talker_inputs.append(
-            OmniTokensPrompt(
-                prompt_token_ids=[0] * (len(prompt_token_ids) + 2), # add 2 for codec pad and start token
-                additional_information={
-                    "thinker_result": thinker_hidden_states[prompt_token_ids_len:].to(torch.float32),
-                    "prompt_embeds": thinker_hidden_states[:prompt_token_ids_len].to(torch.float32),
-                    "prompt_token_ids": prompt_token_ids,
-                    "thinker_output_token_ids": thinker_output_ids,
-                },
-                multi_modal_data=multi_modal_data[thinker_output.request_id] if multi_modal_data is not None else None,
-                mm_processor_kwargs=None,
-            )
-        )
-    return talker_inputs

vllm_omni/outputs.py

@@ -1,9 +0,0 @@
-from vllm.outputs import RequestOutput
-from dataclasses import dataclass
-
-
-@dataclass
-class OmniRequestOutput(RequestOutput):
-    stage_id: int
-    final_output_type: str
-    request_output: RequestOutput

vllm_omni/request.py

@@ -1,299 +0,0 @@
-"""
-OmniRequest: Extended request class for vLLM-omni multimodal processing.
-
-This class extends vLLM's Request to support multimodal and non-autoregressive
-processing with additional fields and methods specific to vLLM-omni.
-"""
-
-import enum
-import time
-from typing import Optional, Dict, Any, List, Union
-from dataclasses import dataclass, field
-from vllm.v1.request import Request as vLLMRequest
-
-
-class RequestType(enum.Enum):
-    """Types of requests supported by vLLM-omni."""
-    TEXT = "text"
-    IMAGE = "image"
-    AUDIO = "audio"
-    VIDEO = "video"
-    MULTIMODAL = "multimodal"
-    DIFFUSION = "diffusion"
-
-
-class ProcessingStage(enum.Enum):
-    """Processing stages for multi-stage models."""
-    PREPROCESSING = "preprocessing"
-    AR_GENERATION = "ar_generation"
-    DIFFUSION_GENERATION = "diffusion_generation"
-    POSTPROCESSING = "postprocessing"
-    COMPLETED = "completed"
-
-
-@dataclass
-class MultimodalData:
-    """Container for multimodal input data."""
-    data_type: str  # "image", "audio", "video", etc.
-    data: Any  # The actual data (numpy array, bytes, etc.)
-    metadata: Dict[str, Any] = field(default_factory=dict)
-    processed: bool = False
-
-
-@dataclass
-class DiffusionParams:
-    """Parameters specific to diffusion model generation."""
-    num_inference_steps: int = 50
-    guidance_scale: float = 7.5
-    seed: Optional[int] = None
-    scheduler: str = "ddpm"
-    strength: float = 1.0
-    eta: float = 0.0
-
-
-class OmniRequest(vLLMRequest):
-    """
-    Extended request class for vLLM-omni multimodal and non-autoregressive processing.
-    
-    This class extends vLLM's Request with additional fields and methods to support:
-    - Multimodal input processing
-    - Non-autoregressive generation (diffusion models)
-    - Multi-stage processing pipelines
-    - Enhanced caching for different model types
-    """
-    
-    def __init__(
-        self,
-        request_id: str,
-        prompt: Optional[str] = None,
-        prompt_token_ids: Optional[List[int]] = None,
-        sampling_params: Optional[Any] = None,
-        arrival_time: Optional[float] = None,
-        lora_request: Optional[Any] = None,
-        multi_modal_data: Optional[Dict[str, Any]] = None,
-        multi_modal_placeholders: Optional[Dict[str, str]] = None,
-        priority: int = 0,
-        # vLLM-omni specific parameters
-        request_type: RequestType = RequestType.TEXT,
-        processing_stage: ProcessingStage = ProcessingStage.PREPROCESSING,
-        multimodal_inputs: Optional[List[MultimodalData]] = None,
-        diffusion_params: Optional[DiffusionParams] = None,
-        output_format: str = "text",
-        cache_key: Optional[str] = None,
-        stage_configs: Optional[Dict[str, Any]] = None,
-        **kwargs
-    ):
-        # Initialize parent class
-        super().__init__(
-            request_id=request_id,
-            prompt=prompt,
-            prompt_token_ids=prompt_token_ids,
-            sampling_params=sampling_params,
-            arrival_time=arrival_time or time.time(),
-            lora_request=lora_request,
-            multi_modal_data=multi_modal_data,
-            multi_modal_placeholders=multi_modal_placeholders,
-            priority=priority,
-            **kwargs
-        )
-        
-        # vLLM-omni specific attributes
-        self.request_type = request_type
-        self.processing_stage = processing_stage
-        self.multimodal_inputs = multimodal_inputs or []
-        self.diffusion_params = diffusion_params or DiffusionParams()
-        self.output_format = output_format
-        self.cache_key = cache_key
-        self.stage_configs = stage_configs or {}
-        
-        # Processing state
-        self.current_stage = 0
-        self.stage_results = {}
-        self.hidden_states = None
-        self.intermediate_outputs = []
-        
-        # Timing and metrics
-        self.stage_timings = {}
-        self.total_processing_time = 0.0
-        
-        # Error handling
-        self.errors = []
-        self.retry_count = 0
-        self.max_retries = 3
-    
-    def add_multimodal_input(self, data: MultimodalData) -> None:
-        """Add a multimodal input to the request."""
-        self.multimodal_inputs.append(data)
-    
-    def get_multimodal_inputs_by_type(self, data_type: str) -> List[MultimodalData]:
-        """Get all multimodal inputs of a specific type."""
-        return [inp for inp in self.multimodal_inputs if inp.data_type == data_type]
-    
-    def update_processing_stage(self, stage: ProcessingStage) -> None:
-        """Update the current processing stage."""
-        self.processing_stage = stage
-        self.stage_timings[stage.value] = time.time()
-    
-    def add_stage_result(self, stage: str, result: Any) -> None:
-        """Add a result from a processing stage."""
-        self.stage_results[stage] = result
-        self.intermediate_outputs.append({
-            'stage': stage,
-            'result': result,
-            'timestamp': time.time()
-        })
-    
-    def get_stage_result(self, stage: str) -> Any:
-        """Get a result from a specific processing stage."""
-        return self.stage_results.get(stage)
-    
-    def set_hidden_states(self, hidden_states: Any) -> None:
-        """Set hidden states for the request."""
-        self.hidden_states = hidden_states
-    
-    def get_hidden_states(self) -> Any:
-        """Get hidden states for the request."""
-        return self.hidden_states
-    
-    def add_error(self, error: str) -> None:
-        """Add an error to the request."""
-        self.errors.append({
-            'error': error,
-            'timestamp': time.time(),
-            'stage': self.processing_stage.value
-        })
-    
-    def has_errors(self) -> bool:
-        """Check if the request has any errors."""
-        return len(self.errors) > 0
-    
-    def can_retry(self) -> bool:
-        """Check if the request can be retried."""
-        return self.retry_count < self.max_retries
-    
-    def increment_retry(self) -> None:
-        """Increment the retry count."""
-        self.retry_count += 1
-    
-    def generate_cache_key(self) -> str:
-        """Generate a cache key for the request."""
-        if self.cache_key:
-            return self.cache_key
-        
-        # Generate cache key based on request content
-        key_parts = [
-            self.request_id,
-            str(self.request_type.value),
-            str(self.prompt_token_ids) if self.prompt_token_ids else str(self.prompt),
-            str(self.diffusion_params.num_inference_steps) if self.diffusion_params else "0",
-            str(len(self.multimodal_inputs))
-        ]
-        return "_".join(key_parts)
-    
-    def get_processing_time(self) -> float:
-        """Get the total processing time for the request."""
-        if self.stage_timings:
-            return time.time() - min(self.stage_timings.values())
-        return 0.0
-    
-    def is_completed(self) -> bool:
-        """Check if the request is completed."""
-        return self.processing_stage == ProcessingStage.COMPLETED
-    
-    def to_dict(self) -> Dict[str, Any]:
-        """Convert the request to a dictionary for serialization."""
-        return {
-            'request_id': self.request_id,
-            'request_type': self.request_type.value,
-            'processing_stage': self.processing_stage.value,
-            'prompt': self.prompt,
-            'prompt_token_ids': self.prompt_token_ids,
-            'output_format': self.output_format,
-            'multimodal_inputs_count': len(self.multimodal_inputs),
-            'diffusion_params': self.diffusion_params.__dict__ if self.diffusion_params else None,
-            'stage_results': list(self.stage_results.keys()),
-            'has_errors': self.has_errors(),
-            'retry_count': self.retry_count,
-            'processing_time': self.get_processing_time()
-        }
-    
-    def __repr__(self) -> str:
-        """String representation of the request."""
-        return (f"OmniRequest(id={self.request_id}, "
-                f"type={self.request_type.value}, "
-                f"stage={self.processing_stage.value}, "
-                f"multimodal_inputs={len(self.multimodal_inputs)})")
-
-
-# Factory functions for creating different types of requests
-def create_text_request(
-    request_id: str,
-    prompt: str,
-    sampling_params: Optional[Any] = None,
-    **kwargs
-) -> OmniRequest:
-    """Create a text-only request."""
-    return OmniRequest(
-        request_id=request_id,
-        prompt=prompt,
-        request_type=RequestType.TEXT,
-        sampling_params=sampling_params,
-        **kwargs
-    )
-
-
-def create_image_request(
-    request_id: str,
-    prompt: str,
-    image_data: Any,
-    diffusion_params: Optional[DiffusionParams] = None,
-    **kwargs
-) -> OmniRequest:
-    """Create an image generation request."""
-    multimodal_input = MultimodalData(
-        data_type="image",
-        data=image_data,
-        metadata={"is_input": True}
-    )
-    
-    return OmniRequest(
-        request_id=request_id,
-        prompt=prompt,
-        request_type=RequestType.IMAGE,
-        multimodal_inputs=[multimodal_input],
-        diffusion_params=diffusion_params,
-        output_format="image",
-        **kwargs
-    )
-
-
-def create_multimodal_request(
-    request_id: str,
-    prompt: str,
-    multimodal_inputs: List[MultimodalData],
-    **kwargs
-) -> OmniRequest:
-    """Create a multimodal request."""
-    return OmniRequest(
-        request_id=request_id,
-        prompt=prompt,
-        request_type=RequestType.MULTIMODAL,
-        multimodal_inputs=multimodal_inputs,
-        **kwargs
-    )
-
-
-def create_diffusion_request(
-    request_id: str,
-    prompt: str,
-    diffusion_params: DiffusionParams,
-    **kwargs
-) -> OmniRequest:
-    """Create a diffusion model request."""
-    return OmniRequest(
-        request_id=request_id,
-        prompt=prompt,
-        request_type=RequestType.DIFFUSION,
-        diffusion_params=diffusion_params,
-        **kwargs
-    )

vllm_omni/worker/gpu_ar_model_runner.py

@@ -1,578 +0,0 @@
-"""AR GPU Model Runner for vLLM-omni.
-
-Exposes per-request hidden representations via ModelRunnerOutput.pooler_output
-and also outputs sampled tokens.
-"""
-
-from __future__ import annotations
-
-from typing import Optional, Union, Any, List
-import numpy as np
-
-import torch
-
-from vllm import envs
-from vllm.v1.worker.gpu_model_runner import (
-    EMPTY_MODEL_RUNNER_OUTPUT,
-    IntermediateTensors,
-    get_pp_group,
-    get_tp_group,
-    has_kv_transfer_group,
-    set_forward_context,
-)
-from vllm.v1.core.sched.output import SchedulerOutput
-from vllm.multimodal.inputs import MultiModalKwargs
-from vllm.v1.attention.backends.utils import CommonAttentionMetadata
-from vllm.v1.spec_decode.eagle import EagleProposer
-
-from vllm_omni.engine import PromptEmbedsPayload, AdditionalInformationPayload
-from vllm_omni.outputs import OmniModelRunnerOutput
-from vllm_omni.worker.gpu_model_runner import OmniGPUModelRunner
-
-
-class GPUARModelRunner(OmniGPUModelRunner):
-    """Autoregressive GPU model runner that returns hidden states per request.
-
-    This runner follows the same preparation and forward path as GPUModelRunner
-    (inputs assembly, multi-modal handling, TP/PP/DP integration, CUDA graphs),
-    and additionally performs lightweight sampling so that sampled tokens are
-    available in outputs. Hidden representations are taken at the same indices
-    that GPUModelRunner would use for sampling/logits (i.e. `logits_indices`).
-    """
-
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        self.input_ids = torch.zeros(self.max_num_tokens,
-                                     dtype=torch.int64,
-                                     device=self.device)
-
-    @torch.inference_mode()
-    def execute_model(
-        self,
-        scheduler_output: "SchedulerOutput",
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> Union[OmniModelRunnerOutput, IntermediateTensors]:
-        # Update internal state with the new schedule
-        self._update_states(scheduler_output)
-
-        # Decode per-request prompt_embeds / additional_hidden_states payloads (if present) into CPU tensors
-        try:
-            new_reqs = getattr(scheduler_output, "scheduled_new_reqs", [])
-            if new_reqs:
-                for nr in new_reqs:
-                    req_id = getattr(nr, "req_id", None) or getattr(nr, "request_id", None)
-                    if req_id is None:
-                        continue
-                    # prompt_embeds
-                    payload_pe = getattr(nr, "prompt_embeds", None)
-                    if payload_pe is not None:
-                        if isinstance(payload_pe, torch.Tensor):
-                            pe_cpu = payload_pe.detach().to("cpu").contiguous()
-                        elif isinstance(payload_pe, PromptEmbedsPayload):
-                            dt = np.dtype(getattr(payload_pe, "dtype", "float32"))
-                            arr = np.frombuffer(payload_pe.data, dtype=dt)
-                            arr = arr.reshape(payload_pe.shape)
-                            pe_cpu = torch.from_numpy(arr)
-                        else:
-                            pe_cpu = None
-                        if pe_cpu is not None and req_id in self.requests:
-                            setattr(self.requests[req_id], "prompt_embeds_cpu", pe_cpu)
-                    # additional_information
-                    payload_info = getattr(nr, "additional_information", None)
-                    if payload_info is not None:
-                        info_dict = {}
-                        if isinstance(payload_info, dict):
-                            # Already decoded
-                            info_dict = payload_info
-                        elif isinstance(payload_info, AdditionalInformationPayload):
-                            for k, entry in payload_info.entries.items():
-                                if entry.tensor_data is not None:
-                                    dt = np.dtype(getattr(entry, "tensor_dtype", "float32"))
-                                    arr = np.frombuffer(entry.tensor_data, dtype=dt)
-                                    arr = arr.reshape(entry.tensor_shape)
-                                    info_dict[k] = torch.from_numpy(arr)
-                                else:
-                                    info_dict[k] = entry.list_data
-                        if info_dict and req_id in self.requests:
-                            setattr(self.requests[req_id], "additional_information_cpu", info_dict)
-        except Exception:
-            pass
-
-        # If there's no work to do, either return empty output or kv-only path
-        if not scheduler_output.total_num_scheduled_tokens:
-            if not has_kv_transfer_group():
-                return EMPTY_MODEL_RUNNER_OUTPUT
-            return self.kv_connector_no_forward(scheduler_output,
-                                                self.vllm_config)
-
-        # Prepare decoder inputs and attention metadata
-        (attn_metadata, attention_cuda_graphs, logits_indices,
-         spec_decode_metadata, num_scheduled_tokens_np,
-         spec_decode_common_attn_metadata) = self._prepare_inputs(
-             scheduler_output)
-
-        # Determine number of input tokens for this iteration
-        num_scheduled_tokens = scheduler_output.total_num_scheduled_tokens
-        if (self.use_cuda_graph
-                and num_scheduled_tokens <= self.cudagraph_batch_sizes[-1]):
-            num_input_tokens = self.vllm_config.pad_for_cudagraph(
-                num_scheduled_tokens)
-        else:
-            tp_size = self.vllm_config.parallel_config.tensor_parallel_size
-            if (self.compilation_config.pass_config.enable_sequence_parallelism
-                    and tp_size > 1):
-                from vllm.utils import round_up  # lazy local import
-                num_input_tokens = round_up(num_scheduled_tokens, tp_size)
-            else:
-                num_input_tokens = num_scheduled_tokens
-
-        # DP padding
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_input_tokens)
-        num_input_tokens += num_pad
-
-        # Multimodal handling (encode and gather embeddings if needed)
-        if self.is_multimodal_model:
-            self._execute_mm_encoder(scheduler_output)
-            mm_embeds = self._gather_mm_embeddings(scheduler_output)
-        else:
-            mm_embeds = []
-
-        # Always assemble inputs_embeds on first PP rank; overlay per-request prompt_embeds and collect additional_hidden_states for prefill only
-        if get_pp_group().is_first_rank:
-            inputs_embeds_scheduled = self.model.get_input_embeddings(
-                input_ids=self.input_ids[:num_scheduled_tokens],
-                multimodal_embeddings=(mm_embeds or None)
-                if self.is_multimodal_model else None,
-            )
-
-            # Copy into persistent buffer to enable CUDA Graph capture
-            self.inputs_embeds[:num_scheduled_tokens].copy_(
-                inputs_embeds_scheduled)
-
-            # Reset per-step additional information collector
-            if hasattr(self, "_forward_additional_information"):
-                self._forward_additional_information = None
-
-            # Overlay custom prompt_embeds per request for the prompt portion; collect additional_information (tensor/list) for prefill portion only
-            for req_index, req_id in enumerate(self.input_batch.req_ids):
-                req_state = self.requests[req_id]
-                pe_cpu = getattr(req_state, "prompt_embeds_cpu", None)
-                addi_cpu = getattr(req_state, "additional_information_cpu", None)
-                num_computed_tokens = int(
-                    self.input_batch.num_computed_tokens_cpu[req_index])
-                prompt_len = len(req_state.prompt_token_ids)
-                prompt_remaining = max(0, prompt_len - num_computed_tokens)
-                sched_tokens = int(num_scheduled_tokens_np[req_index])
-                overlay_len = min(sched_tokens, prompt_remaining)
-                if overlay_len <= 0:
-                    continue
-                if pe_cpu is not None:
-                    src = pe_cpu[num_computed_tokens:
-                                 num_computed_tokens + overlay_len].to(
-                                      dtype=self.dtype,
-                                      device=self.device,
-                                      non_blocking=True)
-                    start_offset = int(self.query_start_loc_cpu[req_index])
-                    self.inputs_embeds[start_offset:start_offset + overlay_len] \
-                        .copy_(src)
-                # For additional_information: handle arbitrary keys
-                if addi_cpu is not None and isinstance(addi_cpu, dict):
-                    # Lazy init collector dict
-                    if not hasattr(self, "_forward_additional_information") or \
-                       self._forward_additional_information is None:
-                        self._forward_additional_information = {}
-                    # Process tensors (slice by scheduled prompt range) and lists (append per-request)
-                    for k, v in addi_cpu.items():
-                        if isinstance(v, torch.Tensor):
-                            # Slice along token dimension for prefill part
-                            try:
-                                seg = v[num_computed_tokens:
-                                        num_computed_tokens + overlay_len].to(
-                                            dtype=self.dtype,
-                                            device=self.device,
-                                            non_blocking=True)
-                            except Exception:
-                                # Fallback: move whole tensor if slicing fails
-                                seg = v.to(dtype=self.dtype,
-                                           device=self.device,
-                                           non_blocking=True)
-                            prev_val = self._forward_additional_information.get(k)
-                            self._forward_additional_information[k] = (
-                                torch.cat([prev_val, seg], dim=0)
-                                if isinstance(prev_val, torch.Tensor) else seg.clone())
-                        elif isinstance(v, list):
-                            prev_val = self._forward_additional_information.get(k)
-                            if prev_val is None:
-                                self._forward_additional_information[k] = [v]
-                            elif isinstance(prev_val, list):
-                                self._forward_additional_information[k].append(v)
-                            else:
-                                # Mixed types: wrap existing into list
-                                self._forward_additional_information[k] = [prev_val, v]
-
-            input_ids = self.input_ids[:num_input_tokens]  # preserved for APIs
-            inputs_embeds = self.inputs_embeds[:num_input_tokens]
-            model_mm_kwargs = (self._extract_mm_kwargs(scheduler_output)
-                               if self.is_multimodal_model else {})
-        else:
-            input_ids = self.input_ids[:num_input_tokens]
-            inputs_embeds = None
-            model_mm_kwargs = {}
-
-        # Positions/mRoPE
-        if self.uses_mrope:
-            positions = self.mrope_positions[:, :num_input_tokens]
-        else:
-            positions = self.positions[:num_input_tokens]
-
-        # Handle pipeline-parallel intermediate tensors
-        if get_pp_group().is_first_rank:
-            intermediate_tensors = None
-        else:
-            intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                num_input_tokens, intermediate_tensors, True)
-
-        # Some attention backends only support CUDA Graphs in pure decode.
-        skip_cuda_graphs = self.full_cuda_graph and not attention_cuda_graphs
-
-        # Forward pass
-        with set_forward_context(
-                attn_metadata,
-                self.vllm_config,
-                num_tokens=num_input_tokens,
-                num_tokens_across_dp=num_tokens_across_dp,
-                skip_cuda_graphs=skip_cuda_graphs,
-        ), self.maybe_get_kv_connector_output(
-                scheduler_output) as kv_connector_output:
-
-            model_kwargs_extra = {}
-            # Only pass additional_information for the prefill part
-            if hasattr(self, "_forward_additional_information") and \
-               self._forward_additional_information is not None and \
-               isinstance(self._forward_additional_information, dict):
-                model_kwargs_extra["additional_information"] = self._forward_additional_information
-            model_output = self.model(
-                input_ids=input_ids,
-                positions=positions,
-                intermediate_tensors=intermediate_tensors,
-                inputs_embeds=inputs_embeds,
-                **MultiModalKwargs.as_kwargs(
-                    model_mm_kwargs,
-                    device=self.device,
-                ),
-                sampling_metadata=self.input_batch.sampling_metadata,
-                logits_index=logits_indices,
-                sampler=self.sampler,
-                **model_kwargs_extra,
-            )
-
-        if self.use_aux_hidden_state_outputs:
-            hidden_states, _aux_hidden_states = model_output
-        else:
-            hidden_states = model_output
-            _aux_hidden_states = None
-
-        text_hidden_states, multimodal_outputs = (
-            self.extract_multimodal_outputs(hidden_states))
-
-        # Mid PP stages return intermediate tensors unmodified
-        if not get_pp_group().is_last_rank:
-            assert isinstance(text_hidden_states, IntermediateTensors)
-            text_hidden_states.kv_connector_output = kv_connector_output
-            return text_hidden_states
-
-        # Broadcast PP output for external_launcher (torchrun)
-        broadcast_pp_output = \
-            self.parallel_config.distributed_executor_backend \
-            == "external_launcher" and len(get_pp_group().ranks) > 0
-        if not get_pp_group().is_last_rank:
-            assert isinstance(text_hidden_states, IntermediateTensors)
-            if not broadcast_pp_output:
-                text_hidden_states.kv_connector_output = kv_connector_output
-                return text_hidden_states
-            get_pp_group().send_tensor_dict(text_hidden_states.tensors,
-                                            all_gather_group=get_tp_group())
-            logits = None
-        else:
-            if self.input_batch.pooling_params:
-                return self._pool(text_hidden_states, num_scheduled_tokens,
-                                  num_scheduled_tokens_np, kv_connector_output)
-
-            sample_hidden_states = text_hidden_states[logits_indices]
-            logits = self.model.compute_logits(sample_hidden_states, None)
-        if broadcast_pp_output:
-            model_output_broadcast_data = {
-                "logits": logits.contiguous(),
-            } if logits is not None else {}
-            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
-                model_output_broadcast_data, src=len(get_pp_group().ranks) - 1)
-            assert model_output_broadcast_data is not None
-            logits = model_output_broadcast_data["logits"]
-
-        # Apply structured output bitmasks if present
-        if scheduler_output.grammar_bitmask is not None:
-            self.apply_grammar_bitmask(scheduler_output, logits)
-
-        # Sample the next token and get logprobs if needed (with spec decode)
-        sampling_metadata = self.input_batch.sampling_metadata
-        if spec_decode_metadata is None:
-            sampler_output = self.sampler(
-                logits=logits,
-                sampling_metadata=sampling_metadata,
-            )
-        else:
-            assert logits is not None
-            bonus_logits = logits[spec_decode_metadata.bonus_logits_indices]
-            sampler_output = self.sampler(
-                logits=bonus_logits,
-                sampling_metadata=sampling_metadata,
-            )
-            bonus_token_ids = sampler_output.sampled_token_ids
-            target_logits = logits[spec_decode_metadata.target_logits_indices]
-            output_token_ids = self.rejection_sampler(
-                spec_decode_metadata,
-                None,
-                target_logits,
-                bonus_token_ids,
-                sampling_metadata,
-            )
-            sampler_output.sampled_token_ids = output_token_ids
-
-        num_nans_in_logits = {}
-        if envs.VLLM_COMPUTE_NANS_IN_LOGITS:
-            num_nans_in_logits = self._get_nans_in_logits(logits)
-
-        # Handle partial prefill: discard sampled tokens and rewind RNG
-        discard_sampled_tokens_req_indices = []
-        for i, req_id in enumerate(self.input_batch.req_ids):
-            req_state = self.requests[req_id]
-            seq_len = (req_state.num_computed_tokens +
-                       scheduler_output.num_scheduled_tokens[req_id])
-            if seq_len < req_state.num_tokens:
-                generator = self.input_batch.generators.get(i)
-                if generator is not None:
-                    generator.set_offset(generator.get_offset() - 4)
-                discard_sampled_tokens_req_indices.append(i)
-
-        # Move CPU sync parts
-        logprobs_tensors = sampler_output.logprobs_tensors
-        logprobs_lists = logprobs_tensors.tolists() \
-            if logprobs_tensors is not None else None
-
-        # Prompt logprobs if needed
-        prompt_logprobs_dict = self._get_prompt_logprobs_dict(
-            text_hidden_states[:num_scheduled_tokens],
-            scheduler_output,
-        )
-
-        # Parse valid sampled tokens
-        sampled_token_ids = sampler_output.sampled_token_ids
-        max_gen_len = sampled_token_ids.shape[-1]
-        if max_gen_len == 1:
-            valid_sampled_token_ids = sampled_token_ids.tolist()
-        else:
-            valid_sampled_token_ids = self.rejection_sampler.parse_output(
-                sampled_token_ids,
-                self.input_batch.vocab_size,
-            )
-        for i in discard_sampled_tokens_req_indices:
-            valid_sampled_token_ids[i].clear()
-
-        # Cache sampled tokens
-        for req_idx, sampled_ids in enumerate(valid_sampled_token_ids):
-            if not sampled_ids:
-                continue
-            start_idx = self.input_batch.num_tokens_no_spec[req_idx]
-            end_idx = start_idx + len(sampled_ids)
-            self.input_batch.token_ids_cpu[req_idx, start_idx:end_idx] = sampled_ids
-            self.input_batch.num_tokens_no_spec[req_idx] = end_idx
-            self.input_batch.num_tokens[req_idx] = end_idx
-            req_id = self.input_batch.req_ids[req_idx]
-            req_state = self.requests[req_id]
-            req_state.output_token_ids.extend(sampled_ids)
-
-        # Speculative decoding draft tokens if configured
-        if not self.speculative_config:
-            spec_token_ids = None
-        else:
-            assert spec_decode_common_attn_metadata is not None
-            spec_token_ids = self.propose_draft_token_ids(
-                scheduler_output,
-                valid_sampled_token_ids,
-                sampling_metadata,
-                text_hidden_states,
-                sample_hidden_states,
-                _aux_hidden_states,
-                spec_decode_metadata,
-                spec_decode_common_attn_metadata,
-            )
-
-        # Convert to per-request tensors on CPU
-        text_hidden_states_cpu = text_hidden_states.detach().to("cpu").contiguous()
-        pooler_output: list[Optional[torch.Tensor]] = []
-        prev_logits_index = 0
-        for logits_index in logits_indices:
-            pooler_output.append(text_hidden_states_cpu[prev_logits_index:logits_index+1])
-            prev_logits_index = logits_index + 1
-
-
-        self.eplb_step()
-
-        return OmniModelRunnerOutput(
-            req_ids=self.input_batch.req_ids,
-            req_id_to_index=self.input_batch.req_id_to_index,
-            sampled_token_ids=valid_sampled_token_ids,
-            spec_token_ids=spec_token_ids,
-            logprobs=logprobs_lists,
-            prompt_logprobs_dict=prompt_logprobs_dict,
-            pooler_output=pooler_output if self.vllm_config.model_config.engine_output_type != "text" else None,
-            kv_connector_output=kv_connector_output,
-            num_nans_in_logits=num_nans_in_logits,
-        )
-    
-    @torch.inference_mode()
-    def extract_multimodal_outputs(self, hidden_states: Union[torch.Tensor, List[torch.Tensor]]) -> dict:
-        if hasattr(self.model, "have_multimodal_outputs") and self.model.have_multimodal_outputs:
-            text_hidden_states = hidden_states.text_hidden_states
-            multimodal_outputs = hidden_states.multimodal_outputs
-
-        elif isinstance(hidden_states, torch.Tensor):
-            text_hidden_states = hidden_states
-            multimodal_outputs = {}
-        elif isinstance(hidden_states, List):
-            text_hidden_states = hidden_states[0]
-            multimodal_outputs = {}
-        else:
-            raise ValueError(f"Invalid hidden states type: {type(hidden_states)}")
-        return text_hidden_states, multimodal_outputs
-    
-    @torch.inference_mode()
-    def _dummy_run(
-        self,
-        num_tokens: int,
-        capture_attn_cudagraph: bool = False,
-        skip_eplb: bool = False,
-        is_profile: bool = False,
-    ) -> tuple[torch.Tensor, torch.Tensor]:
-
-        # Padding for DP
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_tokens)
-        num_tokens += num_pad
-
-        # Set num_scheduled_tokens based on num_tokens and max_num_seqs
-        # for dummy run with LoRA so that the num_reqs collectively
-        # has num_tokens in total.
-        assert num_tokens <= self.scheduler_config.max_num_batched_tokens
-        max_num_reqs = self.scheduler_config.max_num_seqs
-        num_reqs = min(num_tokens, max_num_reqs)
-        min_tokens_per_req = num_tokens // num_reqs
-        num_scheduled_tokens_list = [min_tokens_per_req] * num_reqs
-        num_scheduled_tokens_list[-1] += num_tokens % num_reqs
-        assert sum(num_scheduled_tokens_list) == num_tokens
-        assert len(num_scheduled_tokens_list) == num_reqs
-        num_scheduled_tokens = np.array(num_scheduled_tokens_list,
-                                        dtype=np.int32)
-
-        attn_metadata: Optional[dict[str, Any]] = None
-        if capture_attn_cudagraph:
-            attn_metadata = {}
-
-            # Make sure max_model_len is used at the graph capture time.
-            self.seq_lens_np[:num_reqs] = self.max_model_len
-            self.seq_lens_np[num_reqs:] = 0
-            self.seq_lens[:num_reqs].copy_(self.seq_lens_cpu[:num_reqs],
-                                           non_blocking=True)
-
-            for kv_cache_group_id, kv_cache_group_spec in enumerate(
-                    self.kv_cache_config.kv_cache_groups):
-                common_attn_metadata = CommonAttentionMetadata(
-                    query_start_loc=self.query_start_loc[:num_reqs + 1],
-                    query_start_loc_cpu=self.query_start_loc_cpu[:num_reqs +
-                                                                 1],
-                    seq_lens=self.seq_lens[:num_reqs],
-                    seq_lens_cpu=self.seq_lens_cpu[:num_reqs],
-                    num_computed_tokens_cpu=self.input_batch.
-                    num_computed_tokens_cpu_tensor[:num_reqs],
-                    num_reqs=num_reqs,
-                    num_actual_tokens=num_tokens,
-                    max_query_len=num_tokens,
-                    block_table_tensor=self.input_batch.block_table[
-                        kv_cache_group_id].get_device_tensor()[:num_reqs],
-                    slot_mapping=self.input_batch.
-                    block_table[kv_cache_group_id].slot_mapping[:num_tokens],
-                    causal=True)
-
-                for attn_group in self.attn_groups[kv_cache_group_id]:
-                    attn_metadata_i = attn_group.metadata_builder\
-                        .build_for_cudagraph_capture(common_attn_metadata)
-                    for layer_name in kv_cache_group_spec.layer_names:
-                        attn_metadata[layer_name] = attn_metadata_i
-
-        with self.maybe_dummy_run_with_lora(self.lora_config,
-                                            num_scheduled_tokens):
-            if self.is_multimodal_model:
-                input_ids = None
-                inputs_embeds = self.inputs_embeds[:num_tokens]
-                model_mm_kwargs = self._dummy_mm_kwargs(num_reqs)
-            else:
-                input_ids = self.input_ids[:num_tokens]
-                inputs_embeds = None
-                model_mm_kwargs = {}
-
-            if self.uses_mrope:
-                positions = self.mrope_positions[:, :num_tokens]
-            else:
-                positions = self.positions[:num_tokens]
-
-            if get_pp_group().is_first_rank:
-                intermediate_tensors = None
-            else:
-                if self.intermediate_tensors is None:
-                    self.intermediate_tensors = (
-                        self.model.make_empty_intermediate_tensors(
-                            batch_size=self.max_num_tokens,
-                            dtype=self.model_config.dtype,
-                            device=self.device))
-
-                intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                    num_tokens, None, False)
-
-            with self.maybe_randomize_inputs(input_ids), set_forward_context(
-                    attn_metadata,
-                    self.vllm_config,
-                    num_tokens=num_tokens,
-                    num_tokens_across_dp=num_tokens_across_dp):
-                outputs = self.model(
-                    input_ids=input_ids,
-                    positions=positions,
-                    intermediate_tensors=intermediate_tensors,
-                    inputs_embeds=inputs_embeds,
-                    **MultiModalKwargs.as_kwargs(
-                        model_mm_kwargs,
-                        device=self.device,
-                    ),
-                )
-
-            if self.use_aux_hidden_state_outputs:
-                hidden_states, _ = outputs
-            else:
-                hidden_states = outputs
-
-            if self.speculative_config and self.speculative_config.use_eagle():
-                assert isinstance(self.drafter, EagleProposer)
-                self.drafter.dummy_run(num_tokens)
-
-        # This is necessary to avoid blocking DP.
-        # For dummy runs, we typically skip EPLB since we don't have any real
-        # requests to process.
-        # However, in DP settings, there may be cases when some DP ranks do
-        # not have any requests to process, so they're executing dummy batches.
-        # In such cases, we still have to trigger EPLB to make sure
-        # ranks execute the rearrangement in synchronization.
-        if not skip_eplb:
-            self.eplb_step(is_dummy=True, is_profile=is_profile)
-
-        logit_indices = np.cumsum(num_scheduled_tokens) - 1
-        hidden_states, multimodal_outputs = self.extract_multimodal_outputs(hidden_states)
-        return hidden_states, hidden_states[logit_indices]
-

vllm_omni/worker/gpu_ar_worker.py

@@ -1,67 +0,0 @@
-from vllm.v1.worker.gpu_worker import Worker as GPUWorker
-import torch
-import os
-import gc
-from vllm.utils import GiB_bytes, MemorySnapshot
-from vllm.platforms import current_platform
-from vllm.worker.worker import _check_if_gpu_supports_dtype
-from vllm.v1.worker.gpu_worker import init_worker_distributed_environment
-from vllm.model_executor import set_random_seed
-from vllm.v1.utils import report_usage_stats
-
-from vllm_omni.worker.gpu_ar_model_runner import GPUARModelRunner
-
-
-class GPUARWorker(GPUWorker):
-    def init_device(self):
-        if self.device_config.device.type == "cuda":
-            # torch.distributed.all_reduce does not free the input tensor until
-            # the synchronization point. This causes the memory usage to grow
-            # as the number of all_reduce calls increases. This env var disables
-            # this behavior.
-            # Related issue:
-            # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573
-            os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1"
-
-            # This env var set by Ray causes exceptions with graph building.
-            os.environ.pop("NCCL_ASYNC_ERROR_HANDLING", None)
-            self.device = torch.device(f"cuda:{self.local_rank}")
-            current_platform.set_device(self.device)
-
-            _check_if_gpu_supports_dtype(self.model_config.dtype)
-            gc.collect()
-            torch.cuda.empty_cache()
-
-            # take current memory snapshot
-            self.init_snapshot = MemorySnapshot()
-            self.requested_memory = (self.init_snapshot.total_memory *
-                                     self.cache_config.gpu_memory_utilization)
-            if self.init_snapshot.free_memory < self.requested_memory:
-                GiB = lambda b: round(b / GiB_bytes, 2)
-                raise ValueError(
-                    f"Free memory on device "
-                    f"({GiB(self.init_snapshot.free_memory)}/"
-                    f"{GiB(self.init_snapshot.total_memory)} GiB) on startup "
-                    f"is less than desired GPU memory utilization "
-                    f"({self.cache_config.gpu_memory_utilization}, "
-                    f"{GiB(self.requested_memory)} GiB). Decrease GPU memory "
-                    f"utilization or reduce GPU memory used by other processes."
-                )
-        else:
-            raise RuntimeError(
-                f"Not support device type: {self.device_config.device}")
-        # Initialize the distributed environment.
-        init_worker_distributed_environment(self.vllm_config, self.rank,
-                                            self.distributed_init_method,
-                                            self.local_rank,
-                                            current_platform.dist_backend)
-        # Set random seed.
-        set_random_seed(self.model_config.seed)
-
-        # Construct the model runner
-        self.model_runner: GPUARModelRunner = GPUARModelRunner(
-            self.vllm_config, self.device)
-
-        if self.rank == 0:
-            # If usage stat is enabled, collect relevant info.
-            report_usage_stats(self.vllm_config)

vllm_omni/worker/gpu_diffusion_model_runner.py

@@ -1,362 +0,0 @@
-from __future__ import annotations
-
-from typing import Optional, Union, List
-import numpy as np
-import gc
-import logging
-
-import torch
-
-from vllm.v1.worker.gpu_model_runner import (
-    EMPTY_MODEL_RUNNER_OUTPUT,
-    IntermediateTensors,
-    get_pp_group,
-    set_forward_context,
-)
-from vllm.v1.worker.utils import sanity_check_mm_encoder_outputs
-from vllm.multimodal.inputs import MultiModalKwargs
-
-from vllm.v1.core.sched.output import SchedulerOutput
-from vllm.v1.attention.backends.utils import CommonAttentionMetadata
-
-from vllm_omni.outputs import OmniModelRunnerOutput
-from vllm_omni.worker.gpu_model_runner import OmniGPUModelRunner
-
-
-logger = logging.getLogger(__name__)
-
-
-class GPUDiffusionModelRunner(OmniGPUModelRunner):
-    """Diffusion model runner for vLLM-omni (non-autoregressive).
-
-    - Reuses GPUModelRunner preparation, multimodal handling, and TP/PP/DP glue.
-    - Does not compute logits or perform token sampling.
-    - Executes diffusion process and returns tensors via `pooler_output`.
-    """
-
-    @torch.inference_mode()
-    def execute_model(
-        self,
-        scheduler_output: "SchedulerOutput",
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> Union[OmniModelRunnerOutput, IntermediateTensors]:
-        self._update_states(scheduler_output)
-
-        if not scheduler_output.total_num_scheduled_tokens:
-
-            return EMPTY_MODEL_RUNNER_OUTPUT
-           
-        # Prepare decoder inputs and attention metadata (for batch/order mapping)
-        (attn_metadata, attention_cuda_graphs, logits_indices,
-         spec_decode_metadata, num_scheduled_tokens_np,
-         spec_decode_common_attn_metadata) = self._prepare_inputs(
-             scheduler_output)
-
-        # Input token count for this iteration (not used by diffusion, but
-        # retained to keep DP padding/ordering consistent)
-        num_input_tokens = scheduler_output.total_num_scheduled_tokens
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_input_tokens)
-        num_input_tokens += num_pad
-
-        # Multimodal conditioning (e.g., text/audio/video encoders)
-        if self.is_multimodal_model:
-            self._execute_mm_encoder(scheduler_output)
-            mm_embeds = self._gather_mm_embeddings(scheduler_output)
-        else:
-            mm_embeds = []
-
-        # Build inputs to mirror AR runner: input_ids/positions/embeds
-        if self.is_multimodal_model and get_pp_group().is_first_rank:
-            inputs_embeds_scheduled = self.model.get_input_embeddings(
-                input_ids=self.input_ids[:scheduler_output.total_num_scheduled_tokens],
-                multimodal_embeddings=mm_embeds or None,
-            )
-            self.inputs_embeds[:scheduler_output.total_num_scheduled_tokens].copy_(
-                inputs_embeds_scheduled)
-            input_ids = self.input_ids[:num_input_tokens]
-            inputs_embeds = self.inputs_embeds[:num_input_tokens]
-            model_mm_kwargs = self._extract_mm_kwargs(scheduler_output)
-        else:
-            input_ids = self.input_ids[:num_input_tokens]
-            inputs_embeds = None
-            model_mm_kwargs = {}
-
-        # Positions (mrope or standard)
-        if self.uses_mrope:
-            positions = self.mrope_positions[:, :num_input_tokens]
-        else:
-            positions = self.positions[:num_input_tokens]
-
-        # Intermediate tensors sync for PP (if any)
-        if get_pp_group().is_first_rank:
-            intermediate_tensors = None
-        else:
-            intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                num_input_tokens, intermediate_tensors, True)
-
-        # Set forward context mainly for resource management and kv connector
-        skip_cuda_graphs = True  # diffusion path does not rely on cuda graphs here
-        with set_forward_context(
-                attn_metadata,
-                self.vllm_config,
-                num_tokens=num_input_tokens,
-                num_tokens_across_dp=num_tokens_across_dp,
-                skip_cuda_graphs=skip_cuda_graphs,
-        ), self.maybe_get_kv_connector_output(
-                scheduler_output) as kv_connector_output:
-
-            if not get_pp_group().is_last_rank:
-                # For non-last PP stages, pass through intermediate tensors.
-                assert intermediate_tensors is not None
-                intermediate_tensors.kv_connector_output = kv_connector_output
-                return intermediate_tensors
-
-            outputs = self._run_diffusion(
-                input_ids=input_ids,
-                positions=positions,
-                intermediate_tensors=intermediate_tensors,
-                inputs_embeds=inputs_embeds,
-                multimodal_kwargs=model_mm_kwargs,
-                logits_indices=logits_indices,
-            )
-        _, multimodal_outputs = (
-            self.extract_multimodal_outputs(outputs))
-
-        # Ensure one tensor per request, map to CPU for output struct
-        pooler_output: List[Optional[torch.Tensor]] = []
-        if isinstance(multimodal_outputs, torch.Tensor):
-            # If model returned a single stacked tensor, split by requests
-            assert multimodal_outputs.shape[0] == self.input_batch.num_reqs
-            for i in range(self.input_batch.num_reqs):
-                pooler_output.append(multimodal_outputs[i].detach().to("cpu").contiguous())
-        elif isinstance(multimodal_outputs, list):
-            for out in multimodal_outputs:
-                pooler_output.append(out.detach().to("cpu").contiguous() if out is not None else None)
-        elif isinstance(multimodal_outputs, dict):
-            for out in multimodal_outputs.values():
-                pooler_output.append(out.detach().to("cpu").contiguous() if out is not None else None)
-        else:
-            raise RuntimeError("Unsupported diffusion output type")
-
-        self.eplb_step()
-
-        return OmniModelRunnerOutput(
-            req_ids=self.input_batch.req_ids,
-            req_id_to_index=self.input_batch.req_id_to_index,
-            sampled_token_ids=[],
-            spec_token_ids=None,
-            logprobs=None,
-            prompt_logprobs_dict={},
-            pooler_output=pooler_output,
-            kv_connector_output=kv_connector_output,
-            num_nans_in_logits={},
-        )
-
-    def _run_diffusion(
-        self,
-        *,
-        input_ids: torch.Tensor,
-        positions: torch.Tensor,
-        intermediate_tensors: Optional[IntermediateTensors],
-        inputs_embeds: Optional[torch.Tensor],
-        multimodal_kwargs: dict,
-        logits_indices: torch.Tensor,
-    ) -> Union[torch.Tensor, list[torch.Tensor]]:
-        """Runs the diffusion process and returns per-request tensors.
-
-        Tries model interfaces in the following order for maximal compatibility:
-        1) model.sample(condition=..., **kwargs)
-        2) model.forward(condition=..., **kwargs)
-        3) model.diffuse(condition=..., **kwargs)
-        """
-        # Keep inputs identical to AR runner
-        kwargs = dict(
-            input_ids=input_ids,
-            positions=positions,
-            intermediate_tensors=intermediate_tensors,
-            inputs_embeds=inputs_embeds,
-            **MultiModalKwargs.as_kwargs(multimodal_kwargs, device=self.device),
-            sampling_metadata=self.input_batch.sampling_metadata,
-            logits_index=logits_indices,
-            sampler=self.sampler,
-        )
-
-        # For Qwen 2.5 Omni's current implementation, we only support the forward method
-        if hasattr(self.model, "forward"):
-            return self.model.forward(**kwargs)
-        #TODO: add the diffuse method for other models
-
-        raise RuntimeError(
-            "The loaded model does not expose diffusion interfaces 'sample', "
-            "'forward', or 'diffuse'. Please implement one of them or adapt the runner.")
-
-
-    @torch.inference_mode()
-    def _dummy_run(
-        self,
-        num_tokens: int,
-        capture_attn_cudagraph: bool = False,
-        skip_eplb: bool = False,
-        is_profile: bool = False,
-    ) -> tuple[torch.Tensor, torch.Tensor]:
-        # Padding for DP
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_tokens)
-        num_tokens += num_pad
-
-        # Set num_scheduled_tokens based on num_tokens and max_num_seqs
-        # for dummy run with LoRA so that the num_reqs collectively
-        # has num_tokens in total.
-        assert num_tokens <= self.scheduler_config.max_num_batched_tokens
-        max_num_reqs = self.scheduler_config.max_num_seqs
-        num_reqs = min(num_tokens, max_num_reqs)
-        min_tokens_per_req = num_tokens // num_reqs
-        num_scheduled_tokens_list = [min_tokens_per_req] * num_reqs
-        num_scheduled_tokens_list[-1] += num_tokens % num_reqs
-        assert sum(num_scheduled_tokens_list) == num_tokens
-        assert len(num_scheduled_tokens_list) == num_reqs
-        num_scheduled_tokens = np.array(num_scheduled_tokens_list,
-                                        dtype=np.int32)
-
-        attn_metadata: Optional[dict[str, dict]] = None
-        if capture_attn_cudagraph:
-            attn_metadata = {}
-
-            # Make sure max_model_len is used at the graph capture time.
-            self.seq_lens_np[:num_reqs] = self.max_model_len
-            self.seq_lens_np[num_reqs:] = 0
-            self.seq_lens[:num_reqs].copy_(self.seq_lens_cpu[:num_reqs],
-                                           non_blocking=True)
-
-            for kv_cache_group_id, kv_cache_group_spec in enumerate(
-                    self.kv_cache_config.kv_cache_groups):
-                common_attn_metadata = CommonAttentionMetadata(
-                    query_start_loc=self.query_start_loc[:num_reqs + 1],
-                    query_start_loc_cpu=self.query_start_loc_cpu[:num_reqs + 1],
-                    seq_lens=self.seq_lens[:num_reqs],
-                    seq_lens_cpu=self.seq_lens_cpu[:num_reqs],
-                    num_computed_tokens_cpu=self.input_batch.
-                    num_computed_tokens_cpu_tensor[:num_reqs],
-                    num_reqs=num_reqs,
-                    num_actual_tokens=num_tokens,
-                    max_query_len=num_tokens,
-                    block_table_tensor=self.input_batch.block_table[
-                        kv_cache_group_id].get_device_tensor()[:num_reqs],
-                    slot_mapping=self.input_batch.block_table[
-                        kv_cache_group_id].slot_mapping[:num_tokens],
-                    causal=True)
-
-                for attn_group in self.attn_groups[kv_cache_group_id]:
-                    attn_metadata_i = attn_group.metadata_builder \
-                        .build_for_cudagraph_capture(common_attn_metadata)
-                    for layer_name in kv_cache_group_spec.layer_names:
-                        attn_metadata[layer_name] = attn_metadata_i
-
-        with self.maybe_dummy_run_with_lora(self.lora_config,
-                                            num_scheduled_tokens):
-            if self.is_multimodal_model:
-                input_ids = None
-                inputs_embeds = self.inputs_embeds[:num_tokens]
-                model_mm_kwargs = self._dummy_mm_kwargs(num_reqs)
-            else:
-                input_ids = self.input_ids[:num_tokens]
-                inputs_embeds = None
-                model_mm_kwargs = {}
-
-            if self.uses_mrope:
-                positions = self.mrope_positions[:, :num_tokens]
-            else:
-                positions = self.positions[:num_tokens]
-
-            if get_pp_group().is_first_rank:
-                intermediate_tensors = None
-            else:
-                if self.intermediate_tensors is None:
-                    self.intermediate_tensors = (
-                        self.model.make_empty_intermediate_tensors(
-                            batch_size=self.max_num_tokens,
-                            dtype=self.model_config.dtype,
-                            device=self.device))
-
-                intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                    num_tokens, None, False)
-
-            # Diffusion path: avoid CUDA graphs; we only use context for resource wiring
-            with self.maybe_randomize_inputs(input_ids), set_forward_context(
-                    attn_metadata,
-                    self.vllm_config,
-                    num_tokens=num_tokens,
-                    num_tokens_across_dp=num_tokens_across_dp):
-                outputs = self.model(
-                    input_ids=input_ids,
-                    positions=positions,
-                    intermediate_tensors=intermediate_tensors,
-                    inputs_embeds=inputs_embeds,
-                    **MultiModalKwargs.as_kwargs(
-                        model_mm_kwargs,
-                        device=self.device,
-                    ),
-                    sampler=None,
-                )
-
-            if self.use_aux_hidden_state_outputs:
-                hidden_states, _ = outputs
-            else:
-                hidden_states = outputs
-
-            # Extract multimodal outputs if present; we ignore them here because
-            # dummy run returns tensors only. The actual diffusion runner returns
-            # multimodal outputs via pooler_output in execute_model.
-            text_hidden_states, _ = self.extract_multimodal_outputs(hidden_states)
-
-            if not skip_eplb:
-                self.eplb_step(is_dummy=True, is_profile=is_profile)
-
-        logit_indices = np.cumsum(num_scheduled_tokens) - 1
-        return text_hidden_states, None
-    
-    @torch.inference_mode()
-    def _dummy_sampler_run(self, hidden_states: torch.Tensor) -> None:
-        logger.warning("Dummy sampler run is not implemented for diffusion model")
-        return None
-
-    def profile_run(self) -> None:
-        # Profile with multimodal encoder & encoder cache, similar to base but
-        # without any logits/sampler warming.
-        if self.is_multimodal_model:
-            mm_budget = self.mm_budget
-            assert mm_budget is not None
-
-            # TODO: handle encoder-decoder models once supported.
-            if (encoder_budget := mm_budget.get_encoder_budget()) > 0:
-                (
-                    dummy_modality,
-                    max_tokens,
-                ) = mm_budget.get_modality_with_max_tokens()
-                (
-                    max_mm_items_per_prompt,
-                    max_mm_items_per_batch,
-                ) = mm_budget.get_max_items(dummy_modality, max_tokens)
-
-                batched_dummy_mm_inputs = self._get_mm_dummy_batch(
-                    dummy_modality,
-                    max_mm_items_per_batch,
-                )
-
-                dummy_encoder_outputs = self.model.get_multimodal_embeddings(
-                    **batched_dummy_mm_inputs)
-
-                sanity_check_mm_encoder_outputs(
-                    dummy_encoder_outputs,
-                    expected_num_items=max_mm_items_per_batch,
-                )
-
-                self.encoder_cache["tmp"] = dict(
-                    enumerate(dummy_encoder_outputs))
-
-        hidden_states, _ = self._dummy_run(self.max_num_tokens, is_profile=True)
-        if get_pp_group().is_last_rank:
-            pass  # No sampler/pooler warmup for diffusion
-        self._sync_device()
-        del hidden_states
-        self.encoder_cache.clear()
-        gc.collect()

vllm_omni/worker/gpu_diffusion_worker.py

@@ -1,67 +0,0 @@
-from vllm.v1.worker.gpu_worker import Worker as GPUWorker
-import torch
-import os
-import gc
-from vllm.utils import GiB_bytes, MemorySnapshot
-from vllm.platforms import current_platform
-from vllm.worker.worker import _check_if_gpu_supports_dtype
-from vllm.v1.worker.gpu_worker import init_worker_distributed_environment
-from vllm.model_executor import set_random_seed
-from vllm.v1.utils import report_usage_stats
-
-from vllm_omni.worker.gpu_diffusion_model_runner import GPUDiffusionModelRunner
-
-
-class GPUDiffusionWorker(GPUWorker):
-    def init_device(self):
-        if self.device_config.device.type == "cuda":
-            # torch.distributed.all_reduce does not free the input tensor until
-            # the synchronization point. This causes the memory usage to grow
-            # as the number of all_reduce calls increases. This env var disables
-            # this behavior.
-            # Related issue:
-            # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573
-            os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1"
-
-            # This env var set by Ray causes exceptions with graph building.
-            os.environ.pop("NCCL_ASYNC_ERROR_HANDLING", None)
-            self.device = torch.device(f"cuda:{self.local_rank}")
-            current_platform.set_device(self.device)
-
-            _check_if_gpu_supports_dtype(self.model_config.dtype)
-            gc.collect()
-            torch.cuda.empty_cache()
-
-            # take current memory snapshot
-            self.init_snapshot = MemorySnapshot()
-            self.requested_memory = (self.init_snapshot.total_memory *
-                                     self.cache_config.gpu_memory_utilization)
-            if self.init_snapshot.free_memory < self.requested_memory:
-                GiB = lambda b: round(b / GiB_bytes, 2)
-                raise ValueError(
-                    f"Free memory on device "
-                    f"({GiB(self.init_snapshot.free_memory)}/"
-                    f"{GiB(self.init_snapshot.total_memory)} GiB) on startup "
-                    f"is less than desired GPU memory utilization "
-                    f"({self.cache_config.gpu_memory_utilization}, "
-                    f"{GiB(self.requested_memory)} GiB). Decrease GPU memory "
-                    f"utilization or reduce GPU memory used by other processes."
-                )
-        else:
-            raise RuntimeError(
-                f"Not support device type: {self.device_config.device}")
-        # Initialize the distributed environment.
-        init_worker_distributed_environment(self.vllm_config, self.rank,
-                                            self.distributed_init_method,
-                                            self.local_rank,
-                                            current_platform.dist_backend)
-        # Set random seed.
-        set_random_seed(self.model_config.seed)
-
-        # Construct the model runner
-        self.model_runner: GPUDiffusionModelRunner = GPUDiffusionModelRunner(
-            self.vllm_config, self.device)
-
-        if self.rank == 0:
-            # If usage stat is enabled, collect relevant info.
-            report_usage_stats(self.vllm_config)

vllm_omni/worker/gpu_model_runner.py

@@ -1,755 +0,0 @@
-
-from typing import TYPE_CHECKING, Any, Optional, Union, cast
-
-import numpy as np
-import torch
-
-import vllm.envs as envs
-from vllm.distributed.parallel_state import get_pp_group, get_tp_group
-from vllm.forward_context import set_forward_context
-from vllm.logger import init_logger
-from vllm.model_executor.models.interfaces_base import VllmModelForPooling
-from vllm.multimodal.inputs import MultiModalKwargs
-from vllm.sequence import IntermediateTensors
-from vllm.utils import LazyLoader
-from vllm.v1.attention.backends.utils import CommonAttentionMetadata
-from vllm.v1.outputs import EMPTY_MODEL_RUNNER_OUTPUT
-from vllm.v1.worker.gpu_input_batch import CachedRequestState
-from vllm.sampling_params import SamplingType
-from vllm.distributed.kv_transfer import has_kv_transfer_group
-from vllm.utils import round_up
-from vllm.v1.spec_decode.eagle import EagleProposer
-
-from vllm.v1.worker.gpu_model_runner import GPUModelRunner
-
-from vllm_omni.outputs import OmniModelRunnerOutput
-from vllm_omni.model_executor.layers.mrope import MRotaryEmbedding
-
-if TYPE_CHECKING:
-    from vllm.v1.core.sched.output import SchedulerOutput
-else:
-    xgr = LazyLoader("xgr", globals(), "xgrammar")
-    xgr_torch_compile = LazyLoader(
-        "xgr_torch_compile", globals(),
-        "xgrammar.kernels.apply_token_bitmask_inplace_torch_compile")
-
-logger = init_logger(__name__)
-
-
-class OmniGPUModelRunner(GPUModelRunner):
-    def _update_states(self, scheduler_output: "SchedulerOutput") -> None:
-        """Update the cached states and the persistent batch with the scheduler
-        output.
-
-        The updated states are used by the `_prepare_inputs` function to create
-        the input GPU tensors for the model.
-
-        The SamplingMetadata is updated and copied to the GPU if there is a
-        new/resumed/paused/finished request in the batch.
-        """
-        # Remove finished requests from the cached states.
-        for req_id in scheduler_output.finished_req_ids:
-            self.requests.pop(req_id, None)
-            self.encoder_cache.pop(req_id, None)
-        # Remove the finished requests from the persistent batch.
-        # NOTE(woosuk): There could be an edge case where finished_req_ids and
-        # scheduled_req_ids overlap. This happens when a request is aborted and
-        # then resubmitted with the same ID. In this case, we treat them as two
-        # distinct requests - clearing the cached states for the first request
-        # and handling the second as a new request.
-        for req_id in scheduler_output.finished_req_ids:
-            self.input_batch.remove_request(req_id)
-
-        # Free the cached encoder outputs.
-        for req_id, input_id in scheduler_output.free_encoder_input_ids:
-            encoder_outputs = self.encoder_cache.get(req_id)
-            if encoder_outputs is not None:
-                encoder_outputs.pop(input_id, None)
-                if not encoder_outputs:
-                    self.encoder_cache.pop(req_id, None)
-
-        # Remove the unscheduled requests from the persistent batch.
-        # NOTE(woosuk): The unscheduled requests are either preempted requests
-        # or running requests that are not scheduled in this step. We remove
-        # them from the persistent batch but keep their cached states since
-        # they will be scheduled again sometime in the future.
-        scheduled_req_ids = scheduler_output.num_scheduled_tokens.keys()
-        cached_req_ids = self.input_batch.req_id_to_index.keys()
-        unscheduled_req_ids = cached_req_ids - scheduled_req_ids
-        # NOTE(woosuk): The persistent batch optimization assumes that
-        # consecutive batches contain mostly the same requests. If batches
-        # have low request overlap (e.g., alternating between two distinct
-        # sets of requests), this optimization becomes very inefficient.
-        for req_id in unscheduled_req_ids:
-            self.input_batch.remove_request(req_id)
-
-        req_ids_to_add: list[str] = []
-        # Add new requests to the cached states.
-        for new_req_data in scheduler_output.scheduled_new_reqs:
-            req_id = new_req_data.req_id
-            sampling_params = new_req_data.sampling_params
-            pooling_params = new_req_data.pooling_params
-
-            if sampling_params and \
-                sampling_params.sampling_type == SamplingType.RANDOM_SEED:
-                generator = torch.Generator(device=self.device)
-                generator.manual_seed(sampling_params.seed)
-            else:
-                generator = None
-
-            if pooling_params:
-                assert (task := pooling_params.task) is not None, (
-                    "You did not set `task` in the API")
-
-                model = cast(VllmModelForPooling, self.model)
-                to_update = model.pooler.get_pooling_updates(task)
-                to_update.apply(pooling_params)
-
-            self.requests[req_id] = CachedRequestState(
-                req_id=req_id,
-                prompt_token_ids=new_req_data.prompt_token_ids,
-                mm_inputs=new_req_data.mm_inputs,
-                mm_positions=new_req_data.mm_positions,
-                sampling_params=sampling_params,
-                pooling_params=pooling_params,
-                generator=generator,
-                block_ids=new_req_data.block_ids,
-                num_computed_tokens=new_req_data.num_computed_tokens,
-                output_token_ids=[],
-                lora_request=new_req_data.lora_request,
-            )
-
-            # If prompt embeddings are provided, decode and attach to inter_data
-            try:
-                if getattr(new_req_data, "prompt_embeds", None) is not None:
-                    payload = new_req_data.prompt_embeds
-                    dtype = getattr(np, payload.dtype)
-                    arr = np.frombuffer(payload.data, dtype=dtype)
-                    arr = arr.reshape(payload.shape)
-                    pe_cpu = torch.from_numpy(arr)
-                    # Store temporarily on CPU; later moved to device in builder
-                    setattr(self.requests[req_id], "prompt_embeds_cpu", pe_cpu)
-                    # Also replace payload with Tensor for user visibility in scheduler_output
-                    try:
-                        new_req_data.prompt_embeds = pe_cpu  # type: ignore[assignment]
-                    except Exception:
-                        pass
-            except Exception:
-                pass
-            # Decode additional_information payloads (dictionary)
-            try:
-                if getattr(new_req_data, "additional_information", None) is not None:
-                    payload_info = new_req_data.additional_information
-                    info_dict = {}
-                    if isinstance(payload_info, dict):
-                        info_dict = payload_info
-                    else:
-                        from vllm.v1.engine import AdditionalInformationPayload
-                        if isinstance(payload_info, AdditionalInformationPayload):
-                            for k, entry in payload_info.entries.items():
-                                if entry.tensor_data is not None:
-                                    dt = np.dtype(getattr(entry, "tensor_dtype", "float32"))
-                                    arr = np.frombuffer(entry.tensor_data, dtype=dt)
-                                    arr = arr.reshape(entry.tensor_shape)
-                                    info_dict[k] = torch.from_numpy(arr)
-                                else:
-                                    info_dict[k] = entry.list_data
-                    if info_dict:
-                        setattr(self.requests[req_id], "additional_information_cpu", info_dict)
-            except Exception:
-                pass
-
-            # Only relevant for models using M-RoPE (e.g, Qwen2-VL)
-            if self.uses_mrope:
-                image_grid_thw = []
-                video_grid_thw = []
-                second_per_grid_ts = []
-                audio_feature_lengths = []
-                use_audio_in_video = False
-                for mm_input in self.requests[req_id].mm_inputs:
-                    if mm_input.get("image_grid_thw") is not None:
-                        image_grid_thw.extend(
-                            mm_input["image_grid_thw"].tolist())
-                    if mm_input.get("video_grid_thw") is not None:
-                        video_grid_thw.extend(
-                            mm_input["video_grid_thw"].tolist())
-                    if mm_input.get("second_per_grid_ts") is not None:
-                        second_per_grid_ts.extend(
-                            mm_input["second_per_grid_ts"])
-                    if mm_input.get("audio_feature_lengths") is not None:
-                        audio_feature_lengths.extend(
-                            mm_input["audio_feature_lengths"])
-                    if mm_input.get("use_audio_in_video") is True:
-                        use_audio_in_video = True
-
-                hf_config = self.model_config.hf_config
-
-                self.requests[req_id].mrope_positions, \
-                    self.requests[req_id].mrope_position_delta = \
-                    MRotaryEmbedding.get_input_positions_tensor(
-                        self.requests[req_id].prompt_token_ids,
-                        hf_config=hf_config,
-                        image_grid_thw=image_grid_thw,
-                        video_grid_thw=video_grid_thw,
-                        second_per_grid_ts=second_per_grid_ts,
-                        audio_feature_lengths=audio_feature_lengths,
-                        use_audio_in_video=use_audio_in_video,
-                    )
-
-            req_ids_to_add.append(req_id)
-
-        # Update the states of the running/resumed requests.
-        is_last_rank = get_pp_group().is_last_rank
-        req_data = scheduler_output.scheduled_cached_reqs
-        for i, req_id in enumerate(req_data.req_ids):
-            req_state = self.requests[req_id]
-            num_computed_tokens = req_data.num_computed_tokens[i]
-            new_block_ids = req_data.new_block_ids[i]
-            resumed_from_preemption = req_data.resumed_from_preemption[i]
-
-            # Update the cached states.
-            req_state.num_computed_tokens = num_computed_tokens
-
-            if not is_last_rank:
-                # When using PP, the scheduler sends the sampled tokens back,
-                # because there's no direct communication between the first-
-                # stage worker and the last-stage worker.
-                new_token_ids = req_data.new_token_ids[i]
-                # Add the sampled token(s) from the previous step (if any).
-                # This doesn't include "unverified" tokens like spec tokens.
-                num_new_tokens = (num_computed_tokens + len(new_token_ids) -
-                                  req_state.num_tokens)
-                if num_new_tokens == 1:
-                    # Avoid slicing list in most common case.
-                    req_state.output_token_ids.append(new_token_ids[-1])
-                elif num_new_tokens > 0:
-                    req_state.output_token_ids.extend(
-                        new_token_ids[-num_new_tokens:])
-
-            # Update the block IDs.
-            if not resumed_from_preemption:
-                # Append the new blocks to the existing block IDs.
-                for block_ids, new_ids in zip(req_state.block_ids,
-                                              new_block_ids):
-                    block_ids.extend(new_ids)
-            else:
-                # The request is resumed from preemption.
-                # Replace the existing block IDs with the new ones.
-                req_state.block_ids = new_block_ids
-
-            req_index = self.input_batch.req_id_to_index.get(req_id)
-            if req_index is None:
-                # The request is not in the persistent batch.
-                # The request was either preempted and resumed later, or was not
-                # scheduled in the previous step and needs to be added again.
-                req_ids_to_add.append(req_id)
-                continue
-
-            # Update the persistent batch.
-            self.input_batch.num_computed_tokens_cpu[req_index] = (
-                num_computed_tokens)
-            self.input_batch.block_table.append_row(new_block_ids, req_index)
-
-            # For the last rank, we don't need to update the token_ids_cpu
-            # because the sampled tokens are already cached.
-            if not is_last_rank:
-                # Add new_token_ids to token_ids_cpu.
-                start_token_index = num_computed_tokens
-                end_token_index = num_computed_tokens + len(new_token_ids)
-                self.input_batch.token_ids_cpu[
-                    req_index,
-                    start_token_index:end_token_index] = new_token_ids
-                self.input_batch.num_tokens_no_spec[
-                    req_index] = end_token_index
-                self.input_batch.num_tokens[req_index] = end_token_index
-
-            # Add spec_token_ids to token_ids_cpu.
-            spec_token_ids = (
-                scheduler_output.scheduled_spec_decode_tokens.get(req_id, ()))
-            if spec_token_ids:
-                num_spec_tokens = len(spec_token_ids)
-                start_index = self.input_batch.num_tokens_no_spec[req_index]
-                end_token_index = start_index + num_spec_tokens
-                self.input_batch.token_ids_cpu[
-                    req_index, start_index:end_token_index] = spec_token_ids
-                # NOTE(woosuk): `num_tokens` here may include spec tokens.
-                self.input_batch.num_tokens[req_index] += num_spec_tokens
-
-        # Add the new or resumed requests to the persistent batch.
-        # The smaller empty indices are filled first.
-        for req_id in req_ids_to_add:
-            req_state = self.requests[req_id]
-            self.input_batch.add_request(req_state)
-
-        # Condense the batched states if there are gaps left by removed requests
-        self.input_batch.condense()
-        # Allow attention backend to reorder the batch, potentially
-        self._may_reorder_batch(scheduler_output)
-        # Refresh batch metadata with any pending updates.
-        self.input_batch.refresh_metadata()
-    
-    @torch.inference_mode()
-    def execute_model(
-        self,
-        scheduler_output: "SchedulerOutput",
-        intermediate_tensors: Optional[IntermediateTensors] = None,
-    ) -> Union[OmniModelRunnerOutput, IntermediateTensors]:
-        self._update_states(scheduler_output)
-        if not scheduler_output.total_num_scheduled_tokens:
-            if not has_kv_transfer_group():
-                # Return empty ModelRunnerOutput if there's no work to do.
-                return EMPTY_MODEL_RUNNER_OUTPUT
-
-            return self.kv_connector_no_forward(scheduler_output,
-                                                self.vllm_config)
-
-        # Prepare the decoder inputs.
-        (attn_metadata, attention_cuda_graphs, logits_indices,
-         spec_decode_metadata, num_scheduled_tokens_np,
-         spec_decode_common_attn_metadata) = (
-             self._prepare_inputs(scheduler_output))
-
-        num_scheduled_tokens = scheduler_output.total_num_scheduled_tokens
-        if (self.use_cuda_graph
-                and num_scheduled_tokens <= self.cudagraph_batch_sizes[-1]):
-            # Use piecewise CUDA graphs.
-            # Add padding to the batch size.
-            num_input_tokens = self.vllm_config.pad_for_cudagraph(
-                num_scheduled_tokens)
-        else:
-            # Eager mode.
-            # Pad tokens to multiple of tensor_parallel_size when
-            # enabled collective fusion for SP
-            tp_size = self.vllm_config.parallel_config.tensor_parallel_size
-            if self.compilation_config.pass_config. \
-                enable_sequence_parallelism and tp_size > 1:
-                num_input_tokens = round_up(num_scheduled_tokens, tp_size)
-            else:
-                num_input_tokens = num_scheduled_tokens
-
-        # Padding for DP
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_input_tokens)
-        num_input_tokens += num_pad
-
-        # _prepare_inputs may reorder the batch, so we must gather multi
-        # modal outputs after that to ensure the correct order
-        if self.is_multimodal_model:
-            # Run the multimodal encoder if any.
-            self._execute_mm_encoder(scheduler_output)
-            mm_embeds = self._gather_mm_embeddings(scheduler_output)
-        else:
-            mm_embeds = []
-
-        if get_pp_group().is_first_rank:
-            # Always prepare inputs_embeds on the first PP rank so both
-            # multimodal and text-only models can consume embeddings.
-            inputs_embeds_scheduled = self.model.get_input_embeddings(
-                input_ids=self.input_ids[:num_scheduled_tokens],
-                multimodal_embeddings=(mm_embeds or None)
-                if self.is_multimodal_model else None,
-            )
-
-            # TODO(woosuk): Avoid the copy. Optimize.
-            self.inputs_embeds[:num_scheduled_tokens].copy_(
-                inputs_embeds_scheduled)
-
-            # Overlay per-request custom prompt_embeds for the prompt portion
-            for req_index, req_id in enumerate(self.input_batch.req_ids):
-                req_state = self.requests[req_id]
-                pe_cpu = getattr(req_state, "prompt_embeds_cpu", None)
-                if pe_cpu is None:
-                    continue
-                num_computed_tokens = int(
-                    self.input_batch.num_computed_tokens_cpu[req_index])
-                prompt_len = len(req_state.prompt_token_ids)
-                prompt_remaining = max(0, prompt_len - num_computed_tokens)
-                sched_tokens = int(num_scheduled_tokens_np[req_index])
-                overlay_len = min(sched_tokens, prompt_remaining)
-                if overlay_len <= 0:
-                    continue
-                src = pe_cpu[num_computed_tokens:
-                             num_computed_tokens + overlay_len].to(
-                                 dtype=self.dtype,
-                                 device=self.device,
-                                 non_blocking=True)
-                start_offset = int(self.query_start_loc_cpu[req_index])
-                self.inputs_embeds[start_offset:start_offset + overlay_len]\
-                    .copy_(src)
-
-            input_ids = self.input_ids[:num_input_tokens]  # preserved for APIs
-            inputs_embeds = self.inputs_embeds[:num_input_tokens]
-            model_mm_kwargs = (self._extract_mm_kwargs(scheduler_output)
-                               if self.is_multimodal_model else {})
-        else:
-            # For non-first PP ranks, use token ids as usual; embeddings are
-            # only consumed on the first rank.
-            input_ids = self.input_ids[:num_input_tokens]
-            inputs_embeds = None
-            model_mm_kwargs = {}
-        if self.uses_mrope:
-            positions = self.mrope_positions[:, :num_input_tokens]
-        else:
-            positions = self.positions[:num_input_tokens]
-
-        if get_pp_group().is_first_rank:
-            intermediate_tensors = None
-        else:
-            intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                num_input_tokens, intermediate_tensors, True)
-
-        # Some attention backends only support CUDA Graphs in pure decode.
-        # If attention doesn't support CUDA Graphs for this batch, but we
-        # compiled with full CUDA graphs, we have to skip them entirely.
-        skip_cuda_graphs = self.full_cuda_graph and not attention_cuda_graphs
-
-        # Run the model.
-        # Use persistent buffers for CUDA graphs.
-        with set_forward_context(
-                attn_metadata,
-                self.vllm_config,
-                num_tokens=num_input_tokens,
-                num_tokens_across_dp=num_tokens_across_dp,
-                skip_cuda_graphs=skip_cuda_graphs,
-        ), self.maybe_get_kv_connector_output(
-                scheduler_output) as kv_connector_output:
-
-            model_output = self.model(
-                input_ids=input_ids,
-                positions=positions,
-                intermediate_tensors=intermediate_tensors,
-                inputs_embeds=inputs_embeds,
-                **MultiModalKwargs.as_kwargs(
-                    model_mm_kwargs,
-                    device=self.device,
-                ),
-                sampling_metadata=self.input_batch.sampling_metadata,
-                logits_index=logits_indices,
-                sampler=self.sampler,
-            )
-
-        if self.use_aux_hidden_state_outputs:
-            hidden_states, aux_hidden_states = model_output
-        else:
-            hidden_states = model_output
-            aux_hidden_states = None
-
-        text_hidden_states, multimodal_outputs = self.extract_multimodal_outputs(hidden_states)
-
-        # Broadcast PP output for external_launcher (torchrun)
-        # to make sure we are synced across pp ranks
-        # TODO: Support overlapping mirco-batches
-        # https://github.com/vllm-project/vllm/issues/18019
-        broadcast_pp_output = \
-            self.parallel_config.distributed_executor_backend \
-            == "external_launcher" and len(get_pp_group().ranks) > 0
-        if not get_pp_group().is_last_rank:
-            # For mid-pipeline stages, return the hidden states.
-            assert isinstance(text_hidden_states, IntermediateTensors)
-            if not broadcast_pp_output:
-                text_hidden_states.kv_connector_output = kv_connector_output
-                return text_hidden_states
-            get_pp_group().send_tensor_dict(text_hidden_states.tensors,
-                                            all_gather_group=get_tp_group())
-            logits = None
-        else:
-            if self.input_batch.pooling_params:
-                return self._pool(text_hidden_states, num_scheduled_tokens,
-                                  num_scheduled_tokens_np, kv_connector_output)
-
-            sample_hidden_states = text_hidden_states[logits_indices]
-            logits = self.model.compute_logits(sample_hidden_states, None)
-        if broadcast_pp_output:
-            model_output_broadcast_data = {
-                "logits": logits.contiguous(),
-            } if logits is not None else {}
-            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
-                model_output_broadcast_data, src=len(get_pp_group().ranks) - 1)
-            assert model_output_broadcast_data is not None
-            logits = model_output_broadcast_data["logits"]
-
-        # Apply structured output bitmasks if present
-        if scheduler_output.grammar_bitmask is not None:
-            self.apply_grammar_bitmask(scheduler_output, logits)
-
-        # Sample the next token and get logprobs if needed.
-        sampling_metadata = self.input_batch.sampling_metadata
-        if spec_decode_metadata is None:
-            sampler_output = self.sampler(
-                logits=logits,
-                sampling_metadata=sampling_metadata,
-            )
-        else:
-            # When indexing with a tensor (bonus_logits_indices), PyTorch
-            # creates a new tensor with separate storage from the original
-            # logits tensor. This means any in-place operations on bonus_logits
-            # won't affect the original logits tensor.
-            assert logits is not None
-            bonus_logits = logits[spec_decode_metadata.bonus_logits_indices]
-            sampler_output = self.sampler(
-                logits=bonus_logits,
-                sampling_metadata=sampling_metadata,
-            )
-            bonus_token_ids = sampler_output.sampled_token_ids
-
-            # Just like `bonus_logits`, `target_logits` is a new tensor with
-            # separate storage from the original `logits` tensor. Therefore,
-            # it is safe to update `target_logits` in place.
-            target_logits = logits[spec_decode_metadata.target_logits_indices]
-            output_token_ids = self.rejection_sampler(
-                spec_decode_metadata,
-                None,  # draft_probs
-                target_logits,
-                bonus_token_ids,
-                sampling_metadata,
-            )
-            sampler_output.sampled_token_ids = output_token_ids
-
-        num_nans_in_logits = {}
-        if envs.VLLM_COMPUTE_NANS_IN_LOGITS:
-            num_nans_in_logits = self._get_nans_in_logits(logits)
-
-        # TODO(woosuk): The following loop can be slow since it iterates over
-        # the requests one by one. Optimize.
-        discard_sampled_tokens_req_indices = []
-        for i, req_id in enumerate(self.input_batch.req_ids):
-            req_state = self.requests[req_id]
-            seq_len = (req_state.num_computed_tokens +
-                       scheduler_output.num_scheduled_tokens[req_id])
-            if seq_len < req_state.num_tokens:
-                # Ignore the sampled token for partial prefills.
-                # Rewind the generator state as if the token was not sampled.
-                # This relies on cuda-specific torch-internal impl details
-                generator = self.input_batch.generators.get(i)
-                if generator is not None:
-                    generator.set_offset(generator.get_offset() - 4)
-                # Record the index of the request that should not be sampled,
-                # so that we could clear the sampled tokens before returning.
-                discard_sampled_tokens_req_indices.append(i)
-
-        # NOTE: GPU -> CPU Sync happens here.
-        # Move as many CPU operations as possible before this sync point.
-        logprobs_tensors = sampler_output.logprobs_tensors
-        logprobs_lists = logprobs_tensors.tolists() \
-            if logprobs_tensors is not None else None
-
-        # Compute prompt logprobs if needed.
-        prompt_logprobs_dict = self._get_prompt_logprobs_dict(
-            text_hidden_states[:num_scheduled_tokens],
-            scheduler_output,
-        )
-
-        # Get the valid generated tokens.
-        import os
-        sampled_token_ids = sampler_output.sampled_token_ids
-        max_gen_len = sampled_token_ids.shape[-1]
-        if max_gen_len == 1:
-            # No spec decode tokens.
-            valid_sampled_token_ids = sampled_token_ids.tolist()
-        else:
-            # Includes spec decode tokens.
-            valid_sampled_token_ids = self.rejection_sampler.parse_output(
-                sampled_token_ids,
-                self.input_batch.vocab_size,
-            )
-        # Mask out the sampled tokens that should not be sampled.
-        for i in discard_sampled_tokens_req_indices:
-            valid_sampled_token_ids[i].clear()
-
-        # Cache the sampled tokens in the model runner, so that the scheduler
-        # doesn't need to send them back.
-        # NOTE(woosuk): As an exception, when using PP, the scheduler sends
-        # the sampled tokens back, because there's no direct communication
-        # between the first-stage worker and the last-stage worker.
-        for req_idx, sampled_ids in enumerate(valid_sampled_token_ids):
-            if not sampled_ids:
-                continue
-
-            start_idx = self.input_batch.num_tokens_no_spec[req_idx]
-            end_idx = start_idx + len(sampled_ids)
-            assert end_idx <= self.max_model_len, (
-                "Sampled token IDs exceed the max model length. "
-                f"Total number of tokens: {end_idx} > max_model_len: "
-                f"{self.max_model_len}")
-
-            self.input_batch.token_ids_cpu[req_idx,
-                                           start_idx:end_idx] = sampled_ids
-            self.input_batch.num_tokens_no_spec[req_idx] = end_idx
-            self.input_batch.num_tokens[req_idx] = end_idx
-            req_id = self.input_batch.req_ids[req_idx]
-            req_state = self.requests[req_id]
-            req_state.output_token_ids.extend(sampled_ids)
-
-        if not self.speculative_config:
-            # Speculative decoding is not enabled.
-            spec_token_ids = None
-        else:
-            assert spec_decode_common_attn_metadata is not None
-            spec_token_ids = self.propose_draft_token_ids(
-                scheduler_output,
-                valid_sampled_token_ids,
-                sampling_metadata,
-                text_hidden_states,
-                sample_hidden_states,
-                aux_hidden_states,
-                spec_decode_metadata,
-                spec_decode_common_attn_metadata,
-            )
-
-        self.eplb_step()
-
-        return OmniModelRunnerOutput(
-            req_ids=self.input_batch.req_ids,
-            req_id_to_index=self.input_batch.req_id_to_index,
-            sampled_token_ids=valid_sampled_token_ids,
-            spec_token_ids=spec_token_ids,
-            logprobs=logprobs_lists,
-            prompt_logprobs_dict=prompt_logprobs_dict,
-            pooler_output=[],
-            kv_connector_output=kv_connector_output,
-            num_nans_in_logits=num_nans_in_logits,
-            multimodal_outputs=multimodal_outputs,
-        )
-    
-    @torch.inference_mode()
-    def extract_multimodal_outputs(self, hidden_states: torch.Tensor) -> tuple[torch.Tensor, dict]:
-        if hasattr(self.model, "have_multimodal_outputs") and self.model.have_multimodal_outputs:
-            text_hidden_states = hidden_states.text_hidden_states
-            multimodal_outputs = hidden_states.multimodal_outputs
-
-        else:
-            text_hidden_states = hidden_states
-            multimodal_outputs = {}
-        return text_hidden_states, multimodal_outputs
-    
-    @torch.inference_mode()
-    def _dummy_run(
-        self,
-        num_tokens: int,
-        capture_attn_cudagraph: bool = False,
-        skip_eplb: bool = False,
-        is_profile: bool = False,
-    ) -> tuple[torch.Tensor, torch.Tensor]:
-
-        # Padding for DP
-        num_pad, num_tokens_across_dp = self.get_dp_padding(num_tokens)
-        num_tokens += num_pad
-
-        # Set num_scheduled_tokens based on num_tokens and max_num_seqs
-        # for dummy run with LoRA so that the num_reqs collectively
-        # has num_tokens in total.
-        assert num_tokens <= self.scheduler_config.max_num_batched_tokens
-        max_num_reqs = self.scheduler_config.max_num_seqs
-        num_reqs = min(num_tokens, max_num_reqs)
-        min_tokens_per_req = num_tokens // num_reqs
-        num_scheduled_tokens_list = [min_tokens_per_req] * num_reqs
-        num_scheduled_tokens_list[-1] += num_tokens % num_reqs
-        assert sum(num_scheduled_tokens_list) == num_tokens
-        assert len(num_scheduled_tokens_list) == num_reqs
-        num_scheduled_tokens = np.array(num_scheduled_tokens_list,
-                                        dtype=np.int32)
-
-        attn_metadata: Optional[dict[str, Any]] = None
-        if capture_attn_cudagraph:
-            attn_metadata = {}
-
-            # Make sure max_model_len is used at the graph capture time.
-            self.seq_lens_np[:num_reqs] = self.max_model_len
-            self.seq_lens_np[num_reqs:] = 0
-            self.seq_lens[:num_reqs].copy_(self.seq_lens_cpu[:num_reqs],
-                                           non_blocking=True)
-
-            for kv_cache_group_id, kv_cache_group_spec in enumerate(
-                    self.kv_cache_config.kv_cache_groups):
-                common_attn_metadata = CommonAttentionMetadata(
-                    query_start_loc=self.query_start_loc[:num_reqs + 1],
-                    query_start_loc_cpu=self.query_start_loc_cpu[:num_reqs +
-                                                                 1],
-                    seq_lens=self.seq_lens[:num_reqs],
-                    seq_lens_cpu=self.seq_lens_cpu[:num_reqs],
-                    num_computed_tokens_cpu=self.input_batch.
-                    num_computed_tokens_cpu_tensor[:num_reqs],
-                    num_reqs=num_reqs,
-                    num_actual_tokens=num_tokens,
-                    max_query_len=num_tokens,
-                    block_table_tensor=self.input_batch.block_table[
-                        kv_cache_group_id].get_device_tensor()[:num_reqs],
-                    slot_mapping=self.input_batch.
-                    block_table[kv_cache_group_id].slot_mapping[:num_tokens],
-                    causal=True)
-
-                for attn_group in self.attn_groups[kv_cache_group_id]:
-                    attn_metadata_i = attn_group.metadata_builder\
-                        .build_for_cudagraph_capture(common_attn_metadata)
-                    for layer_name in kv_cache_group_spec.layer_names:
-                        attn_metadata[layer_name] = attn_metadata_i
-
-        with self.maybe_dummy_run_with_lora(self.lora_config,
-                                            num_scheduled_tokens):
-            if self.is_multimodal_model:
-                input_ids = None
-                inputs_embeds = self.inputs_embeds[:num_tokens]
-                model_mm_kwargs = self._dummy_mm_kwargs(num_reqs)
-            else:
-                input_ids = self.input_ids[:num_tokens]
-                inputs_embeds = None
-                model_mm_kwargs = {}
-
-            if self.uses_mrope:
-                positions = self.mrope_positions[:, :num_tokens]
-            else:
-                positions = self.positions[:num_tokens]
-
-            if get_pp_group().is_first_rank:
-                intermediate_tensors = None
-            else:
-                if self.intermediate_tensors is None:
-                    self.intermediate_tensors = (
-                        self.model.make_empty_intermediate_tensors(
-                            batch_size=self.max_num_tokens,
-                            dtype=self.model_config.dtype,
-                            device=self.device))
-
-                intermediate_tensors = self.sync_and_slice_intermediate_tensors(
-                    num_tokens, None, False)
-
-            with self.maybe_randomize_inputs(input_ids), set_forward_context(
-                    attn_metadata,
-                    self.vllm_config,
-                    num_tokens=num_tokens,
-                    num_tokens_across_dp=num_tokens_across_dp):
-                outputs = self.model(
-                    input_ids=input_ids,
-                    positions=positions,
-                    intermediate_tensors=intermediate_tensors,
-                    inputs_embeds=inputs_embeds,
-                    **MultiModalKwargs.as_kwargs(
-                        model_mm_kwargs,
-                        device=self.device,
-                    ),
-                    sampler = None
-                )
-
-            if self.use_aux_hidden_state_outputs:
-                hidden_states, _ = outputs
-            else:
-                hidden_states = outputs
-
-            text_hidden_states, multimodal_outputs = self.extract_multimodal_outputs(hidden_states)
-
-
-            if self.speculative_config and self.speculative_config.use_eagle():
-                assert isinstance(self.drafter, EagleProposer)
-                self.drafter.dummy_run(num_tokens)
-
-        # This is necessary to avoid blocking DP.
-        # For dummy runs, we typically skip EPLB since we don't have any real
-        # requests to process.
-        # However, in DP settings, there may be cases when some DP ranks do
-        # not have any requests to process, so they're executing dummy batches.
-        # In such cases, we still have to trigger EPLB to make sure
-        # ranks execute the rearrangement in synchronization.
-        if not skip_eplb:
-            self.eplb_step(is_dummy=True, is_profile=is_profile)
-
-        logit_indices = np.cumsum(num_scheduled_tokens) - 1
-        return text_hidden_states, text_hidden_states[logit_indices]

vllm_omni/worker/model_runner.py

@@ -1,199 +0,0 @@
-from dataclasses import dataclass
-from typing import Optional, Dict, Any, List
-import itertools
-
-import torch
-
-from vllm.worker.model_runner import ModelInputForGPU, ModelInputForGPUBuilder
-
-from vllm.utils import flatten_2d_lists, async_tensor_h2d
-from vllm.lora.layers import LoRAMapping
-from vllm.sequence import MultiModalKwargs
-
-
-@dataclass(frozen=True)
-class OmniModelInputForGPU(ModelInputForGPU):
-    # New: optional additional information dict for current scheduled tokens
-    additional_information: Optional[Dict[str, Any]] = None
-
-
-class OmniModelInputForGPUBuilder(ModelInputForGPUBuilder):
-    def build(self) -> OmniModelInputForGPU:
-        """Finalize the builder intermediate data and
-        create on-device tensors.
-        """
-        # Combine and flatten intermediate data.
-        input_tokens = list[int]()
-        inputs_embeds_list = list[torch.Tensor]()
-        token_types = list[int]()
-        for inter_data in self.inter_data_list:
-            for cur_input_tokens in inter_data.input_tokens:
-                input_tokens.extend(cur_input_tokens)
-            for cur_token_types in inter_data.token_types:
-                token_types.extend(cur_token_types)
-            if inter_data.inputs_embeds is not None:
-                inputs_embeds_list.append(
-                    inter_data.inputs_embeds.to(
-                        dtype=self.runner.model_config.dtype,
-                        device=self.runner.device))
-            # Support v1 direct-transfer prompt embeds attached on request state
-            if inter_data.inputs_embeds is None:
-                try:
-                    # Locate req_id -> request state to fetch decoded CPU embeds
-                    req_id = inter_data.request_id  # type: ignore[attr-defined]
-                    req_state = getattr(self.runner, "requests", {}).get(req_id)
-                    pe_cpu = getattr(req_state, "prompt_embeds_cpu", None)
-                    if pe_cpu is not None:
-                        inter_data.inputs_embeds = pe_cpu.to(
-                            dtype=self.runner.model_config.dtype,
-                            device=self.runner.device)
-                        inputs_embeds_list.append(inter_data.inputs_embeds)
-                except Exception:
-                    pass
-        inputs_embeds: Optional[torch.Tensor]
-        if len(inputs_embeds_list) == 0:
-            inputs_embeds = None
-        else:
-            inputs_embeds = torch.cat(inputs_embeds_list, dim=0).to(
-                dtype=self.runner.model_config.dtype,
-                device=self.runner.device)
-            assert len(inputs_embeds) == len(input_tokens)
-
-        if not input_tokens and inputs_embeds is None:
-            # This may happen when all prefill requests hit
-            # prefix caching and there is no decode request.
-            return self.model_input_cls()
-
-        mrope_input_positions: Optional[List[List[int]]] = None
-        if any(inter_data.mrope_input_positions is not None
-               for inter_data in self.inter_data_list):
-            mrope_input_positions = [[] for _ in range(3)]
-            for idx in range(3):
-                for inter_data in self.inter_data_list:
-                    msections = inter_data.mrope_input_positions
-                    if msections is None:
-                        for _seq_input_positions in inter_data.input_positions:
-                            mrope_input_positions[idx].extend(
-                                _seq_input_positions)
-                    else:
-                        for _seq_mrope_input_positions in msections:
-                            mrope_input_positions[idx].extend(
-                                _seq_mrope_input_positions[idx])
-            input_positions = None
-        else:
-            input_positions = []
-            for inter_data in self.inter_data_list:
-                for cur_input_positions in inter_data.input_positions:
-                    input_positions.extend(cur_input_positions)
-
-        seq_lens = []
-        query_lens = []
-        max_decode_seq_len = 0
-        max_encoder_seq_len = 0
-        for inter_data in self.inter_data_list:
-            seq_lens.extend(inter_data.seq_lens)
-            query_lens.extend(inter_data.query_lens)
-            if not inter_data.is_prompt:
-                max_decode_seq_len = max(max_decode_seq_len,
-                                         max(inter_data.seq_lens))
-                if self.runner.model_config.is_encoder_decoder:
-                    max_encoder_seq_len = max(max_encoder_seq_len,
-                                              inter_data.encoder_seq_len)
-
-        # Mapping from request IDs to sequence IDs. Used for Jamba models
-        # that manages the cache by itself.
-        request_ids_to_seq_ids = {
-            data.request_id: data.seq_ids
-            for data in self.inter_data_list
-        }
-
-        cuda_graph_pad_size = self._get_cuda_graph_pad_size(
-            num_seqs=len(seq_lens),
-            max_decode_seq_len=max_decode_seq_len,
-            max_encoder_seq_len=max_encoder_seq_len)
-
-        batch_size = len(input_tokens)
-        if cuda_graph_pad_size != -1:
-            # If cuda graph can be used, pad tensors accordingly.
-            # See `capture_model` API for more details.
-            # vLLM uses cuda graph only for decoding requests.
-            batch_size += cuda_graph_pad_size
-
-        # Tokens and positions.
-        if cuda_graph_pad_size:
-            input_tokens.extend(itertools.repeat(0, cuda_graph_pad_size))
-        assert self.runner.device is not None
-        input_tokens_tensor = async_tensor_h2d(input_tokens, torch.long,
-                                               self.runner.device,
-                                               self.runner.pin_memory)
-
-        token_types_tensor = async_tensor_h2d(token_types, torch.long,
-                                               self.runner.device,
-                                               self.runner.pin_memory) \
-                                                if token_types else None
-
-        if mrope_input_positions is not None:
-            for idx in range(3):
-                mrope_input_positions[idx].extend(
-                    itertools.repeat(0, cuda_graph_pad_size))
-            input_positions_tensor = async_tensor_h2d(mrope_input_positions,
-                                                      torch.long,
-                                                      self.runner.device,
-                                                      self.runner.pin_memory)
-        else:
-            input_positions.extend(itertools.repeat(0, cuda_graph_pad_size))
-            input_positions_tensor = async_tensor_h2d(input_positions,
-                                                      torch.long,
-                                                      self.runner.device,
-                                                      self.runner.pin_memory)
-        # Sequence and query lengths.
-        if cuda_graph_pad_size:
-            seq_lens.extend(itertools.repeat(1, cuda_graph_pad_size))
-
-        # Attention metadata.
-        attn_metadata = self.attn_metadata_builder.build(
-            seq_lens, query_lens, cuda_graph_pad_size, batch_size)
-
-        # LoRA data.
-        lora_requests = set()
-        lora_mapping = None
-        if self.enable_lora:
-            lora_requests = set(r for data in self.inter_data_list
-                                for r in data.lora_requests)
-            lora_index_mapping = flatten_2d_lists([
-                flatten_2d_lists(inter_data.lora_index_mapping)
-                for inter_data in self.inter_data_list
-            ])
-            if cuda_graph_pad_size:
-                lora_index_mapping.extend(
-                    itertools.repeat(0, cuda_graph_pad_size))
-            lora_prompt_mapping = flatten_2d_lists([
-                flatten_2d_lists(inter_data.lora_prompt_mapping)
-                for inter_data in self.inter_data_list
-            ])
-
-            lora_mapping = LoRAMapping(
-                **dict(index_mapping=lora_index_mapping,
-                       prompt_mapping=lora_prompt_mapping,
-                       is_prefill=not self.decode_only))
-
-        # Multi-modal data.
-        multi_modal_kwargs_list = [
-            data.multi_modal_kwargs for data in self.inter_data_list
-            if data.multi_modal_kwargs is not None
-        ]
-        multi_modal_kwargs = MultiModalKwargs.batch(multi_modal_kwargs_list)
-
-        return self.model_input_cls(
-            input_tokens=input_tokens_tensor,
-            inputs_embeds=inputs_embeds,
-            input_positions=input_positions_tensor,
-            token_types=token_types_tensor,
-            attn_metadata=attn_metadata,
-            seq_lens=seq_lens,
-            query_lens=query_lens,
-            lora_mapping=lora_mapping,
-            lora_requests=lora_requests,
-            multi_modal_kwargs=multi_modal_kwargs,
-            request_ids_to_seq_ids=request_ids_to_seq_ids,
-            finished_requests_ids=self.finished_requests_ids)