[Feature]Support async computation and communication across stages by chunks (#727)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Signed-off-by: Rein Yang <ruiruyang2@gmail.com> Co-authored-by: Rein Yang <ruiruyang2@gmail.com> Co-authored-by: CHEN <116010019@link.cuhk.edu.cn> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
29 files changed
+1258
-396
No files matched your search
@@ -207,10 +207,11 @@ class GPUDiffusionModelRunner:
|
||||
start_time = time.time()
|
||||
|
||||
while True:
|
||||
get_key = f"omni_{from_stage}_to_{to_stage}_kv_cache_{req.request_id}"
|
||||
result = self.connector.get(
|
||||
from_stage=from_stage,
|
||||
to_stage=to_stage,
|
||||
request_id=f"kv_cache_{req.request_id}",
|
||||
get_key=get_key,
|
||||
)
|
||||
if result:
|
||||
break
|
||||
|
||||
Reference in new issue
Block a user