Skip to content

Conversation

@qjia7
Copy link
Contributor

@qjia7 qjia7 commented Nov 27, 2025

Needed by updating position id with an onnx model in genai.

Continuous decoding mode: position_ids[i] = i + total_length - new_kv_length

We can use an onnx model which includes a Range op to implement update the position ids:
Inputs: start (total_length - new_kv_length), limit (total_length), delta (1)
Output: position_ids (1D tensor of size new_kv_length)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants