Skip to content

Request: Add Backward Pass support for SageAttention3 (Training/Fine-tuning) #317

@xTimeCrystal

Description

@xTimeCrystal

Hello,

I noticed that the current implementation of SageAttention3 lacks support for the backward pass. This is somewhat confusing, as the SageAttention3 paper includes graphs and data explicitly benchmarking the backward pass performance.

Since many users (myself included) intend to use this library for training and fine-tuning models rather than just inference, the lack of backward gradients is a major blocker.

Could you please prioritize adding backward pass support to align the repository with the capabilities presented in the paper?

Thank you!

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions