Generalization to other architectures besides Llama and Mistral

[immrz]

Dear authors, thanks for your great work. I tried the method on Qwen2.5-7B-Instruct, but the model performance drops drastically. I have tried to tune the L1 regularization weight within [0.05, 0.1, 0.15, 0.2, 0.5, 1], but none of them leads to a satisfying pattern. I wonder whether you have tried other model architectures not mentioned in the paper. Do you have suggestions on how to generalize this method to other models?

[zzk2021]

how much data you used?

[zzk2021]

Do you use other data? (Except for the passkey-retrieval dataset mentioned in the paper.)

[zhuwentao666]

I really want to know how to modify qwen to fit it. Do you have any suggestions or guidance?

[xiazhuo]

@zzk2021, I found that the current data link is invalid. Can you still provide your passkey-retrieval dataset?