Latest release fails because of transformer key overlap?

[vrdn-23]

Hey @Ingvarstep,
Thanks for cutting the new release! I think the latest release for flashDeberta is failing due to an issue with how the _attn_implementation key is defined? I think huggingface does an internal check to see if the flash-attn library is installed in order to load the model. I think just renaming the key used by flashDeberta should be enough to resolve the issue, but I'm not a 100% sure

The example in the README fails at this step:

>>> model = FlashDebertaV2Model.from_pretrained("microsoft/deberta-v3-base").to('cuda')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4971, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flashdeberta/model.py", line 572, in __init__
    super().__init__(config)
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2076, in __init__
    self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2686, in _check_and_adjust_attn_implementation
    applicable_attn_implementation = self.get_correct_attn_implementation(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2714, in get_correct_attn_implementation
    self._flash_attn_2_can_dispatch(is_init_check)
  File "/app/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 2422, in _flash_attn_2_can_dispatch
    raise ImportError(f"{preface} the package flash_attn seems to be not installed. {install_message}")
ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2.

[vrdn-23]

I also think the GliNER package does not actually use flashDeberta automatically if it is installed (as mentioned in the docs here) because this check

FlashDeBERTa/src/flashdeberta/model.py

Lines 537 to 540 in b83c803

	if getattr(config, "_attn_implementation", None) and not getattr(
	config, "_attn_implementation_autoset", False
	):
	return config

and the fact that the default value for Gliner config is None

https://github.com/urchade/GLiNER/blob/df118370dd8a18c431cfb02382cfded46168417f/gliner/config.py#L33-L38

means that the flash_attention_2 value is never set for Gliner models? Please correct me if my understanding is wrong here!
cc @Ingvarstep @urchade

[Ingvarstep]

Hi @vrdn-23 , thanks for sharing this issue, I also observed it. It looks like transformers now require a strict name match with predefined attention types. Ideally, we need to have the ability to specify any of our custom types of attention. I will try to find a way to do it. Otherwise, my proposition is to always use the kernels in the case of FlashDeberta and add a custom parameter in GLiNER that controls whether to use a given flash attention.

[vrdn-23]

Thanks for your quick attention on the matter @Ingvarstep ! Please do let me know if you need any help testing out the new version!

[Ingvarstep]

Sure, feel free to test in your projects, any feedback is welcome!

[Ingvarstep]

@vrdn-23 , @urchade, I decided to go with the variant when FlashDeberta uses always kernels, while in GLiNER, we can activate FlashDeberta by specifying an environment variable. What do you think?