EVAX as a feature extractor

[Masaaki-75]

Hi all. Thanks for open-sourcing this great work!

I was trying to use EVA-X-small as a chest x-ray feature extractor. I was not sure which layer outputs the "feature" after i printed the model architecture as below:

EVA_X(
  (patch_embed): PatchEmbed(
    (proj): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))
    (norm): Identity()
  )
  (pos_drop): Dropout(p=0.0, inplace=False)
  (rope): RotaryEmbeddingCat()
  (blocks): ModuleList(
    ...
  )
  (norm): Identity()
  (fc_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
  (head_drop): Dropout(p=0.0, inplace=False)
  (head): Linear(in_features=384, out_features=1000, bias=True)
)

Should i use the output from fc_norm which will give a 384-dim vector, or from head which will give a 1000-dim vector? Or should i treat the final 1000-dim output as the logits?