question about the randomly initialized patch embedding layer

[ydhongHIT]

Thanks for your great work. It seems that the shallow layers of the pre-trained backbone are more important because they can be frozen or given a smaller learning rate. I make a preliminary try to replace the 16x16 non-overlapped convolution with several randomly initialized 3x3 convolutions in semantic segmentation but it does not converge well. Do you try such scheme on other tasks or datasets? Or are you using any particular optimization techniques?