pith. sign in

arxiv: 2302.01327 · v3 · pith:P2QCOYP6new · submitted 2023-02-02 · 💻 cs.CV · cs.LG

Dual PatchNorm

Manoj Kumar , Mostafa Dehghani , Neil Houlsby This is my paper
classification 💻 cs.CV cs.LG
keywords dualpatchnormlayertransformersvisionaccuracyalternativebefore
0
0 comments X
read the original abstract

We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization

    cs.CV 2024-12 unverdicted novelty 6.0

    DOLLAR combines variational score and consistency distillation for few-step video generation plus latent reward optimization, reporting 82.57 VBench score and up to 278x speedup over the teacher diffusion model for 12...