Comparatively Studying Modern Optimizers Capability for Fitting Vision Transformers

Abdullah Nazhat Abdullah, Tarkan Aydin

Araştırma sonucu: Kitap/Rapor/Konferans sürecindeki bölümKonferans katkısıbilirkişi

Özet

The Transformer architectures have been achieving great strides in both research and industry, garnering high adoption due to their versatility and generality. These qualities, combined with the availability of internet-scale datasets, open the path to constructing deep learning systems that can target many modalities and several tasks within each modality. Throughout the years, many optimization algorithms have been proposed and utilized in fitting Deep Learning models. Although many comparative assessments were made that investigated analyzing and selecting the best optimizer to fit architectures prior to Transformers, the literature lacks such extensive assessments in relation to optimizing Transformer-based deep learning models. In this paper, we investigated modern and recently introduced deep learning optimizers and applied the comparative assessment to multiple Transformer architectures implemented for the task of image classification. It was discovered experimentally by our comparative study that the novel optimizer LION provided the best performance on the target task and datasets, proving that the algorithmic design of optimizers can compete with and surpass handcrafted optimization schemes that are normally used in fitting Transformer architectures.

Orijinal dilİngilizce
Ana bilgisayar yayını başlığı7th EAI International Conference on Robotic Sensor Networks - EAI ROSENET 2023
EditörlerÖmer Melih Gül, Paolo Fiorini, Seifedine Nimer Kadry
YayınlayanSpringer Science and Business Media Deutschland GmbH
Sayfalar77-87
Sayfa sayısı11
ISBN (Basılı)9783031644948
DOI'lar
Yayın durumuYayınlanan - 2024
Harici olarak yayınlandıEvet
Etkinlik7th EAI International Conference on Robotics and Networks, ROSENET 2023 - Istanbul, Turkey
Süre: 15 Ara 202316 Ara 2023

Yayın serisi

AdıEAI/Springer Innovations in Communication and Computing
ISSN (Basılı)2522-8595
ISSN (Elektronik)2522-8609

???event.eventtypes.event.conference???

???event.eventtypes.event.conference???7th EAI International Conference on Robotics and Networks, ROSENET 2023
Ülke/BölgeTurkey
ŞehirIstanbul
Periyot15/12/2316/12/23

Parmak izi

Comparatively Studying Modern Optimizers Capability for Fitting Vision Transformers' araştırma başlıklarına git. Birlikte benzersiz bir parmak izi oluştururlar.

Bundan alıntı yap