flax-community/clip-vit-base-patch32_mbart-large-50 - Каталог нейросетей
Генерация текста

flax-community/clip-vit-base-patch32_mbart-large-50

Добавлено:
flax-community/clip-vit-base-patch32_mbart-large-50

Pretrained CLIP-Vision-mBART50 pre-trained on subset of translated Conceptual-12M image-text pairs using a seq2seq model training objective. 2.5M cleaned English image-text pairs are translated using Marian Model for respective languages to 2.5M examples each in English, French, German and Spanish. We trained CLIP-Vision-mBART50 model during community week hosted by Huggingface 🤗 using JAX/Flax. CLIP-Vision-mBART50 is a modified transformers model which takes in visual embeddings from CLIP-Vision transformer and feeds into the encoderhiddenstates of a mBART50 decoder. This is done for deep cross-modal interaction via cross-attention between the two modes. The decoder then predicts logits for the inputids` provided and can be used for generation. You can use the raw model for encoder decoder network where you want the encoder to encode images and decoder to decode text. Note that this model is primarily aimed at being fine-tuned on tasks like multi-lingual/mono-lingual image captioning. You will need to clone the model from here. An example of usage is shown below: The Multi-lingual image captioning model was trained on a subset of Conceptual 12M dataset by Google: Conceptual…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: flax-community
Теги: jax, tensorboard, clip-vision-mbart, endpoints_compatible
Лайков: 2  |  Загрузок: 29

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.