Introducing «LongCAT» — the purrfect alternative to that other 10.7B Frankenmerger in town! Our long feline friend here is created through merging rishiraj/CatPPT-base using a passthrough merge using a new process called Interwoven Depth Up-Scaling resulting in the longest cat! We developed the Interwoven Depth Up-Scaling technique. Built on the Mistral architecture, LongCAT incorporates the innovative Interwoven Depth Up-Scaling. We then interwove Cat 7B weights into the upscaled layers, and finally, did absolutely no extended pre-training. All joking aside, this is an attempt to more coherently merge Mistral-7B models together than the typical Undi95/»Depth UP Scaling» technique that is typically used. The typical approach is to lay out the front 75% of one model and then place the back 75% of the second model together: i.e. [0, 24] + [8, 32] for a 7B merger. When laid out flat, this can be broken down as [0, 8]+[8, 24]+[8, 24]+[24, 32] with two discrete 16 layer blocks duplicated twice in a row. This typically is better than laying the entirety of one model out flat, ostensibly because of the locality of the duplicated layers to their original location. Taking this to its…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: SanjiWatsuki
Теги: mistral, merge, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 21
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
