Holy cannoli, it’s been a huge week for AI models! We’ve had major releases for LLMs, image generation, video generation, and more. Altogether, we have 20 models from just the past week and there’s a good chance we’ve still overlooked a couple. Without further ado, let’s jump in.

LLMs

Anthropic Claude 3.5 Sonnet (new)

Anthropic has recently unveiled a significant update to Claude 3.5 Sonnet, introducing groundbreaking capabilities and performance improvements. The upgraded model now boasts the ability to use computers, allowing it to interact with screens, move cursors, click buttons, and type text – a feature currently in public beta. This update has led to substantial enhancements in coding and tool use tasks, with Claude 3.5 Sonnet outperforming other publicly available models on benchmarks like SWE-bench Verified and TAU-bench. Additionally, users have reported faster response generation, improved reasoning capabilities, and enhanced code generation and debugging abilities1. While still experimental, this update positions Claude 3.5 Sonnet at the forefront of AI-powered coding and automation, offering exciting possibilities for developers and businesses alike.

IBM Granite 3.0 Series

IBM has released Granite 3.0, the latest generation of its open-source large language models designed specifically for enterprise use. Trained on over 12 trillion tokens across 12 natural languages and 116 programming languages, Granite 3.0 includes a range of models optimized for different enterprise tasks. The flagship Granite 3.0 8B Instruct model matches or exceeds the performance of leading similarly-sized open models on academic and enterprise benchmarks while prioritizing safety and efficiency. Released under the permissive Apache 2.0 license, Granite 3.0 aims to provide enterprises with powerful, flexible AI capabilities that can be customized for specific business needs at a fraction of the cost of larger proprietary models.

Cohere Aya Expanse Series

Cohere, a Canadian AI company, has recently unveiled its Aya Expanse series, a family of high-performance multilingual large language models that aim to bridge the global language divide in AI. The series includes two models with 8 billion and 32 billion parameters, covering 23 languages including English, Arabic, Chinese, and several others. These models, released with open weights, demonstrate superior performance compared to leading rivals like Google’s Gemma, Meta’s Llama, and Mistral AI’s models, even outperforming some larger models. Aya Expanse incorporates innovative techniques such as synthetic data generation, human feedback in late-stage training, and model merging to achieve its impressive multilingual capabilities. This release marks a significant step in Cohere’s ongoing Aya initiative, which has involved collaboration with over 3,000 researchers from 119 countries to advance multilingual AI research. The only downside is these models are released under a CC-by-NC 4.0 license, so commercial use isn’t an option.

Image Generation

Stability AI Stable Diffusion 3.5

Stability AI has recently unveiled Stable Diffusion 3.5, their most advanced text-to-image AI model to date. This release includes multiple variants, with Stable Diffusion 3.5 Large boasting 8 billion parameters and superior image quality at 1 megapixel resolution. The model excels in prompt adherence, diverse outputs, and versatile style generation, making it suitable for both professional and hobbyist use. Notably, Stable Diffusion 3.5 is released under the moderately permissive Stability AI Community License, allowing free use for non-commercial purposes and commercial use for individuals or organizations with annual revenue under $1 million. This licensing model aims to empower creators and businesses while maintaining accessibility to cutting-edge AI technology.

OmniGen

OmniGen is an innovative unified image generation model that represents a significant leap forward in AI-powered visual creation. Unlike traditional diffusion models that often require additional modules for specific tasks, OmniGen offers a simplified yet powerful architecture that can handle a wide range of image generation tasks within a single framework. This versatile model not only excels at text-to-image generation but also supports image editing, subject-driven generation, and visual-conditional generation without the need for extra preprocessing steps or additional encoders. OmniGen’s ability to transfer knowledge across different tasks and manage previously unseen domains makes it a truly adaptable tool for various creative applications. While the model shows great promise, it’s important to note that OmniGen is released under the CC BY 4.0 license, allowing for broad use and adaptation with proper attribution.

LibreFLUX

LibreFLUX is an exciting new open-source text-to-image generation model that aims to provide a free alternative to the popular FLUX model. Released under the Apache 2.0 license, LibreFLUX allows for commercial use without requiring additional licensing. This de-distilled version of FLUX restores the original training objective, enabling the use of classifier-free guidance (CFG) for improved image quality. LibreFLUX supports the full 512-token context length of the T5-XXL text encoder and implements attention masking, which helps maintain image quality for longer prompts. While it may not be fully de-distilled yet, as evidenced by some artifacts at higher CFG scales, LibreFLUX shows promise in generating diverse and creative images across various subjects.

DiGIT

DiGIT is an auto-regressive generative model for image generation that has achieved impressive results on the ImageNet dataset. Developed by researchers from DAMO-NLP-SG and accepted at NeurIPS 2024, DiGIT performs next-token prediction in an abstract latent space derived from self-supervised learning models. By employing K-Means clustering on the hidden states of the DINOv2 model, it creates a novel discrete tokenizer that significantly boosts image generation performance. DiGIT has achieved state-of-the-art FID scores, while also enhancing image understanding. The model’s architecture and approach offer a unified perspective on stabilizing the latent space for image autoregressive modeling. Importantly, DiGIT is released under the MIT License, making it accessible for both academic research and commercial applications.

Video Generation

Rhymes AI Allegro

Rhymes AI has recently unveiled Allegro, a groundbreaking open-source text-to-video generation model that rivals commercial-grade offerings. This powerful model can produce high-quality videos up to 6 seconds long at 15 FPS with 720p resolution, all from simple text prompts. Allegro boasts impressive efficiency, featuring a 175M parameter VideoVAE and a 2.8B parameter VideoDiT model, while requiring only 9.3 GB of GPU memory in BF16 mode with CPU offloading. What sets Allegro apart is its open-source nature – the full model weights and code are available to the community under the Apache 2.0 license, making it freely accessible for both academic and commercial use. This release marks a significant step forward in democratizing advanced video generation technology, allowing researchers and developers to build upon and improve the model for various applications.

Genmo Mochi 1 Preview

Genmo’s Mochi 1 Preview is making waves in the AI video generation space as an impressive open-source text-to-video model. Released under the permissive Apache 2.0 license, this 10 billion-parameter diffusion model showcases remarkable capabilities in creating high-fidelity motion and adhering closely to user prompts. Built on Genmo’s proprietary Asymmetric Diffusion Transformer (AsymmDiT) architecture, Mochi 1 stands out for its ability to generate realistic character movements and dynamic camera actions, offering creators unprecedented control over their AI-generated videos. While currently limited to 480p resolution, Genmo plans to release a full version with 720p HD video generation by the end of the year. With its open-source nature, Mochi 1 not only democratizes access to cutting-edge AI video technology but also invites developers and researchers to explore, modify, and integrate the model into their own workflows.

Multimodal Models

Microsoft OmniParser

Microsoft OmniParser is an innovative screen parsing tool designed to enhance the capabilities of AI agents interacting with user interfaces. This powerful tool converts UI screenshots into structured formats, enabling more accurate identification of interactable elements and better understanding of their semantic meanings. OmniParser combines a fine-tuned YOLOv8 model for detecting interactive regions with a fine-tuned BLIP-2 model for generating descriptive captions. By providing this structured information, OmniParser significantly improves the performance of large vision-language models like GPT-4V when generating grounded actions for various UI tasks across different platforms and applications. Microsoft has made OmniParser publicly available on GitHub under an MIT license, encouraging further research and development in the field of vision-based GUI agents.

H2O Mississippi Series

H2O.ai has recently introduced the Mississippi series of multimodal foundation models, specifically designed for OCR and Document AI applications. The series includes two models: H2OVL Mississippi-2B and H2OVL Mississippi-0.8B, both optimized for high-resolution image handling and text recognition tasks. These models are built on H2O’s Danube architecture and offer impressive performance in visual understanding and text extraction, outperforming many larger state-of-the-art Vision Language Models (VLMs) despite their smaller size. The Mississippi models are now available on Hugging Face under the permissive Apache 2.0 license, allowing developers and researchers to modify and adapt them for specific OCR and Document AI requirements. With their lightweight architecture and multimodal expertise, these models provide a cost-effective solution for businesses seeking to implement AI-powered document analysis and text recognition capabilities.

Oryx-1.5 Series

The Oryx-1.5 Series represents a significant advancement in large vision-language models developed by MBZUAI. This series includes a unified multimodal architecture designed for spatial-temporal understanding of images, videos, and multi-view 3D scenes. At its core is the OryxViT model, which can encode images at any resolution into LLM-friendly visual representations, along with a dynamic compressor module that supports 1x to 16x compression on visual tokens as needed. This flexibility allows Oryx to efficiently process visual inputs of varying spatial sizes and temporal lengths, from small icons to long videos. The series includes both 7 billion and 32 billion parameter models, both licensed under the permissive Apache 2.0. These models demonstrate enhanced capabilities in multimodal understanding, benefiting from specialized training on long-context retrieval and spatial-aware data.

DeepSeek Janus 1.3B

DeepSeek AI has recently unveiled Janus 1.3B, a groundbreaking autoregressive framework that unifies multimodal understanding and generation. This novel model addresses limitations of previous approaches by decoupling visual encoding into separate pathways while still utilizing a single, unified transformer architecture for processing. Janus 1.3B demonstrates impressive capabilities in both multimodal comprehension and image generation tasks, surpassing previous unified models and matching or exceeding the performance of many task-specific models. The model leverages DeepSeek’s expertise in building effective Mixture-of-Experts (MoE) language models and is constructed based on the DeepSeek-LLM-1.3b-base, which was trained on approximately 500B text tokens. It’s worth noting that while the code repository for Janus is licensed under the MIT License, the use of the Janus models themselves is subject to the DeepSeek Model License.

BAAI Emu3-Stage1

BAAI’s Emu3-Stage1 represents a significant advancement in multimodal AI models. This model is the result of the first stage of pre-training for Emu3, a state-of-the-art suite of multimodal models trained solely using next-token prediction. By tokenizing images and text into a discrete space, Emu3-Stage1 is trained from scratch on a mixture of multimodal sequences, with a context length of 5120 for text and image data. The model excels in both generation and perception tasks, outperforming well-established task-specific models and surpassing flagship open models like SDXL and LLaVA-1.6. Notably, Emu3-Stage1 supports image captioning and can generate images at a resolution of 512×512, making it a versatile tool for various image generation and perception tasks. While the model’s capabilities are impressive, it’s important to note that Emu3-Stage1 is released under the Apache License, Version 2.0, which allows for both commercial and non-commercial use with proper attribution.

Speech-to-Text

Moonshine

Moonshine is an exciting new family of speech-to-text models that promises to revolutionize automatic speech recognition (ASR) on resource-constrained devices. Developed by Useful Sensors, Moonshine offers faster and more efficient performance compared to the current state-of-the-art models like OpenAI’s Whisper, while maintaining or even surpassing their accuracy. What sets Moonshine apart is its ability to process input audio in real-time, with computational requirements that scale based on the length of the audio input. This makes it ideal for applications like live transcription and voice command processing on edge devices. The Moonshine models are open-sourced and available on GitHub, allowing developers and researchers to easily integrate them into their projects. While the model code is released under an MIT license, no information is given about the model license.

Looking Ahead

The rapid pace of AI model releases shows no signs of slowing down. As we move forward, we can expect:

1. Further integration of multimodal capabilities
2. Continued emphasis on open-source and accessible AI
3. More efficient models suitable for edge devices and real-time applications
4. Increased specialization for industry-specific use cases

These advancements are likely to accelerate the adoption of AI across various sectors and pave the way for more sophisticated AI-powered applications in the near future.

As always, it’s crucial for developers, researchers, and businesses to stay informed about these new models and carefully consider their capabilities, limitations, and licensing terms when integrating them into projects or workflows.

Stay tuned for our next Model Rundown, where we’ll continue to keep you updated on the latest and most impactful AI model releases shaping the future of technology.