Preference Model Raises $16M Seed Led by Andreessen Horowitz for RL Environments

Preference Model Secures $16M Seed Round

Preference Model, a superintelligence data research company, has raised $16 million in seed funding. The round was led by Andreessen Horowitz, with participation from a notable group of investors including Scale Angels, Fei-Fei Li, SignalFire, Ian Goodfellow, Swyx, South Park Commons, Julian Schrittwieser, Manifund, MoE Capital, Jacob Jackson, Sammy Sidhu, Barry McCardel, and Dylan Patel.

What Preference Model Does

Preference Model builds reinforcement learning environments for training more capable, better-aligned AI models. The San Francisco-based company spent its first year building reward-hacking-resistant RL environments for frontier AI laboratories, focused on machine-learning engineering and AI research tasks.

Founded by Jennifer Zhou, an early member of Anthropic's data team, and Ning Cao, an early employee at DatologyAI, Preference Model is addressing a critical challenge in AI development: creating training environments that resist reward hacking and promote better alignment.

Open-Source Framework: Karotte

Alongside its seed round, Preference Model open-sourced Karotte, the production framework it uses to build and grade adversarially hardened training environments. This move signals the company's commitment to advancing the broader AI research community while commercializing its core technology.

Funding Details

• Round: Seed
• Amount: $16,000,000 USD
• Date: October 7, 2026
• Lead Investor: Andreessen Horowitz
• Additional Investors: Scale Angels, Fei-Fei Li, Ian Goodfellow, Swyx, South Park Commons, Julian Schrittwieser, Manifund, MoE Capital, Jacob Jackson, Sammy Sidhu, Barry McCardel, Dylan Patel

Why It Matters

As frontier AI laboratories push toward more capable models, the need for robust, reward-hacking-resistant training environments becomes increasingly important. Preference Model's focus on machine-learning engineering and AI research tasks positions it at the intersection of AI safety and capability development.

The involvement of high-profile investors and researchers — including Fei-Fei Li, Ian Goodfellow, and Dylan Patel — underscores the perceived importance of this work. The open-sourcing of Karotte could accelerate adoption and establish Preference Model as a key infrastructure provider for RL environment creation.

Acquisition Angle

For acquirers in AI infrastructure, developer tools, or frontier AI labs, Preference Model represents a potential target with specialized expertise in RL environments and alignment. The company's open-source framework and relationships with frontier labs could provide strategic value. As demand for better-aligned, more capable AI models grows, Preference Model's technology and talent could become attractive to larger players seeking to strengthen their training pipelines.

About Preference Model

• Founded by: Jennifer Zhou and Ning Cao
• Headquarters: San Francisco
• Focus: Reinforcement learning environments for AI training
• Stage: Seed
• Website: preferencemodel.com