AI Safety Founders

𝐉𝐨𝐢𝐧 𝐮𝐬 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐕𝐂 𝐒𝐩𝐨𝐭𝐥𝐢𝐠𝐡𝐭 𝐞𝐯𝐞𝐧𝐭 𝐟𝐞𝐚𝐭𝐮𝐫𝐢𝐧𝐠 𝐌𝐚𝐠𝐧𝐮𝐬 𝐇𝐚𝐦𝐛𝐥𝐞𝐭𝐨𝐧, 𝐕𝐂 𝐚𝐭 𝐛𝐲𝐅𝐨𝐮𝐧𝐝𝐞𝐫𝐬 𝐭𝐡𝐢𝐬 𝐓𝐡𝐮𝐫𝐬𝐝𝐚𝐲 💥 Hear about Deeptech/AI investing, and industry-defining startups across the Nordics. Magnus, an investor in Lovable & DataCrunch, has a strong track record of supporting ambitious founders and previously hosted The Last Supper event on AI advancements. Gain insights into the Nordic/European startup ecosystem, funding trends, and what investors think about current AI assurance opportunities. https://lu.ma/yceq9x2q

𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐀𝐈 𝐚𝐠𝐞𝐧𝐭𝐬 𝐢𝐬 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐝𝐚𝐧𝐠𝐞𝐫𝐨𝐮𝐬 𝐩𝐚𝐭𝐡 𝐰𝐞'𝐫𝐞 𝐧𝐨𝐰 𝐞𝐧𝐭𝐞𝐫𝐢𝐧𝐠 𝐢𝐧𝐭𝐨. - Yoshua Bengio at Davos 2025 During a landmark AI panel with DeepMind's Demis Hassabis, Berkeley's Dawn Song, and former Baidu President Ya-Qin Zhang, the Turing Award winner cut through the complexity: systems without agency (self-preservation) do not pose the risk of losing control. Yet we're racing toward agentic AI. The twist? Our biggest AI breakthroughs - from medical discoveries to climate solutions - don't even require agency.

𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐛𝐮𝐢𝐥𝐝 𝐬𝐨𝐦𝐞𝐭𝐡𝐢𝐧𝐠 𝐭𝐫𝐮𝐥𝐲 𝐢𝐦𝐩𝐚𝐜𝐭𝐟𝐮𝐥 𝐰𝐡𝐢𝐥𝐞 𝐠𝐨𝐢𝐧𝐠 𝐭𝐡𝐞 𝐕𝐂 𝐫𝐨𝐮𝐭𝐞? Leonard Tang (Haize Labs) joined us to share his journey from AI safety research at Harvard to building a company that's now used by major AI labs and enterprises to test their models. The most striking insight? The intersection of academic AI safety work and commercial viability isn't just possible, it's powerful. Haize turned research on model evaluation and robustness into a platform that helps companies answer "can we trust this model in production?" Leonard's approach challenges a common assumption in our community: that you have to choose between meaningful safety work and commercial success. His team publishes their research openly with a recent paper accepted at ICLR, while building a business that makes AI deployments measurably safer. 𝐅𝐨𝐫 𝐚𝐬𝐩𝐢𝐫𝐢𝐧𝐠 𝐟𝐨𝐮𝐧𝐝𝐞𝐫𝐬: Notice how Haize connects safety to concrete business outcomes. They're helping companies ship AI with confidence. That's the kind of framing that got them backing from General Catalyst and allows them to keep doing substantive technical work. 𝐓𝐡𝐢𝐬 𝐢𝐬 𝐰𝐡𝐚𝐭 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐀𝐈 𝐬𝐚𝐟𝐞𝐭𝐲 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞 𝐢𝐧 𝟐𝟎𝟐𝟓: Deep technical work on robustness and evaluation, packaged into tools that make safer AI deployment the easier choice.

Can we trust AI? “Everyone in the world is getting onto this brand new airplane that really has never been tested before. And it’s going to take off and it’s never going to land. It has to fly, forever.” – Stuart Russell, IASEAI President and Distinguished Professor of Computer Science at the University of California, Berkeley.

Some thoughts on the AI Action Summit in Paris after being here over the week... and it seems like Macron and France to use this Summit primarily as a fundraising platform for their announced €100 billion investments in AI. I think AI Safety advocates might have eased off too early after securing wins with the AI Safety Institutes and the Bletchley and Seoul Summits. Acting mostly in good faith, we probably underestimated how much continuous pressure is needed to fully win a campaign. To really get a safe world with AI we cannot put the pressure off until it's done. With frontier labs racing to AGI, we need to put real pressure on governments to keep us safe from them. The slow first three quarters of 2024 meant the bloc didn't press as hard as they could have. Maybe this is happening because they don't fear repercussions/pressure from the AI safety bloc anymore? Have we shown ourselves to be a soft touch and put our foot off the pedal? Maybe the solution is: all the AISIs (AI Safety Institutes) need their own dedicated conferences, separate from these broader AI Summits. We shouldn't have to constantly worry about pushing host nations to prioritize AI safety. But it was encouraging to read that the Secretary of State for DSIT Peter Kyle, who oversees the UK AISI, wants to give the UK AI Safety Institute legislative power to compel frontier labs to submit their models for pre-deployment evaluation. So I'll look out and hope we see that in their AI Bill. JD Vance made that clear on the US side what they think of making the world safe from advanced AI, but I still hold out *some* hope Labour Party (UK) will build on Rishi Sunak's AI safety legacy. In the Don't Worry About the Vase newsletter I said that it "Seems like France used the Summit as a fundraiser for [Macron's] €100 billion" and I mean it! [https://lnkd.in/eTKNBx8i]

Apart News: Esben Kran at International Association for Safe & Ethical AI & Studio Progress Report. *Esben at IASEAI* Esben gave his Engineering a World Designed for Safe Superintelligence talk at The Inaugural Conference of the International Association for Safe and Ethical AI, and it was incredibly well received. Whilst Connor Axiotes attended the launch of the Hiroshima AI Process (HAIP) Reporting Framework with the OECD - OCDE in Paris. *Upcoming Studio Progress Report* Coming next week: our collection of AI safety research from our Studio's inaugural cohort shows how fast promising ideas can move from hackathon to real-world impact. They came primarily from these sprints: - Reprogramming AI Models - Agent Security - AI Policy at Johns Hopkins University In just two months, six teams of researchers have produced compelling findings across critical areas of AI safety - from detecting AI-powered cyber attacks to understanding if LLMs make moral decisions. One team deployed "honeypot" servers to detect AI hacking agents in the wild, finding that while they exist, they represent only 0.0001% of current attacks. Another team used Minecraft to study AI goal drift, discovering that agents gradually deviate from their objectives over time. Some teams tackled medical AI hallucinations using sophisticated neural feature analysis, others developed new ways to visualize the exponentially growing field of AI safety research through interactive knowledge graphs. Our Studio accelerates the path from initial concept to meaningful research. By providing structured support and resources to promising hackathon projects, the program helps early-stage researchers make rapid progress on crucial AI safety challenges. The full collection of research blogs will be released next week, offering detailed insights into each team's methodology and discoveries! For those interested in joining future cohorts, sign up for upcoming hackathons for your chance to join the Studio. Stay tuned for next week when we will have the full writeup! https://lnkd.in/eifpx6H7

https://lnkd.in/eUkWn2ZQ

Our *BRAND NEW* 'Hunting for AI Hackers: LLM Agent Honeypot' Apart Lab Studio research blog attempts to ascertain the current state of AI-powered hacking in the wild through an innovative 'honeypot' system designed to detect LLM-based attackers. The Studio process picks the most promising hackathon projects and ask fellows to build on their work, develop their ideas, and write up their research as a blog. The following piece is written by Reworr R., one of our first Apart Lab Studio Fellows. https://lnkd.in/eiC5_WQx

Demo: Scheming reasoning evaluations. In December 2024, we released our paper on scheming reasoning evaluations where we showed that several frontier AI systems are capable of in-context scheming against their developers or users. We have now released an accompanying demo video and blog post that talks through the details of the demo video.

𝗨𝗽𝗰𝗼𝗺𝗶𝗻𝗴 𝗦𝘁𝗮𝗿𝘁𝘂𝗽 𝗦𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁 𝘄𝗶𝘁𝗵 𝗖𝗼-𝗙𝗼𝘂𝗻𝗱𝗲𝗿 & 𝗖𝗘𝗢 𝗼𝗳 Haize Labs Haize Labs provides the trust, safety, and reliability layer for AI models across industries. Through systematic stress-testing and red-teaming (haizing), they identify and mitigate failure modes to enable secure AI adoption. Sing-up to get the chance to chat with one of the most promising AI safety startups out there! 👉 https://lu.ma/ljt4bk8o 𝗝𝗮𝗻 𝟯𝟬𝘁𝗵 𝟭𝟬 𝗔𝗠 𝗣𝗧 / 𝟭𝟵:𝟬𝟬 𝗖𝗘𝗧