Caroline Mitterdorfer began her fertility journey after a cancer diagnosis at age 27. She co-founded Levy Health to help speed Read more
Large corporations typically play it safe when it comes to consumer hardware, sticking to incremental updates year after year. Lenovo, Read more
Welcome to the Edtech World Big news in the education tech world! PowerSchool, the edtech giant, recently experienced a data Read more
Telegram Expanding Access to Crypto Wallet Telegram is now offering its crypto wallet to its 87 million users in the Read more
Thought Pokémon was a tough benchmark for AI? Well, some researchers argue that Super Mario Bros. is even tougher. Hao AI Lab at the University of California San Diego recently put AI to the test in live Super Mario Bros. games. Anthropic’s Claude 3.7 came out on top, with Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o struggling to keep up.
The game wasn’t exactly the same as the original 1985 release, as it ran in an emulator and integrated with a framework called GamingAgent. This framework gave the AIs control over Mario by providing basic instructions and in-game screenshots.
Interestingly, the lab found that reasoning models, which “think” through problems step by step, performed worse than “non-reasoning” models in the game. This is because reasoning models take longer to decide on actions, which can be detrimental in real-time games like Super Mario Bros.
While some experts question the significance of AI’s gaming skills in technological advancement, flashy gaming benchmarks like these highlight an “evaluation crisis.” As Andrej Karpathy from OpenAI mentioned, it’s unclear which metrics to focus on when evaluating AI models. But hey, at least we can enjoy watching AI play Mario.
