Tracely is an innovative open-source developer tool designed to solve one of the biggest challenges in AI engineering: preventing regression failures in AI agents. Built by ML engineer Julien Wuthrich, Tracely transforms real-world production failures into robust, automated regression tests without requiring developers to manually author evaluation datasets.
As agent traces land, Tracely automatically grades them, intelligently clusters failures into actionable issues, and freezes problematic runs into hermetic, replayable test cases. By integrating directly into your development workflow, it acts as a safeguard that can block pull requests which risk shipping the same failures back into production. Licensed under the permissive MIT license, Tracely brings seamless reliability and observability infrastructure to teams building advanced AI systems.
Tracely is an open-source developer tool under the MIT license that automatically grades AI agent traces, clusters production failures into issues, and converts bad runs into hermetic, replayable regression test cases.
Tracely freezes problematic production runs into replayable test cases and blocks pull requests that would ship those same failures back into your application.
No. Tracely relies on the philosophy that "the trace is the test," eliminating the need to hand-author test datasets by leveraging live production failures instead.
0 community reviews for Tracely
Help the community by sharing your experience. Your review helps others make better decisions.
Ml engineer in SF
