Data Scientist (Performance & Trust)
Amigo
This job is no longer accepting applications
See open jobs at Amigo.See open jobs similar to "Data Scientist (Performance & Trust)" General Catalyst.Data Science
Posted on May 21, 2025
Data Scientist (Performance & Trust)
To apply, send us your resume and anything else you'd like to careers@amigo.ai
About Amigo
We're helping enterprises build autonomous agents that reliably deliver specialized, complex services—healthcare, legal, and education—with practical precision and human-like judgment. Our mission is to build safe, reliable AI agents that organizations can genuinely depend on. We believe superhuman level agents will become an integral part of our economy over the next decade, and we've developed our own agent architecture to solve the fundamental trust problem in AI. Learn more here.
Role
As a Data Scientist at Amigo, you'll apply rigorous statistical methodologies to establish the quantitative foundation of our Trust Center Framework. Working as part of our Research team, you'll design measurement systems that validate agent performance with academic rigor, create confidence calculation frameworks across simulation runs, and quantify the composition of our agent architecture. Your work will provide the statistical backbone for our public validation of agent performance and enable evidence-based decision making as we scale from initial customers to millions of conversations per month. This role is critical for establishing the industry standard for autonomous agent validation and providing the quantitative evidence that builds trust in regulated industries.
Responsibilities
Design statistical frameworks for our Trust Center Framework (agi.work) that validate agent performance with academic rigor
Develop mathematical models that quantify trust, reliability, and performance in high-stakes domains
Create autoscaling algorithms for compute fleets that optimize resource allocation as we scale to 100x volume
Build projection models for quota growth and capacity planning across multi-workspace, multi-region deployments
Establish comprehensive state analysis methodologies that measure the composition of agent operations (percentage of identity, constraints, guidelines, data, dynamic behavior in prompts)
Design systems for context traceability that enable statistical validation of agent reasoning pathways
Develop confidence calculation methodologies for multiple simulation runs, analyzing variance across scenarios and deployment conditions
Create mathematical frameworks for judge coverage measurement, ensuring comprehensive evaluation of agent performance
Implement metrics that tie directly to interpretability, safety, and business value across our agent architecture
Design attribution systems that identify which components of our architecture (context graphs, memory systems, dynamic behaviors) contribute to performance improvements
Develop statistical methodologies for the neighborhood expansion problem, measuring capability growth while maintaining reliability
Collaborate with Verification and Simulation researchers to establish quantitative evaluation standards
Contribute to academic publications and technical content that demonstrate our scientific approach
Work with engineering teams to implement measurement systems in production environments
Qualifications
Advanced degree in statistics, data science, applied mathematics, or related quantitative field
Deep understanding of statistical methods, experimental design, and measurement frameworks
Experience applying quantitative approaches to complex system evaluation
Background developing metrics systems for performance evaluation of AI or software systems
Strong knowledge of confidence calculation, variance analysis, and statistical validation techniques
Experience designing experiments that quantify system behavior under various conditions
Proficiency with Python, data analysis libraries, and statistical tools
Understanding of how metrics tie to business objectives and technical capabilities
Background in data visualization for communicating complex performance metrics
Publication experience in relevant scientific or technical venues preferred
Passion for establishing rigorous, scientifically valid approaches to AI measurement
Location: NYC (Onsite)
To apply, send us your resume and anything else you'd like to careers@amigo.ai
This job is no longer accepting applications
See open jobs at Amigo.See open jobs similar to "Data Scientist (Performance & Trust)" General Catalyst.