Inside the SAAT Success: A Q&A on Transforming Summer Associate Evaluations

At AltaClaro, we’re always looking for ways to innovate and improve how law firms evaluate and develop their summer associates. This past summer, we partnered with a leading Am Law 100 firm to pilot our Summer Associate Assessment Tool (SAAT), and the results were remarkable. Today, we’re diving into a Q&A session to explore the outcomes of this pilot program and what it means for the future of legal recruiting and training.

.

Q: Can you briefly explain what the Summer Associate Assessment Tool (SAAT) is?

A: The SAAT is AltaClaro’s cutting-edge, scalable, hybrid AI/HI competency-assessment framework. It helps law firms evaluate their summer associates’ critical thinking skills on real legal matters through a standardized experiential assignment. It leverages data-driven feedback to make better hiring decisions, while also providing high-quality experiential training and personalized feedback to the associates. It uses a standardized, structured rubric for evaluating associate performance on simulated assignments, which unlocks a repeatable and consistent assessment process that leverages AI and HI for accuracy and efficiency.

Q: Why did the firm decide to partner with AltaClaro and pilot the SAAT?

A: The firm has been a valued partner of AltaClaro's since 2020. They previously used our M&A bootcamp, and their associates were always asking for more feedback and training. They recognized that providing a great summer associate experience is key to their recruitment and retention strategy. They wanted to give their summer associates access to the same high-quality, experiential training that their other associates were receiving. The firm also wanted to leverage AltaClaro for training and assessment because our assignments enable them to assess critical thinking skills, particularly how well associates can synthesize information and provide practical recommendations. Additionally, they were looking for a more standardized way to evaluate summer associates to mitigate bias and ensure fairness in hiring decisions.

Q: What were the key goals the firm had for implementing the SAAT?

A: Their key goals were multifaceted:

• Enhancing Training and Recruitment: The firm wanted to provide high-quality training to summer associates, aligning it with their recruitment and retention strategies.
• Assessing Critical Thinking and Writing Quality: They aimed to evaluate critical thinking skills and writing abilities, particularly the ability to write concisely and practically for real-life legal scenarios.
• Improving Hiring Decisions: The firm sought to use data from the SAAT to make more informed, objective, and fair hiring decisions.
• Ensuring Fairness: They wanted to mitigate unconscious bias and ensure the evaluation process was based on merit.

Providing Personalized Feedback: The goal was also to identify associates who might need extra support and provide tailored coaching.

Q: What were some of the significant outcomes of the summer pilot program?

A: The pilot program yielded several significant outcomes:

• Valuable Data for Borderline Decisions: The SAAT provided crucial data that helped the firm make more informed decisions regarding borderline candidates. Submission timeliness, engagement, and participation were also key factors.
• Insights into Learning Mindset: AltaClaro’s data revealed insights into the students’ mindset towards learning and engaging with challenging tasks, which the firm found uniquely valuable.
• No Correlation with Traditional Metrics: Surprisingly, there was no correlation between the AltaClaro results and traditional data points like law school ranking and GPA. This highlighted the tool's ability to identify potential beyond academic performance.
• Identifying Practical Application Skills: The tool helped identify students who excelled academically but lacked practical application skills, which is crucial for the firm’s work environment.
• Boosting Confidence: For some high-performing students, the detailed feedback from AltaClaro boosted their confidence.
• Reinforcing and Solidifying Hiring Decisions: The SAAT reinforced decisions on which candidates not to offer positions and helped solidify decisions for borderline candidates, making the hiring process more efficient and objective.

Surprising Results: In a few cases, the SAAT data provided surprising results that changed the firm’s perspective on certain candidates’ potential.

Q: Can you elaborate on the "no correlation with traditional metrics" finding?

A: Yes, this was a particularly interesting outcome. Traditionally, law firms often rely on factors like law school ranking and GPA to evaluate candidates. However, the firm found that the AltaClaro results showed no direct correlation with these metrics. This suggests that the SAAT is tapping into a different set of skills—practical application, critical thinking, and problem-solving—that aren't necessarily reflected in academic performance. It helps identify individuals who may excel in real-world legal practice, even if their academic record isn't stellar, and vice-versa.

Q: How did the SAAT help with fairness and mitigating bias in hiring decisions?

A: The firm specifically aimed to ensure fairness and mitigate bias in their hiring process. They recognized that relying solely on “cultural fit” or subjective feedback can lead to unfair evaluations and limit opportunities for diverse candidates. The standardized rubric and data-driven feedback provided by the SAAT offered a more objective way to assess candidates, focusing on their actual skills and performance. This helped the firm identify and foster talent based on merit.

Q: What kind of feedback did the summer associates receive?

A: The summer associates received detailed, personalized feedback based on a standardized rubric. The SAAT leverages both AI and human intelligence (HI) to provide this feedback. AI tools analyze the assignments, and then AltaClaro-certified assessors use that data to create personalized assessments. The feedback covers various factors, including following instructions, identifying legal and business issues, and developing practical solutions. It also includes information on how they did compared to their cohort and the AltaClaro average.

Q: Any final thoughts on the SAAT pilot program?

A: The SAAT pilot program was a resounding success. It demonstrated the power of data-driven assessments in transforming summer associate evaluations and hiring decisions. We’re excited to continue innovating and helping law firms identify and develop the next generation of legal talent. We believe the SAAT is not just a tool for evaluation, but also a powerful instrument for growth, development, and ensuring a more equitable legal profession.