Skills Proximity: Validating scoring

•  Define job descriptions with clear, varied skill requirements.
•  Prepare candidate profiles with:
  •  Exact skill matches
  •  Adjacent, related skills
  •  Unrelated skills
•  Establish ground truth scores with subject matter experts (SME) input.

• Compare AI scores to SME scores.
•  Confirm high scores for strong matches.
•  Confirm low scores for weak or irrelevant matches.
•  Validate that adjacent skills are scored appropriately.

• Rerun identical inputs to check score consistency.
•  Slightly modify inputs to test sensitivity.
•  Ensure similar profiles yield similar scores.

• Review skill match breakdowns.
• Confirm relevant skills are weighted correctly.
• Check for unexpected or missing skill matches.

•  Plot score ranges across test cases.
•  Identify clustering or outliers.
•  Ensure scores reflect meaningful differentiation.

• Test profiles with:
  •  Missing skills
  •  Ambiguous terminology
  •  Outdated or uncommon skills
•  Validate scoring behavior in these cases.

•  Retest after model updates.
•  Confirm no unintended changes in scoring.

• Collect feedback from end users.
• Compare feedback with scoring results.
• Adjust test cases based on feedback trends.

• Record test scenarios and outcomes.
• Note assumptions and limitations.
• Share findings with stakeholders.

Review the Understanding the Components article to learn how scores are generated.

• Skills Proximity scores are based on semantic similarity between job requirements and candidate profiles.
• The model uses AI embeddings to compare skills contextually, not just by keyword matching.

• Have SMEs manually score candidate-to-job matches.
• Use these expert scores as a benchmark to compare against AI-generated scores.

Use realistic requisitions and candidate profiles

• Start with a few job postings and a variety of candidate profiles.

• Create a few sample job descriptions with clearly defined skills.

• Create diverse job descriptions with varying skill requirements.

• Prepare candidate profiles or resumes with varying degrees of skill overlap.

• Use realistic candidate profiles or resumes with overlapping, adjacent, and unrelated skills.

• Include candidates with exact, related, and missing skills.

• Aim to test the spectrum of match quality from strong to weak.

• Include edge cases like missing skills, outdated terminology, or ambiguous phrasing.

Evaluate Consistency & Relevance

• Submit the job descriptions and candidate profiles through an Enterprise site where Skills Proximity is set up.

• Test across multiple roles and skill domains to verify consistent, meaningful scoring.

• Confirm that related skills are appropriately inferred, for example, React for JavaScript roles.

• Run the same profiles against similar job descriptions to check for score stability.
• Slight changes in wording should not cause large score fluctuations unless justified.
• Submit a resume in a language other than English.
• Capture the Skills Proximity scores generated for each candidate.

Review the Proximity Score and Job Rank

• Use the guidance in Maximizing AI Data to interpret score ranges and thresholds.

• Validate whether candidates with closely aligned skills appear at the top.

• Ensure the Skill Proximity Score and Job Rank Match reflect expected alignment.

• Compare the scores against your expectations:
  • Do candidates with closely matching skills receive higher scores?
  • Are scores consistent across similar profiles?
• Plot score distributions to identify:
  • Clustering around certain values
  • Outliers or unexpected gaps
• Ensure scores reflect meaningful differentiation between candidates.

Check the Skill Breakdown

• Use the skill match breakdowns and the job rank matches and skills proximity scores in the Understanding the Components article to verify:

  1. Which skills contributed most to the score.
  2. Whether irrelevant skills were weighted incorrectly.
  3. Validate how each skill contributes to the overall proximity score.
• Examine the AI-generated summary of:

  • Matched skills

  • Inferred skills (relevant but not explicitly listed)

  • Missing skills

• Make sure these categories are accurate and provide clear reasoning for the overall score.

• Rerun previous test cases after any model updates to ensure no unintended changes. 
• Maintain a versioned test suite for ongoing validation.

Collect User Feedback

• Involve recruiters or hiring managers to confirm whether the system is surfacing the right candidates.

• Look for consensus between human judgment and AI scoring.

• Collect feedback from end users on score relevance and usefulness.

• Use this feedback to refine test cases and improve model alignment with user expectations.

• Refer to the Overview article for how Beeline Enterprise continuously trains and improves the model using client feedback and broader labor market data.
• Clearly note what the model does and doesn’t consider, for example, certifications, experience depth.
• Educate your users on how to interpret scores appropriately based on context.
• Document any anomalies or unexpected results.