Each skill is a few short lessons, then practice. You rate yourself at the start and again at the end.
Core to the job
Chapter 2
AI language models
Measure the size of AI requests and compare randomness settings
What you'll learn
Tokens and context
Sampling variation
Model trade-offs
Embeddings for search
Practice
Count Tokens and Predict Context Limits15 min
Choose and Justify a Model Stack for Three Real Scenarios25 min
Chapter 4
AI evaluation
Collect examples for an AI feature and grade its answers
What you'll learn
Golden dataset
Scoring rubric
Judge model limits
Quality threshold
Production sampling
A real drop or noise
Scoring an agent run
Practice
Score fifty live answers against the offline rubric and explain the gap25 min
Read one trace and name every place the answer could have gone wrong20 min
Chapter 6
Programming languages
Build small programs in one language, handling input and errors
What you'll learn
Types and values
Control flow
Error handling
Asynchronous work
Practice
Trace a Function and Predict Its Output10 min
Chapter 7
Writing tests
Write tests for small functions, including odd and wrong inputs
What you'll learn
Testing pyramid
Behaviour test
Test isolation
Regression test
Practice
Write Test Cases for a Tip Calculator20 min
Write unit tests for a small function20 min
Chapter 8
AI feature safety
Spot tricks that make an AI ignore its rules
What you'll learn
Prompt injection
AI attack surface
Guardrail layers
Guardrail trade-off
AI threat model
Agent action risk
Practice
Spot the Guardrail Gap in These Test Prompts15 min
9 more in the plan
Prompt engineeringWrite prompts that use examples and ask for machine-readable answersCh 2
What you'll learn
System and user prompts
Format constraints
Few-shot examples
Prompt versions
Practice
Fix a Flaky Prompt by Adding a Format Constraint15 min
Data labellingLabel AI outputs using a guideline and check labellers agreeCh 4
What you'll learn
Labeling guidelines
Fair sampling
Baselines and noise
Inter-rater agreement
DebuggingUse error messages to find bugs and explain your fixesCh 6
What you'll learn
Hypothesis before keyboard
Reliable reproduction
Root cause, not symptom
Narrow it down
Reading a stack trace
Practice
Find the Bug Using an Error Message10 min
Automated releasesExplain what each automatic check on a code change catchesCh 7
What you'll learn
CI and CD
Pipeline stages
Release gate
Pipeline logs
Pull or push
Reconciliation loop
Where the check runs
Environments and promotion
Practice
Plan a simple CI pipeline for a small repo30 min
Problem solvingTest each possible cause of a problem instead of guessingCh 9
What you'll learn
Problem vs symptom
Clean problem splits
Root cause
Deciding under uncertainty
Practice
Separate the Problem from the Symptom10 min
Critical thinkingSpot a weak assumption and ask where numbers come fromCh 9
What you'll learn
The hidden assumption
Argue the other side
The base rate
Fact vs inference
Wanting it to be true
Practice
Spot the Weak Link in an Argument10 min
Adapting to changeTry a new tool and describe how you handled changeCh 9
What you'll learn
Change vs your response
The messy in-between
Not there yet
Real change vs a feeling
CommunicationRewrite an update for two different readers, cutting extra linesCh 10
What you'll learn
Lead with the point
Match detail to the reader
Said versus heard
Specific feedback
Practice
Rewrite the Same Update for Two Different Readers20 min
Working with othersContribute to group work, ask for help and credit teammatesCh 10
What you'll learn
Early team friction
Clear ownership
Challenge ideas, not people
Lead or follow
Practice
Navigate a Disagreement Without Damaging the Relationship20 min
Your plan
Chapter by chapter.
14 chapters, in this order. Chapter 1 is free.
1~2 wks
See the work of an Evals and AI quality engineer
StartFree
2~2 wks
Write prompts that give reliable answers
Skills
3~2 wks
Explain a system you built
Project
4~2 wks
Measure AI quality with labels and statistics
Skills
51–2 wks
Talk to two working AI quality engineers
People
6~2 wks
Write code and fix your own bugs
Skills
7~2 wks
Automate tests and releases
Skills
8~2 wks
Find unsafe AI answers before users do
Skills
9~2 wks
Think problems through when plans change
Skills
10~2 wks
Make yourself understood across a team
Skills
11~2 wks
Capstone: build a working AI feature
Project
121–2 wks
Answer the questions interviewers ask
Interview
131–2 wks
Decide how you will get in
Decide
14on your timeline
Send your first applications
Apply
The job
What a Evals and AI Quality Engineer does
Builds evaluation suites and test harnesses that measure whether AI features work well and stay safe. This role checks large language model (LLM) features and agents, then sets quality gates so weak versions never reach users.
The work
Design evaluation suites that score how well AI features answer real tasks
Build offline evals that replay saved examples and grade each output
Write regression tests so new model versions do not break old behavior
Set up LLM-as-judge checks where one model grades another model's output
Create guardrail and red-team tests that probe for unsafe or wrong answers
Add quality gates that block a release when scores drop below a limit
Monitor live AI features and flag drops in accuracy or new failure types
A typical day
Check overnight eval runs and monitoring dashboards for new failures
Pick a weak feature, collect failing examples, and label them
Build or tune an eval, then run it against two model versions
Review scores with researchers and agree what counts as good enough
Wire a passing eval into the release pipeline as a quality gate
Write up findings and share which prompts or models to ship
Tools
Languages: Python, TypeScript, SQLEval frameworks: OpenAI Evals, Promptfoo, DeepEval, RagasAI model access: OpenAI API, Anthropic API, Hugging Face modelsExperiment tracking: LangSmith, Weights and Biases, MLflowVersion control: Git, GitHubCI and CD: GitHub Actions, GitLab CIData and notebooks: Pandas, JupyterMonitoring: Grafana, custom dashboards
Ways in
There is more than one route.
Ways to study
Bachelor degree in computer science, software engineering, or data science
Diploma in programming or machine learning (ML) with strong project work
Coding bootcamp plus self-study in testing and AI evaluation
Self-taught path with online courses on LLMs and a portfolio of evals
How people get their first job
Build 2 to 4 eval projects that grade an AI feature and share clear results
Apply for internships in quality assurance, ML, or AI product teams
Start in software testing or data labeling and move into eval work
Contribute eval cases or fixes to open-source AI test frameworks
Take a short course on prompt testing and ship a public eval suite
How long does it take to become a evals and ai quality engineer with Welica?
The plan is 14 chapters, 5–6 months at 15 to 30 minutes a day. You can go faster or slower; the plan moves with you.
Do I need a degree?
Not always. Common routes are bachelor degree in computer science, software engineering, or data science, diploma in programming or machine learning (ML) with strong project work, coding bootcamp plus self-study in testing and AI evaluation, or self-taught path with online courses on LLMs and a portfolio of evals.
What is free?
Chapter 1, See the work of an Evals and AI quality engineer, is free for good. Premium unlocks the rest of the plan.