Human Feedback FOUNDATIONS
How human feedback and model evaluation work: applying rubrics, ranking responses, and rating consistently.
Counts towards Data Collection Foundations certification: 8 courses required.
Course Fee
Overview
Start your Human Feedback journey.
When an AI model learns what a better answer means, it learns it from people who compared two answers and said so.
That preference data trains a reward model directly: your comparisons become its scoring function, and the model it shapes will answer millions of people. The work is desk-based and rubric-driven, and consistency matters far more than speed.
This course teaches you to apply a rubric even when it contradicts your taste, including the classic case where the answer most raters prefer is the wrong one; to compare and rank responses while resisting length bias, confidence bias and fluency mistaken for accuracy; to write a justification that holds up when someone reads it back to you; and what gold checks, agreement scores, drift and fatigue are really measuring. Unusually for the programme, this course is mostly about recognising failure modes in yourself rather than in someone else's work: the auto-marked exercises name the bias you fell for. Around 45 minutes to 1.5 hours, self-paced; the free Data Collection Basics first is recommended.
One of the eight courses of the Data Collection Foundations certification programme: pass all eight to become a Certified Data Collector. Available on its own or under the programme fee; certificate included on passing.
What You Get
What's included in your training
Structured Learning Modules
Learn at your own pace through structured modules on the DeeLab Academy learning platform.
Self-Paced Schedule
No fixed schedule. Start when you want, finish at your own pace.
Knowledge Checks
Worked examples and per-unit practice quizzes with instant feedback, plus a final knowledge assessment on the e-learning platform (80% to pass).
Certificate Included
Your verifiable certificate for this course is included in the programme price, issued once you pass the final assessment.
Outline
What you'll learn
-
What Human Feedback Work Is
How preference data and ratings become a reward model.
-
Reading and Applying a Rubric
Why the rubric outranks personal taste, and how dimensions are ordered.
-
Comparing and Ranking Responses
How length, position, and confidence bias corrupt ratings.
-
Consistency and Integrity
What agreement scores, gold checks, drift, and fatigue mean in practice.
-
Judging Feedback Submissions
What reviewers check, and how self-review protects an agreement score.
Counts toward the Data Collection Foundations (DCF) certification programme.
How it works
From enrolling to certified
1
Learn at your own pace
Start the moment you enrol and work through the material in your own time. There is no cohort to wait for and no fixed schedule to keep up with.
2
Pass the assessment
The course ends with a graded assessment you take when you are ready. The result is what counts, not how long you took.
3
Receive your certificate
Issued within one week of passing, with a public verification page anyone can check.
Two ways in
This course, or the whole programme
Just this course
Human Feedback Foundations
One course, with its certificate when you pass.
at your own pace
Best value
The full Data Collection Foundations
All 8 courses, and the programme certificate at the end.
the courses one at a time
Frequently Asked Questions
Everything you need to know about this course.
Most learners finish in 45 minutes to 1.5 hours, self-paced. Each of the five units has reading and a practice quiz with instant feedback, and interactive exercises run throughout the course. The final knowledge assessment has an 80% pass mark. You can retake it as many times as you need, with 24 hours between attempts, and your highest score counts.
It's designed for beginners moving into evaluation work. No AI knowledge and no coding background are required, though existing expertise in another field is the biggest single advantage in this modality. Data Collection Basics is the recommended starting point, but this course doesn't assume it.
Reinforcement learning from human feedback. People compare model responses and say which is better; those comparisons become preference data; the preference data trains a reward model; and the reward model shapes the answers the assistant gives afterwards. Your ratings do not correct one answer; they nudge the behaviour of every answer that follows, which is why consistency matters so much.
Reading, interactive scenario exercises, and knowledge checks on the e-learning platform. It teaches the conventions of human feedback work: reading and applying a rubric, comparing and ranking responses, the biases that corrupt ratings, agreement scores and gold checks, and how reviewers decide. There is no supervised rating practice assessed by DeeLab and nothing to submit.
By agreement rather than by opinion. Platforms hide gold-standard items with known answers among the real ones, and compare each rater against other raters on the same task. The rubric is what makes that possible: it fixes which dimension decides when two answers are close, so two careful raters reach the same result. The course covers agreement scores, gold checks, drift, and fatigue, which is what that number is really measuring.
The other Data Collection Foundations courses are the natural next step; each is available on its own, or the programme fee covers them all. Conversational AI Data Foundations is the closest companion: that course is about writing responses, this one is about judging them, and many contributors do both. You can also record your completed courses, domain expertise, languages and availability in the freelance and gig work section of your trainee profile, which is what DeeLab looks at when a project needs contributors.
Yes. The certificate is included in the course fee and is issued automatically once you pass the final assessment and leave your course feedback, and in any case within a week of passing. Pass all eight Data Collection courses and the programme certificate, the Certified Data Collector award, follows automatically as well.
You have three months to complete the course, counted from the day you sign up. Within that window it is entirely self-paced and always open; your progress is saved automatically and you can return whenever suits you.