The 5 Types of Data Annotation Every AI Team Should Know

Data annotation is not one thing. It's a family of techniques, each suited to different data types and model objectives. Understanding the landscape before you start a project saves time, money, and a great deal of re-labelling.

1. Image and video annotation

The most visually intuitive form of annotation. Subcategories include:

• Bounding boxes — rectangular regions drawn around objects of interest. Fast, scalable, and suitable for object detection models.
• Polygon and instance segmentation — precise outlines following the true shape of an object. Slower and more expensive, but necessary for models that need to reason about exact object boundaries.
• Semantic segmentation — every pixel in an image is assigned a class label. Used in autonomous driving, medical imaging, and satellite analysis.
• Keypoint annotation — specific points on objects (e.g. joint positions on a human body) used for pose estimation and motion capture.
• Video tracking — annotations that follow objects across frames, producing temporal consistency for video understanding models.

Related article Image Annotation: How Humans Teach Machines to See Image annotation turns ordinary pixels into training data machines can learn from. Here's what it is, where it's used, the skills and tools it takes — and how t... Read more →

Related article Video Labeling: Teaching Machines to Understand Motion Video labeling teaches AI to track objects, mark event boundaries, and recognise actions across a timeline. It underpins autonomous vehicles, sports analytics,... Read more →

2. Text annotation

Text annotation is the backbone of NLP and large language model training. Common tasks include:

• Named entity recognition (NER) — tagging entities like people, organisations, and locations within text.
• Sentiment classification — labelling text as positive, negative, or neutral (or more granular emotional categories).
• Intent classification — assigning user queries to categories for dialogue systems and search.
• Coreference resolution — linking pronouns and references to the entities they describe.
• RLHF preference labelling — ranking or comparing model outputs to provide human feedback signal for reinforcement learning.

Related article Text Labeling: Teaching AI to Read Every chatbot reply, spam filter and translated sentence rests on text that a human once labelled. Text labeling marks entities, sentiment, intent and structure... Read more →

3. Audio and speech annotation

Audio annotation supports voice assistants, speech recognition, and audio classification. Key types:

• Transcription — converting speech to text, with or without speaker diarisation.
• Phonetic alignment — mapping audio segments to phoneme-level labels for pronunciation models.
• Sound event detection — labelling ambient sounds (traffic, speech, music) for environmental audio models.

Related article Audio Labeling: How Machines Learn to Listen To a computer, sound is just numbers; audio labelling is how humans add the meaning machines learn from. Annotators segment recordings, transcribe speech and ta... Read more →

4. 3D point cloud annotation

LiDAR-generated point clouds are the primary data source for autonomous vehicles and robotics. Annotation in 3D space requires specialist tools and annotators, and typically includes 3D bounding boxes, track IDs, and surface classifications.

Related article Point Cloud Labeling: A Beginner's Guide to Annotating the 3D World Point cloud labeling is one of the fastest-growing skills in data annotation — powering self-driving cars, robotics, and 3D mapping. This beginner's guide expla... Read more →

5. Multimodal annotation

Increasingly, modern AI models ingest multiple modalities simultaneously. A visual question answering (VQA) model requires image annotation and text annotation that are semantically aligned. Autonomous vehicles fuse LiDAR, camera, and radar data — all of which must be annotated consistently across the same coordinate space.

Choosing the right type for your project

The right annotation type is determined by the model architecture and the task it must perform, not by what's easiest to collect. Before specifying your annotation approach, work backwards from the model's inference requirements: what does it need to output, and what input representation best supports that output?

If you're unsure, DeeLab Academy's enterprise consulting team can help you design an annotation specification that matches your model architecture before a single item is labelled.