5 minute read

What Kind of Data Can You Annotate?

Data Labeling and the Right data to annotate.

Understand Labeling for Each Data Type and Which to Choose for Your Machine Learning Project

Data labeling is essential for supervised learning. The more labeled data you provide your machine learning model to help it understand what it’s “seeing,” the more accurate your model outputs will be. Ahead, learn about the types of data you can use for machine learning projects as well as considerations for choosing each type for data labeling.

Types of Data That Can Be Labeled

Image data is essential for computer vision models. When labeling an image, Images are scanned for discrete objects or figures. For an autonomous vehicle application, labeled objects might include telephone poles, vehicles, and road signs. For a customer behavior analysis project, labeled objects might include facial expressions, body postures, and body movements. In addition to objects and figures, different sections of an image might be labeled in order to provide a machine learning algorithm with “big picture” information. This might be whether a scene is captured at day or night and which sections of the scene might include multiple objects or are part of the sky and ground.


Video data is annotated in much the same way that image data is annotated. The difference between image and video data is simply that videos comprise a sequence of images called frames. If your video records at 30 frames per second (fps), then there are 30 individual frames, or still images, that can be labeled in a second of video footage. Most videos are recorded and displayed at 24fps according to modern cinema standards, and most social media platforms recommend uploading videos with a maximum of 30fps rate. Videos may require thousands or even millions of hours of annotation—a single 5-minute video could be 9,000 frames!


Lidar (light detection and ranging) technology produces dense 3D representations of ground landscapes for computer vision applications. Lidar data might include objects such as trees, bodies of water, and other landscape features. For example, for an agricultural use case where an environmental governing body wants to track the eutrophication of bodies of water, Lidar data can track the height, color, and texture of the water to get more sophisticated data on the scope of the pollution issue.


Text data is used in natural language processing (NLP). This might entail entity recognition, which identifies and tags certain words or phrases as common nouns, proper nouns, numeric values, and locations. Text for data labeling can include documents, such as receipts that are scanned for billing information, or customer reviews that identify objects they highlighted as a part of the service or sale. NLP can also be used to classify sentiments, which, for example, may use a review to determine if a customer was positive, neutral, or negative about their service.


Audio data is another data type that is used in natural language processing by “listening” to speech and identifies questions, commands, music, or ambient sounds. Audio data is annotated much in the same way that both language and video data are annotated, by selecting words, phrases, sentences, or types of sounds from a sequence of audio and labeling them into categories.

Determining Which Data Format to Use for Machine Learning

Determining which data type to use for a machine learning project is often a question of what infrastructure is already in place. For example, a warehouse or nursing center may use video cameras to record workers to track health and safety anomalies in real time. Despite the costly nature of video annotation, video will be the go-to data type to use for these machine learning projects since video data is readily available for analysis. 

When determining how to generate data without having pre-existing data collection infrastructure in place, is it useful to think about the goal of the machine learning model. If a car insurance company is assessing damage claims for vehicle accidents, the data that needs to be automatically analyzed is customer-generated images. For this project, still images showing car damage can be annotated and fed into a computer vision model to generate predictions for future claims based off customer-provided images.

In the case of Lidar data, this data type is very dense and costly to annotate. As a result, it may be better to record with and annotate 2D video for the same application. In the previous example of a eutrophication computer vision model, using video data would trade out the accuracy of Lidar—which can capture more information about a scene using 3D data—for the 2D data of water color, texture, and a less accurate prediction of water level.

Working with Data Labeling Experts

It can be useful to partner with a data labeling service to advise your company or organization on which type of data will be most effective to annotate for your machine learning project. Alegion offers Fully Managed Services as well as an industry-leading Labeling Platform to address high-volume labeling needs across any industry. When it comes to labeling busy images, videos, audio, and text, our data experts are ready to take a deep dive into a tailored solution for your unique use case. Request a consultation with a data expert to learn more and understand how Alegion builds better data to power better AI. 

Learn More About Our Annotation Solutions