What is High Cardinality?

High cardinality refers to having a large number of unique values in a dataset, particularly in the context of databases and time series data. Imagine you have a list of people's names. If every name is different, like John, Alice, Michael, and so on, you have high cardinality.

It often arises in sectors such as E-commerce, Healthcare, and Telecommunications, where extensive monitoring and event data are collected.

In simpler terms, high cardinality means having many different and unique pieces of information to manage and process within a dataset.

High Cardinality Example

Let's say we are collecting weather data. Each day, we record the temperature in various cities using different weather stations.

Now, imagine we want to find out the temperature in a specific city on a particular day. To make this process faster, we organize the data using metadata or tags. These could include the city name, the date, and the weather station ID.

Indexing comes into play here. It's like creating a roadmap that helps us quickly find the temperature readings we need.

However, if we have many cities, days, and weather stations, the number of unique combinations can become quite large.

For instance, if we have 100 cities, 365 days, and 50 weather stations, the total combinations would be 100 * 365 * 50 = 1,825,000. This is what we mean by high cardinality – lots of unique combinations.

Efficiently managing and querying this data becomes crucial to extract meaningful insights or answer specific questions, like what was the temperature in a particular city on a specific day.

Key Concepts in High Cardinality

Before getting into the understanding of time series data and related concepts, let's familiarize ourselves with some key terms.

Time Series Data: This is data collected over time, like names, weather readings, or website traffic.
Metadata or Tags: These are additional pieces of information that describe the data points in the time series. For example, for weather data, metadata might include location, temperature sensor ID, etc.
Indexing: This is like creating a roadmap for your data. It helps you quickly find specific data points based on certain criteria. Just like an index at the back of a book helps you find topics quickly.
Cardinality: This refers to the number of unique values in a dataset. In a time series dataset with metadata, the cardinality is determined by the number of unique values in each metadata category.

Strategies for Simplifying Data

High cardinality can make data hard for computers to deal with, which causes problems in analyzing data. But, there are some ways to fix this. If we use these ways, we can handle high cardinality data better and make better models.

1. Data Encoding Simplified

Data encoding is a way to change categories, like colors or countries, into numbers that computers can understand. There are different methods, like one-hot encoding or label encoding, depending on the type of category.

If there are too many categories, we can group them together or use special methods to make it easier for the computer to handle.

2. Data Transformation Made Easy

Data transformation is when we change numbers, like age or income, to make them easier to use in models. There are different reasons for doing this, like making the data more even or creating new features. We can use methods like log transformation or standardization to do this.

3. Dealing with High-Cardinality Categories

High-cardinality categories are ones with lots of different values, like product IDs or names. These can be hard for computers to handle, so we need to make them simpler. We can group them together based on how often they appear or how similar they are.

Popular Grouping

One way to make high-cardinality categories easier to handle is by grouping them based on how often they appear. For example, if we have a list of cities, we can group them into big cities, small cities, and unknown cities.

Similarity Sorting

Another way to make high-cardinality categories easier to handle is by grouping them based on how similar they are. For example, if we have a list of products, we can group them into categories like electronics, clothes, or food.

Grouping by Target

A third way to make high-cardinality categories easier to handle is by grouping them based on what we want to predict. For example, if we have a list of customers, we can group them based on how likely they are to buy something.

Thus, High-cardinality categorical features can pose challenges, but by employing effective grouping strategies, we can simplify and organize complex data, making it more manageable and suitable for machine learning algorithms.

One-Hot Encoding

One-hot encoding is closely related to the concept of high cardinality. High cardinality refers to a situation where a categorical feature has a large number of unique values.

For example, if you have a dataset of customers and one of the features is "Country", which has many unique values, you can use one-hot encoding to convert this feature into a set of binary variables, one for each country.

This allows you to include the "Country" feature in your machine learning model without having to worry about the high cardinality.

Let's consider the example of a dataset with a high cardinality categorical feature, such as a list of countries.

Sample dataset:

import pandas as pd

# Create a sample dataset
data = {'Country': ['USA', 'Canada', 'Germany', 'USA', 'Germany', 'Canada', 'Germany']}
df = pd.DataFrame(data)

One-hot encoding:

# Perform one-hot encoding
one_hot_encoded_df = pd.get_dummies(df, columns=['Country'])

print(one_hot_encoded_df)

Output:

As you can see, the "Country" column has been replaced with three new columns, "Country_Canada", "Country_Germany", and "Country_USA".

Each column represents one possible value of the original "Country" column. The value in each column is 1 if the country matches the column's country, and 0 otherwise.

So, for example, the first row in the original dataset has the value "USA" for the "Country" column, so the "Country_USA" column has a value of 1 and the other two columns have a value of 0. The second row has the value "Canada" for the "Country" column, so the "Country_Canada" column has a value of 1 and the other two columns have a value of 0. And so on.

This is a simple example, but one-hot encoding can be used with larger datasets and more complex categorical variables. It is a powerful tool for converting categorical data into a format that can be used in machine learning algorithms.

Factors and Challenges with High Cardinality Data

High cardinality data, characterized by attributes with numerous unique values relative to the total records, can pose challenges in analysis. Understanding these factors is crucial for effective data management and analysis.

Factors contributing to high cardinality include,

Diverse Categories: Attributes with a wide range of categories can lead to high cardinality. For example, a dataset containing information about products may have a "Product Name" attribute with a large number of unique values if there is a wide variety of products.
Unique Identifiers: Unique identifiers, such as customer IDs or hash values, are designed to be distinct for each entity. As a result, attributes containing these identifiers tend to have high cardinality.
Continuous Variables: Numeric attributes with a wide range of values, such as timestamps or sensor readings, can also contribute to high cardinality. For example, a dataset containing sensor data may have a "Timestamp" attribute with a large number of unique values if the data is collected at a high frequency.
High Dimensionality: In datasets with many attributes, the combination of different attributes can result in a large number of unique value combinations. This can lead to high cardinality in certain attributes.
Data Quality: Poor data quality, such as misspellings or inconsistencies, can increase the number of unique values in an attribute. For example, a dataset containing customer names may have a "Customer Name" attribute with a large number of unique values if there are misspellings or variations in the way names are entered.

Why is High Cardinality Critical for Observability?

High cardinality plays a crucial role in observability due to its ability to offer detailed analysis and differentiation within datasets. In monitoring systems, especially those operating in distributed environments like microservices or cloud-native architectures, high cardinality refers to the presence of a large number of unique values in the data.

Here's why high cardinality is vital for observability:

Enhanced Detail: High cardinality allows for a finer level of detail. For instance, in a microservices setup, where multiple instances of various services may run, each instance might have unique characteristics such as version numbers, deployment environments, or configurations. High cardinality enables tracking and analysis of each instance separately, providing detailed insights into system behavior.
Effective Problem Solving: When troubleshooting issues or investigating performance problems, high cardinality data helps narrow down investigations to specific instances, components, or even individual requests. This granularity is essential for pinpointing the root cause of problems and resolving them efficiently.
Correlation and Contextual Understanding: High cardinality data facilitates correlation between metrics and events across different dimensions. For example, it enables correlating CPU usage with user sessions or error rates with geographical regions. This contextual information aids in understanding how various factors interact and influence system behavior.
Flexibility and Adaptability: High cardinality data provides flexibility to adapt to changing requirements and evolving systems. As new services or dimensions are added, high cardinality allows capturing and analyzing data without being limited by predefined schemas or restrictions on unique values.
Anomaly Detection: High cardinality data improves anomaly detection by enabling the identification of outliers and unusual patterns at a granular level. This is particularly important in large-scale distributed systems where anomalies may manifest differently across various components and instances.

Conclusion

High cardinality is a concept that refers to datasets with a large number of unique values, often presenting challenges in data management and analysis. This situation is particularly relevant in fields like E-commerce, Healthcare, and Telecommunications, where extensive monitoring and event data are collected.

Understanding high cardinality involves grasping key concepts like time series data, metadata, indexing, and cardinality itself. Time series data is information collected over time, while metadata provides additional context. Indexing is like creating a roadmap for data, helping to quickly find specific data points. Cardinality refers to the number of unique values in a dataset.

High cardinality can pose challenges due to diverse categories, unique identifiers, continuous variables, high dimensionality, and data quality issues. To simplify high-cardinality data, strategies like data encoding, data transformation, and dealing with high-cardinality categories can be employed.

One-hot encoding is a method to convert categorical features with high cardinality into a format suitable for machine learning models. It represents each unique value as a binary variable. Data transformation involves altering numerical values to make them suitable for models. Grouping high-cardinality categories based on frequency, similarity, or target can also be useful.

Monitor Your Entire Application with Atatus

Atatus is a Full Stack Observability Platform that lets you review problems as if they happened in your application. Instead of guessing why errors happen or asking users for screenshots and log dumps, Atatus lets you replay the session to quickly understand what went wrong.

We offer Application Performance Monitoring, Real User Monitoring, Server Monitoring, Logs Monitoring, Synthetic Monitoring, Uptime Monitoring and API Analytics. It works perfectly with any application, regardless of framework, and has plugins.

Atatus can be beneficial to your business, which provides a comprehensive view of your application, including how it works, where performance bottlenecks exist, which users are most impacted, and which errors break your code for your frontend, backend, and infrastructure.

If you are not yet a Atatus customer, you can sign up for a 14-day free trial .

Table of Contents

What is Cardinality?