Akansha YadavData Science & Coding (Python) | MySQL

3 hacks for high cardinality data #ai #coding

30 Jul 2026 · 2 min

Watch on YouTube

1:35

In this tutorial, Akansha Yadav explains why one-hot encoding is often unsuitable for high‑cardinality categorical features and presents five practical alternatives. She first outlines the problem: one-hot encoding creates a large number of sparse columns, leading to inefficiency and over‑fitting. She then walks through each hack: (1) Frequency encoding, which replaces each category with its occurrence count; (2) Target encoding, which substitutes categories with the mean of the target variable; (3) Hash encoding, which hashes categories into a fixed number of columns to limit dimensionality; (4) Entity (or embedding) encoding, where dense vector representations are learned, especially useful in deep‑learning models; and (5) Dropping identifier columns that serve only as unique keys. For each method she describes the basic operation, why it mitigates high cardinality issues, and hints at appropriate use cases. The video ends with a call for viewers to share which technique they prefer.

Chapters

So we know that high cardinality is not good for our data, right? So we have a better way to handle the high cardinalities as well. So instead of using one-hot encoding, generally what students used to do is they go for one-hot encoding. Why? Because we have to convert a categorical data into numeric. So one-hot encoding feels ideal for us, right? But that's not true. Have multiple options as well. One is the frequency encoding. So in frequency encoding, what do we do is we replace each category with its occurrence, right? So if let's say male is— the male is repeating, let's say, 30 times and female is repeated 40 times. So instead of using zeros and one, we go for 30 and 40 as a value, and that becomes more within the range for any dataset to be in. Then next is the target encoding. So in this target encoding, what do we do is we replace the categories with the mean value of the target variable. Whatever the target variable is, like what are you trying to predict from that model, according to the target you can replace it with. Then the next is hash encoding. What is hash encoding? We basically hash the categories into fixed number of columns. We mention that, okay, I want 5 columns, 6 columns, 10 columns. Because if, let's say, if any column is having 100 of high card, like 100 of unique values, so will it create 100 of new columns for us? Obviously that is going to create another chaos for us, right, within the data. So instead we will fix the number of columns. Then comes the entity embedding, where we learn the dense vector representation, and we generally use this entity embedding in the deep learning projects that we create. The next is drop identifier column. It means whichever column is as an identifier for that particular table, you can drop it for that instance and can proceed ahead with the other columns. So there are many other ways by which you can handle high cardinality. So which one do you use? Do let me know in comments.