Data Cleaning

and Why It Matters

• April 2026 • 5 min read

Garbage in, garbage out. It is one of the oldest rules in computing — and the most frequently ignored in data science.

Studies consistently show that data professionals spend 60–80% of their time not building models or dashboards, but cleaning data. Yet this unglamorous step is what separates reliable insights from dangerously wrong ones. A model trained on dirty data does not just underperform — it can actively mislead decision-makers, erode trust, and cause real financial or operational damage.

This blog breaks down what data cleaning actually involves, why it matters, and how to approach it systematically.

What Is 'Dirty' Data?

Dirty data refers to any data that is inaccurate, incomplete, inconsistent, or improperly formatted. It enters systems through human error, system migrations, multiple data sources merging, or simply the passage of time as real-world conditions change.

Why Data Cleaning Matters

Core Data Cleaning Techniques

1. Handling Missing Values

Not all missing data is equal. The right strategy depends on why data is missing:

• Mean / Median Imputation — for numerical columns with random missingness (e.g., fill missing age with median age)
• Mode Imputation — for categorical columns (e.g., fill missing city with the most frequent city)
• Forward / Backward Fill — for time-series data where the previous or next value is a reasonable estimate
• Model-Based Imputation — use a regression or KNN model to predict missing values from other features
• Drop Rows / Columns — when missing data exceeds ~20–30% and imputation would introduce too much noise

2. Removing Duplicates

Duplicates arise when data is merged from multiple sources, when users submit forms twice, or when ETL jobs run without deduplication checks. In Python:

3. Fixing Data Types

Type mismatches are among the most common issues in real-world datasets. A date stored as a string cannot be sorted or aggregated. A numeric ID stored as a float will cause merge failures. Always audit column dtypes as the first step of any data cleaning workflow.

4. Standardising Text & Categories

Inconsistent text categories silently fragment your data. 'Mumbai', 'mumbai', 'MUMBAI', and 'Bombay' may all refer to the same city — but a GROUP BY query will treat them as four distinct values, distorting any city-level aggregation.

• Convert to lowercase: df['city'] = df['city'].str.lower().str.strip()
• Use mapping dictionaries for known aliases: {'bombay': 'mumbai', 'madras': 'chennai'}
• Apply fuzzy matching (fuzzywuzzy / rapidfuzz) for large-scale standardisation

5. Handling Outliers

Outliers require context: they may be data entry errors (age = 999), legitimate extreme values (a billionaire's income), or signals worth investigating (a sudden 10x spike in API calls). Three common strategies:

• Z-Score Method — flag values more than 3 standard deviations from the mean
• IQR Method — flag values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR
• Domain Capping — clip values to a known valid range (e.g., age between 0 and 120)

A Practical Data Cleaning Workflow

Follow this sequence to clean any new dataset systematically:

1. Profile the data — check shape, dtypes, missing counts, and summary statistics first.
2. Resolve structural issues — fix column names, data types, and encoding problems.
3. Handle missing values — choose an imputation or drop strategy per column.
4. Remove duplicates — identify the correct key columns for deduplication.
5. Standardise categorical values — normalise text, units, and codes.
6. Detect and treat outliers — investigate before removing.
7. Validate — run constraint checks (e.g., no negative ages, all dates within valid range).
8. Document — record every transformation for reproducibility and audits.

Key Takeaways