Exploratory Data Analysis Python and Pandas with Examples (2024)

This article is about Exploratory Data Analysis(EDA) in Pandas and Python. The article will explain step by step how to do Exploratory Data Analysis plus examples.

EDA is an important step in Data Science. The goal of EDA is to identify errors, insights, relations, outliers and more.

The image below illustrate the data science workflow and where EDA is located:

Source: Exploratory Data Analysis - wikipedia

Background Story

Imagine that you are expecting royal guests for dinner. You are asked to research a special menu from a cooking book with thousands of recipes. As they are very pretentious you need to avoid some ingredients or find exact quantities for others. Dinner and launch menus are needed.

Unfortunately some recipes are wrong and others are incomplete.

How to start? You may start with the main ingredients. Another option is by selecting the meal type and course. There should be a special menu for Vegetarians.

The story above reminds of Exploratory Data Analysis(EDA). All the steps and questions are part of understanding unknown territory. Let's try to solve this with Data Science.

What is Exploratory Data Analysis?

Exploratory data analysis is like detective work: searching for insights that identify problems and hidden patterns.

Start with one variable at a time, then explore two variables, and so on.

Definition
Exploratory Data Analysis(EDA) - analyze and investigate datasets and summarize their main characteristics and apply visualization methods.

Tools for EDA:

Most popular Tools:

• Pandas/Python
• R

Techniques for EDA:

Visual techniques

• Box plot
• Histogram
• Heat map
• Scatter plot

Dimensionality reduction

• Principal component analysis (PCA)

Exploratory Data Analysis - Example Project

Let's do an example on Exploratory Data Analysis using Food Recipes.

We will do step by step analysis on this data set and answer on questions like:

• What data do we have?
• What is the dimension of this data?
• Are there any dependent variables?
• What are the data types?
• Missing data?
• Duplicate data?
• Correlations?

You can find the notebook on and Google Colab:

• GitHub
• Google Colab - you can play with this notebook without installing Python

Step 1: Load Data and Initial Analysis

First we will import the needed libraries and then load the dataset into Pandas:

import pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltdf = pd.read_csv('https://raw.githubusercontent.com/softhints/Pandas-Exercises-Projects/main/data/food_recipes.csv', low_memory=False)

Step 2: Initial Analysis of Pandas DataFrame

We will check the data by using the following methods:

• df - returns first and last 5 records; returns number of rows and columns
• head(n) - returns first n rows
• tail(n) - returns last n rows
• sample(n) - sample random n rows

The first 2 rows transposed looks like:

We are lucky with:

df.sample(2)

as we can see missing values and non latin symbols:

The initial findings for this dataset are:

Insights

• dataset has header column
• separator is ';'
• NaN represents missing values
• 2 columns with nested data - ingredients and tags
  • if we take closer look we can find values divided by pipes - "|"
  • for further analysis - data need to be normalized
• there are non latin characters in this dataset

Let's continue with the analysis and not rely on luck or first impressions.

Step 3: Dimension and data types of DataFrame

Next we will use several Pandas methods in order to find information for the columns, dimensions, nested data and more:

df.shape

will return the dimension of this dataset:

(8009, 16)

We can get columns by:

df.columns

Index(['recipe_title', 'url', 'record_health', 'vote_count', 'rating', 'description', 'cuisine', 'course', 'diet', 'prep_time', 'cook_time', 'ingredients', 'instructions', 'author', 'tags', 'category'], dtype='object')

To get information about the data types we can use:

df.info()df.dtypes

which results into:

<class 'pandas.core.frame.DataFrame'>RangeIndex: 8009 entries, 0 to 8008Data columns (total 16 columns):Column Non-Null Count Dtype --- ------ -------------- ----- 0 recipe_title 8009 non-null object 1 url 8009 non-null object 2 record_health 8009 non-null object 3 vote_count 8009 non-null int64 4 rating 8009 non-null float64 5 description 7994 non-null object 6 cuisine 7943 non-null object 7 course 7854 non-null object 8 diet 7858 non-null object 9 prep_time 7979 non-null object 10 cook_time 7979 non-null object 11 ingredients 7997 non-null object 12 instructions 8009 non-null object 13 author 8009 non-null object 14 tags 7930 non-null object 15 category 8009 non-null objectdtypes: float64(1), int64(1), object(14)memory usage: 1001.2+ KB

The above gives use:

• 3 data types - float64(1), int64(1), object(14)
• memory usage: 1001.2+ KB
• information for the rows and the columns

By using the method df.shape() we can find the total number of rows and columns. Which has synonyms:

• row - observation, record, trial
• column - variable, feature, characteristic

Data insights from this step:

Insights:

• dataset contains - 8009 rows and 16 columns
  • 8009 observations
  • 16 characteristics
• 2 numeric columns
• 2 which need conversion - prep_time and cook_time
  • convert string to int
  • 15 M -> 15

Step 4: Statistics - min, max, mean, percentile

We can get useful statistics for numeric and non numeric columns of this DataFrame.

4.1. Descriptive statistics for numeric columns

For numeric columns we can use:

df.describe()

which returns stats like:

• mean
• standard deviation
• percentile
• count
• min and max

Insights:

• mean value is close to 75%
• no missing values
• big difference between max and 75% percentile for vote_count - potential outliers

4.2. Pandas describe() for non numeric columns

For the rest of the columns we can use include='object':

df.describe(include='object')

the result is:

Few columns seems to be more interesting than the rest:

Insights:

• 4 categorical columns - category, cuisine, diet and course
• 1 column with single value - record_health

Step 5: Inspect single column of DataFrame

df.diet.nunique()

returns the number of unique values in this column:

10

df.diet.unique()

will give us all unique values:

array(['Vegetarian', 'High Protein Vegetarian', 'Non Vegeterian', 'No Onion No Garlic (Sattvic)', 'High Protein Non Vegetarian', 'Diabetic Friendly', 'Eggetarian', 'Vegan', 'Gluten Free', nan, 'Sugar Free Diet'], dtype=object)

df.diet.value_counts(dropna=False)

returns the values and their frequency:

Vegetarian 5478High Protein Vegetarian 792Non Vegeterian 447Eggetarian 391Diabetic Friendly 296High Protein Non Vegetarian 225NaN 151No Onion No Garlic (Sattvic) 77Vegan 71Gluten Free 66Sugar Free Diet 15Name: diet, dtype: int64

Insights:

• 10 unique values
• categorical column
• Vegetarian - is the most frequent value
• 151 - missing values

Step 6: Data conversion in Pandas

Sometimes data needs to be processed for proper analysis. For example nested data to be expanded, string needs to be converted to number or date.

We'll see two examples of data processing required for further data analysis.

6.1 Convert String to numeric

To analyze columns cook_time and prep_time as numeric we can convert them by:

df['cook_time'] = df['cook_time'].str.replace(' M', '').fillna(0).astype(int)df['prep_time'] = df['prep_time'].str.replace(' M', '').fillna(0).astype(int)

Now we can apply visual techniques to analyze those columns for outliers - this is done in section 7.3 of this article.

6.2 Expand nested column

Next we will expand a column "ingredients" - in order to analyze ingredients for each recipe:

df['ingredients'].str.split('|', expand=True)

The result is new DataFrame with expanded values from the ingredient column:

from:

0 Tortillas|Extra Virgin Olive Oil|Garlic|Mozzar...1 Sesame (Gingelly) Oil|Mustard seeds (Rai/ Kadu...2 Extra Virgin Olive Oil|Pineapple|White onion|R...

to:

To find the most common 'ingredient' in the all recipes we can use the following steps:

• split the column on separator "|"
  • expand=True - will expand data to DataFrame
• melt() - transform DataFrame columns to rows - pairs of variable and value
• count the values by .value_counts()

df['ingredients'].str.split('|', expand=True).melt()['value'].value_counts()

we found that salt is the most popular ingredient:

Salt 5698Sunflower Oil 2241Garlic 2227Turmeric powder (Haldi) 2221Onion 2012 ...नारीयल पानी 1नारीयल का दूध 1सिंघाड़े का आटा 1Black currant (dried) 1Corn chips 1Name: value, Length: 1823, dtype: int64

Step 7: Visualization for Data Analysis

Finally lets cover a few visualization techniques to find missing values, outliers and correlations.

7.1 Missing values

The first visualization technique is about finding missing values in Pandas DataFrame.

Use library Seaborn with heat map to plot missing values for the whole dataset:

• df.isna() - returns new DataFrame with True/False values depending on the missing values in a cell
• cmap = 'Greens' - is a color map. For more info check: Get color palette with matplotlib

sns.heatmap(df.isna(),cmap = 'Greens')

result is:

Next let's check the same visualization for the last 3000 rows:

sns.heatmap(df.tail(3000).isna(),cmap = 'Greens')

result:

From the result above we can conclude:

Insights:

• data set has missing values in several columns
  • represented by darker green lines
• cuisine, course and diet
  • plot suggest pattern for missing values of those columns

7.2 Correlations

Again will use Seaborn's heatmap in order to visualize correlations between two and more columns:

sns.heatmap(df.corr(),cmap='Greens',annot=False)

The first option is for the whole dataset shows - no correlation between the columns.

sns.heatmap(df[df.diet=='Vegan'].corr(),cmap='Greens',annot=True)

While the second one is showing correlations for rows with type "Vegan". We can see a correlation between columns 'prep_time' and 'cook_time'.

Insights:

• no correlation of numerical columns for the whole DataFrame
• correlation for - Vegan recipes - between prep_time and cook_time
• darker color represent positive correlation
  • lighter - negative
• to get values use annot=True

7.3 Detect outliers in Pandas DataFrame

Finally we will use boxplot to detect outliers:

cols = ['vote_count', 'rating', 'prep_time', 'cook_time']plt.figure(figsize=(20, 5))sns.boxplot(data=df[cols], orient='h')

The picture below suggest about outliers in the numeric columns

Insights:

• vote_count - suggest outliers
• cook_time - has possible outliers
• rating - doesn't indicate outliers

7.4 Explain Seaborn boxplot and outliers

In order to get a better understanding we will work only with two columns: 'prep_time', 'cook_time' and the first 18 records.

The stats for them are:

df[['prep_time', 'cook_time']].head(18).describe()

The picture below illustrates how to analyze a dataset for outliers with boxplot.

Boxplot has 3 main components:

• box (Interquartile Range - IQR)
  • q1 - 25%
  • q2 - 50% or median
  • q3 - 75%
• whiskers
  • min
  • max
• outliers
  • points beyond the whiskers

We can notice that sometimes the whiskers don't match the min/max. They are calculated based on formulas. Upper and low whiskers are calculated as 1.5 multiplied on the interquartile range - IQR(the box).

Definition
An outlier(extreme) is a value that lies in the extremes of a data Series(on either end). They can affect overall observation and insights.

In the next article we will cover more techniques and several Python libraries for exploratory data analysis.

Conclusion

In this article, we covered the basics of Exploratory Data Analysis and discussed most of the common methods which every data scientist should know while working with datasets. Through practical examples, we learned how to explore new datasets.

We started by initial data analysis and first glance of data. After that, we made data conversion and preprocessing for further analysis. Finally, we saw multiple visual techniques in order to find missing values, outliers and correlations.