Facial Emotion Recognition using Python | Image Classification | CNN | Deep Learning
Updated: 3 days ago
Explore the realm of facial emotion recognition with Python! This tutorial delves into image classification, CNN, and deep learning techniques to decipher emotions from facial expressions. Learn to build an accurate model that can detect and classify emotions in real-time. Enhance your understanding of computer vision and dive into the fascinating world of deep learning. Unlock the power of facial emotion recognition with this comprehensive project tutorial. #FacialEmotionRecognition #Python #ImageClassification #CNN #DeepLearning #ComputerVision
In this project tutorial we will use Convolutional Neural Network (CNN) for image feature extraction and visualize the results with plot graphs.
You can watch the video-based tutorial with step by step explanation down below.
Dataset Information
The objective of the project is to detect facial expression using facial image dataset. Convolutional Neural Network is used to classify the images. The output class consists of 7 different types namely angry, disgust, fear, happy, neutral, sad, surprise.
Environment: kaggle
You may download the facial emotion dataset here
Import Modules
import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
import random
from tqdm.notebook import tqdm
warnings.filterwarnings('ignore')
%matplotlib inline
import tensorflow as tf
from tensorflow.keras.utils import to_categorical
from keras.preprocessing.image import load_img
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Dropout, Flatten, MaxPooling2Dpandas - used to perform data manipulation and analysis
numpy - used to perform a wide variety of mathematical operations on arrays
matplotlib - used for data visualization and graphical plotting
seaborn - built on top of matplotlib with similar functionalities
os - used to handle files using system commands
tqdm - progress bar decorator for iterators
warnings - to manipulate warnings details, filterwarnings('ignore') is to ignore the warnings thrown by the modules (gives clean results)
load_img - used for loading the image as numpy array
tensorflow – backend module for the use of Keras
Dense - single dimension linear layer
Dropout - used to add regularization to the data, avoiding over fitting & dropping out a fraction of the data
Activation - layer for the use of certain threshold
Flatten - convert a 2D array into a 1D array
Conv2D - convolutional layer in 2 dimension
MaxPooling2D - function to get the maximum pixel value to the next layer
Load the Dataset
Now we load the test data and the train data for processing
TRAIN_DIR = '../input/facial-expression-dataset/train/train/'
TEST_DIR = '../input/facial-expression-dataset/test/test/'Now we define the directory for the images and with the corresponding labels
def load_dataset(directory):
image_paths = []
labels = []
for label in os.listdir(directory):
for filename in os.listdir(directory+label):
image_path = os.path.join(directory, label, filename)
image_paths.append(image_path)
labels.append(label)
print(label, "Completed")
return image_paths, labelsNow we initialize the dataframe
## convert into dataframe
train = pd.DataFrame()
train['image'], train['label'] = load_dataset(TRAIN_DIR)
# shuffle the dataset
train = train.sample(frac=1).reset_index(drop=True)
train.head()surprise Completed fear Completed angry Completed neutral Completed sad Completed disgust Completed happy Completed
Display of image paths with labels of the train dataset
Data was shuffled and the index was removed
test = pd.DataFrame()
test['image'], test['label'] = load_dataset(TEST_DIR)
test.head()surprise Completed fear Completed angry Completed neutral Completed sad Completed disgust Completed happy Completed
Display of image paths with labels of the test dataset
It is not necessary to shuffle the test data
Exploratory Data Analysis
sns.countplot(train['label'])
Display of the no. of samples in the dataset
Happy has more data samples compared to other classes
The rest of the classes has a uniform distribution
Disgust has less samples for training
from PIL import Image
img = Image.open(train['image'][0])
plt.imshow(img, cmap='gray');
The images are in grayscale, so color map is declared as gray
This is the first sample from the data, which is surprise
The resolution of the image is 48 x 48
Now we will display a grid of images to see various images at once
# to display grid of images
plt.figure(figsize=(20,20))
files = train.iloc[0:25]
for index, file, label in files.itertuples():
plt.subplot(5, 5, index+1)
img = load_img(file)
img = np.array(img)
plt.imshow(img)
plt.title(label)
plt.axis('off')
Display of 25 different images from different classes
Image size was reduced to minimize the blurriness of the images
Feature Extraction
Now we define the feature extraction method
def extract_features(images):
features = []
for image in tqdm(images):
img = load_img(image, grayscale=True)
img = np.array(img)
features.append(img)
features = np.array(features)
features = features.reshape(len(features), 48, 48, 1)
return featuresPixel features of the images are extracted
Images are converted into arrays and reshaped to the proper format
tqdm creates a loading bar to help track the conversion process of the images
Now we extract the features from both train and test
train_features = extract_features(train['image'])test_features = extract_features(test['image'])Now we normalize the images
## normalize the image
x_train = train_features/255.0
x_test = test_features/255.0This converts the pixel value from 1 to 255 into a normalized range of 0 to 1
Normalization is useful for the neural network to easily capture the information
## convert label to integer
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
le.fit(train['label'])
y_train = le.transform(train['label'])
y_test = le.transform(test['label'])Convert the data type of 'label' to integer for easier processing
y_train = to_categorical(y_train, num_classes=7)
y_test = to_categorical(y_test, num_classes=7) One hot encoding the data
The model works better if the data is one hot encoded
y_train[0]array([0., 0., 0., 0., 0., 0., 1.], dtype=float32)
# config
input_shape = (48, 48, 1)
output_class = 7input_shape = (48, 48, 1) - Converts the input image into 48 x 48 resolution in grayscale
output_class = 7 - Total no. of clases
Model Creation
model = Sequential()
# convolutional layers
model.add(Conv2D(128, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
model.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
model.add(Conv2D(512, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
model.add(Conv2D(512, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
model.add(Flatten())
# fully connected layers
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.3))
# output layer
model.add(Dense(output_class, activation='softmax'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy')Augmentation is not necessary since the images are small
optimizer=’adam’ - automatically adjust the learning rate for the model over the no. of epochs
activation='softmax' - used for multi-classification output
loss=’categorical_crossentropy’ - loss function for category outputs