top of page
  • Writer's pictureHackers Realm

Facial Emotion Recognition using Python | Image Classification | CNN | Deep Learning

Updated: 3 days ago

Explore the realm of facial emotion recognition with Python! This tutorial delves into image classification, CNN, and deep learning techniques to decipher emotions from facial expressions. Learn to build an accurate model that can detect and classify emotions in real-time. Enhance your understanding of computer vision and dive into the fascinating world of deep learning. Unlock the power of facial emotion recognition with this comprehensive project tutorial. #FacialEmotionRecognition #Python #ImageClassification #CNN #DeepLearning #ComputerVision

Facial Emotion Recognition using CNN
Facial Emotion Recognition using CNN

In this project tutorial we will use Convolutional Neural Network (CNN) for image feature extraction and visualize the results with plot graphs.


You can watch the video-based tutorial with step by step explanation down below.


Dataset Information

The objective of the project is to detect facial expression using facial image dataset. Convolutional Neural Network is used to classify the images. The output class consists of 7 different types namely angry, disgust, fear, happy, neutral, sad, surprise.


Environment: kaggle


You may download the facial emotion dataset here


Import Modules

import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
import random
from tqdm.notebook import tqdm
warnings.filterwarnings('ignore')
%matplotlib inline

import tensorflow as tf
from tensorflow.keras.utils import to_categorical
from keras.preprocessing.image import load_img
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Dropout, Flatten, MaxPooling2D
  • pandas - used to perform data manipulation and analysis

  • numpy - used to perform a wide variety of mathematical operations on arrays

  • matplotlib - used for data visualization and graphical plotting

  • seaborn - built on top of matplotlib with similar functionalities

  • os - used to handle files using system commands

  • tqdm - progress bar decorator for iterators

  • warnings - to manipulate warnings details, filterwarnings('ignore') is to ignore the warnings thrown by the modules (gives clean results)

  • load_img - used for loading the image as numpy array

  • tensorflow – backend module for the use of Keras

  • Dense - single dimension linear layer

  • Dropout - used to add regularization to the data, avoiding over fitting & dropping out a fraction of the data

  • Activation - layer for the use of certain threshold

  • Flatten - convert a 2D array into a 1D array

  • Conv2D - convolutional layer in 2 dimension

  • MaxPooling2D - function to get the maximum pixel value to the next layer


Load the Dataset


Now we load the test data and the train data for processing

TRAIN_DIR = '../input/facial-expression-dataset/train/train/'
TEST_DIR = '../input/facial-expression-dataset/test/test/'

Now we define the directory for the images and with the corresponding labels

def load_dataset(directory):
    image_paths = []
    labels = []
    
    for label in os.listdir(directory):
        for filename in os.listdir(directory+label):
            image_path = os.path.join(directory, label, filename)
            image_paths.append(image_path)
            labels.append(label)
            
        print(label, "Completed")
    
    return image_paths, labels

Now we initialize the dataframe

## convert into dataframe
train = pd.DataFrame()
train['image'], train['label'] = load_dataset(TRAIN_DIR)
# shuffle the dataset
train = train.sample(frac=1).reset_index(drop=True)
train.head()

surprise Completed fear Completed angry Completed neutral Completed sad Completed disgust Completed happy Completed

Facial Expression Train Dataset
Facial Expression Train Dataset
  • Display of image paths with labels of the train dataset

  • Data was shuffled and the index was removed


test = pd.DataFrame()
test['image'], test['label'] = load_dataset(TEST_DIR)
test.head()

surprise Completed fear Completed angry Completed neutral Completed sad Completed disgust Completed happy Completed

Facial Expression Test Dataset
Facial Expression Test Dataset
  • Display of image paths with labels of the test dataset

  • It is not necessary to shuffle the test data


Exploratory Data Analysis


sns.countplot(train['label'])

Visualization of Label Count for the facial expression Dataset
Visualization of Label Count for the Dataset
  • Display of the no. of samples in the dataset

  • Happy has more data samples compared to other classes

  • The rest of the classes has a uniform distribution

  • Disgust has less samples for training



from PIL import Image
img = Image.open(train['image'][0])
plt.imshow(img, cmap='gray');
Sample facial expression image
Sample Image
  • The images are in grayscale, so color map is declared as gray

  • This is the first sample from the data, which is surprise

  • The resolution of the image is 48 x 48


Now we will display a grid of images to see various images at once

# to display grid of images
plt.figure(figsize=(20,20))
files = train.iloc[0:25]

for index, file, label in files.itertuples():
    plt.subplot(5, 5, index+1)
    img = load_img(file)
    img = np.array(img)
    plt.imshow(img)
    plt.title(label)
    plt.axis('off')
Sample Images with Different Facial Expressions
Sample Images with Different Facial Expressions
Sample Images with Different Facial Expressions
  • Display of 25 different images from different classes

  • Image size was reduced to minimize the blurriness of the images


Feature Extraction


Now we define the feature extraction method

def extract_features(images):
    features = []
    for image in tqdm(images):
        img = load_img(image, grayscale=True)
        img = np.array(img)
        features.append(img)
    features = np.array(features)
    features = features.reshape(len(features), 48, 48, 1)
    return features
  • Pixel features of the images are extracted

  • Images are converted into arrays and reshaped to the proper format

  • tqdm creates a loading bar to help track the conversion process of the images


Now we extract the features from both train and test

train_features = extract_features(train['image'])
test_features = extract_features(test['image'])

Now we normalize the images

## normalize the image
x_train = train_features/255.0
x_test = test_features/255.0
  • This converts the pixel value from 1 to 255 into a normalized range of 0 to 1

  • Normalization is useful for the neural network to easily capture the information


## convert label to integer
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
le.fit(train['label'])
y_train = le.transform(train['label'])
y_test = le.transform(test['label'])
  • Convert the data type of 'label' to integer for easier processing


y_train = to_categorical(y_train, num_classes=7)
y_test = to_categorical(y_test, num_classes=7) 
  • One hot encoding the data

  • The model works better if the data is one hot encoded


y_train[0]

array([0., 0., 0., 0., 0., 0., 1.], dtype=float32)


# config
input_shape = (48, 48, 1)
output_class = 7
  • input_shape = (48, 48, 1) - Converts the input image into 48 x 48 resolution in grayscale

  • output_class = 7 - Total no. of clases


Model Creation

model = Sequential()
# convolutional layers
model.add(Conv2D(128, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))

model.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))

model.add(Conv2D(512, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))

model.add(Conv2D(512, kernel_size=(3,3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))

model.add(Flatten())
# fully connected layers
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.3))
# output layer
model.add(Dense(output_class, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy')
  • Augmentation is not necessary since the images are small

  • optimizer=’adam’ - automatically adjust the learning rate for the model over the no. of epochs

  • activation='softmax' - used for multi-classification output

  • loss=’categorical_crossentropy’ - loss function for category outputs