Diego Alzate

🤖 Proyectos de IA

• Natural Language Processing
• Computer Vision
• Deep Learning
• Supervised Learning
• Unsupervised Learning
• Ensemble Techniques

📍 Navegación

• Inicio
• Proyectos

Introduction to Neural Networks: Bank Churn prediction

Problem Statement

Context

Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.

Objective

You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.

Data Dictionary:

• CustomerId: Unique ID which is assigned to each customer

• Surname: Last name of the customer

• CreditScore: It defines the credit history of the customer.

• Geography: A customer’s location

• Gender: It defines the Gender of the customer

• Age: Age of the customer

• Tenure: Number of years for which the customer has been with the bank

• NumOfProducts: refers to the number of products that a customer has purchased through the bank.

• Balance: Account balance

• HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.

• EstimatedSalary: Estimated salary

• isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )

• Exited : whether or not the customer left the bank within six month. It can take two values

** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )

Importing necessary libraries

from tensorflow.keras.layers import Dense, Dropout, InputLayer
from sklearn.metrics import classification_report, confusion_matrix
from tensorflow.keras import optimizers
from tensorflow.keras.models import Sequential
from tensorflow.keras import backend
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve
from sklearn import metrics
from sklearn.model_selection import GridSearchCV
from imblearn.over_sampling import SMOTE
from keras import callbacks
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
import warnings
warnings.filterwarnings("ignore")

Loading the dataset

from google.colab import drive
drive.mount('/content/drive/')
original_data = pd.read_csv("/content/drive/My Drive/5-DeepLearning/Project/Churn.csv")
data = original_data.copy()

Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount("/content/drive/", force_remount=True).

# Remove unnecessary columns 'RowNumber', 'CustomerId', and 'Surname'
data.drop(['RowNumber', 'CustomerId', 'Surname'], axis=1, inplace=True)

Data Overview

data.head()

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

data.tail()

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
9995	771	France	Male	39	5	0.00	2	1	0	96270.64	0
9996	516	France	Male	35	10	57369.61	1	1	1	101699.77	0
9997	709	France	Female	36	7	0.00	1	0	1	42085.58	1
9998	772	Germany	Male	42	3	75075.31	2	1	0	92888.52	1
9999	792	France	Female	28	4	130142.79	1	1	0	38190.78	0

data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   CreditScore      10000 non-null  int64  
 1   Geography        10000 non-null  object 
 2   Gender           10000 non-null  object 
 3   Age              10000 non-null  int64  
 4   Tenure           10000 non-null  int64  
 5   Balance          10000 non-null  float64
 6   NumOfProducts    10000 non-null  int64  
 7   HasCrCard        10000 non-null  int64  
 8   IsActiveMember   10000 non-null  int64  
 9   EstimatedSalary  10000 non-null  float64
 10  Exited           10000 non-null  int64  
dtypes: float64(2), int64(7), object(2)
memory usage: 859.5+ KB

data.nunique()

CreditScore         460
Geography             3
Gender                2
Age                  70
Tenure               11
Balance            6382
NumOfProducts         4
HasCrCard             2
IsActiveMember        2
EstimatedSalary    9999
Exited                2
dtype: int64

data.shape

(10000, 11)

data.describe().T

	count	mean	std	min	25%	50%	75%	max
CreditScore	10000.0	650.528800	96.653299	350.00	584.00	652.000	718.0000	850.00
Age	10000.0	38.921800	10.487806	18.00	32.00	37.000	44.0000	92.00
Tenure	10000.0	5.012800	2.892174	0.00	3.00	5.000	7.0000	10.00
Balance	10000.0	76485.889288	62397.405202	0.00	0.00	97198.540	127644.2400	250898.09
NumOfProducts	10000.0	1.530200	0.581654	1.00	1.00	1.000	2.0000	4.00
HasCrCard	10000.0	0.705500	0.455840	0.00	0.00	1.000	1.0000	1.00
IsActiveMember	10000.0	0.515100	0.499797	0.00	0.00	1.000	1.0000	1.00
EstimatedSalary	10000.0	100090.239881	57510.492818	11.58	51002.11	100193.915	149388.2475	199992.48
Exited	10000.0	0.203700	0.402769	0.00	0.00	0.000	0.0000	1.00

CreditScore:

• The mean credit score is approximately 650.53, with a standard deviation of 96.65.
• Credit scores range from a minimum of 350 to a maximum of 850.
• The distribution appears to be approximately symmetric, with a median (50th percentile) of 652.

Age:

• The average age of customers in the dataset is around 38.92 years, with a standard deviation of 10.49.
• Customer ages range from a minimum of 18 years to a maximum of 92 years.
• The age distribution seems to be slightly right-skewed, with the median (50th percentile) at 37 years.

Tenure:

• On average, customers have a tenure of approximately 5.01 years, with a standard deviation of 2.89.
• Tenure varies from a minimum of 0 years (possibly indicating new customers) to a maximum of 10 years.
• The distribution shows that most customers have tenures between 3 and 7 years.

Balance:

• The mean balance is quite high at approximately 76,485.89, but with a substantial standard deviation of 62,397.41.
• Some customers have zero balance, while others have balances exceeding 250,000.
• The distribution seems to be right-skewed, with a median balance of 97,198.54.

NumOfProducts:

• Customers typically have 1.53 products on average, with a standard deviation of 0.58.
• The number of products per customer ranges from a minimum of 1 to a maximum of 4.
• The majority of customers have 1 or 2 products.

HasCrCard:

• About 70.55% of customers have a credit card (1), while the rest do not (0).
• This column is binary, indicating whether a customer has a credit card.

IsActiveMember:

• Approximately 51.51% of customers are active members (1), while the remaining 48.49% are not (0).
• This column represents whether a customer is an active member.

EstimatedSalary:

• The average estimated salary is approximately 100,090.24, with a standard deviation of 57,510.49.
• Estimated salaries range from a minimum of 11.58 to a maximum of 199,992.48.
• The distribution appears to be relatively uniform.

Exited:

• The "Exited" column is binary, indicating whether a customer has exited the service.
• About 20.37% of customers have exited (1), while the majority (79.63%) have not (0).

Exploratory Data Analysis

• EDA is an important part of any project involving data.
• It is important to investigate and understand the data better before building a model with it.
• A few questions have been mentioned below which will help you approach the analysis in the right manner and generate insights from the data.
• A thorough analysis of the data, in addition to the questions mentioned below, should be done.

Questions:

1. What is the distribution of the credit score of customers? Are there any noticeable patterns or outliers in the distribution?
2. How many active members are there with the bank?
3. How are the different customer attributes correlated to each other?
4. Who is churning more when compared to males and females?
5. Customers from which geographical part are churning more?

# function to plot a boxplot and a histogram along the same scale.


def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to the show density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a triangle will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram

# function to create labeled barplots


def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot

def bar_chart(x_name, y_name, title, labels, data):
    sns.set(style="darkgrid")
    plt.figure(figsize=(8, 6))
    sns.countplot(x=x_name, hue=y_name, data=data)
    plt.title(title)
    plt.xlabel(x_name)
    plt.ylabel("Count")
    plt.legend(title=y_name, labels=labels)
    plt.show()

1. What is the distribution of the credit score of customers? Are there any noticeable patterns or outliers in the distribution?

# histogram_box plot
histogram_boxplot(data, "CreditScore")

Observations

It seems to be a left-skewed distribution.

2. How many active members are there with the bank?

# labeled bar plot
labeled_barplot(data, "IsActiveMember", perc=True)

data['IsActiveMember'].eq(1).sum()

5151

Observations

Active members: 5151

3. How are the different customer attributes correlated to each other?

columnas_numericas = data.select_dtypes(include=['number']).columns
plt.figure(figsize=(15, 7))
sns.heatmap(data[columnas_numericas].corr(), annot=True, vmin=-1,
            vmax=1, fmt=".2f", cmap="Spectral")
plt.show()

CreditScore vs. Age: There is a very weak negative correlation (-0.003965) between CreditScore and Age. This suggests that, on average, older customers do not necessarily have lower credit scores.

Age vs. Tenure: Age and Tenure have a weak negative correlation (-0.009997), implying that older customers tend to have slightly shorter tenures with the bank.

Balance vs. NumOfProducts: There is a significant negative correlation (-0.304180) between Balance and NumOfProducts, indicating that customers with more products tend to maintain lower account balances.

Age vs. IsActiveMember: Age and IsActiveMember have a positive correlation (0.085472), suggesting that older customers are slightly more likely to be active members.

NumOfProducts vs. Exited: NumOfProducts and Exited have a negative correlation (-0.047820), indicating that customers with more products are slightly less likely to exit the bank.

IsActiveMember vs. Exited: There is a moderate negative correlation (-0.156128) between IsActiveMember and Exited, suggesting that active members are less likely to churn.

Age vs. Exited: Age and Exited have a positive correlation (0.285323), indicating that older customers are more likely to exit the bank. This could be due to various factors, such as retirement or changing financial needs.

EstimatedSalary vs. Exited: There is a very weak positive correlation (0.012097) between EstimatedSalary and Exited, suggesting that there is no strong relationship between salary and customer churn.

CreditScore vs. Exited: CreditScore and Exited have a negative correlation (-0.027094), indicating that customers with lower credit scores are slightly more likely to exit the bank.

4. Who is churning more when compared to males and females?

bar_chart("Gender", "Exited", "Churn Comparison Between Males and Females", [
          "No Churn", "Churn"], data)

Observations

Females churn more than males

5. Customers from which geographical part are churning more?

bar_chart("Geography", "Exited", "Churn Comparison by Geography",
          ["No Churn", "Churn"], data)

Observations

Germany and France have similar churn values but both are above Spain

Univariate analysis

# histogram_box plot for all the numerical values
for column in columnas_numericas:
    histogram_boxplot(data, column)

Bivariate Analysis

sns.pairplot(data=data, diag_kind="kde")

<seaborn.axisgrid.PairGrid at 0x7bc9f2235e70>

Data Preprocessing

• Missing value treatment
• Feature engineering (if needed)
• Outlier detection and treatment (if needed)
• Preparing data for modeling
• Any other preprocessing steps (if needed)

# Duplicate value check
data.duplicated().sum()

0

Observations

There are no duplicated values

# Checking missing values
data.isnull().sum()

CreditScore        0
Geography          0
Gender             0
Age                0
Tenure             0
Balance            0
NumOfProducts      0
HasCrCard          0
IsActiveMember     0
EstimatedSalary    0
Exited             0
dtype: int64

# Checking missing values
data.isna().sum()

CreditScore        0
Geography          0
Gender             0
Age                0
Tenure             0
Balance            0
NumOfProducts      0
HasCrCard          0
IsActiveMember     0
EstimatedSalary    0
Exited             0
dtype: int64

Observations

There are no missing values

Outliers

threshold = 3
outlier = {}
columnas_numericas = data.select_dtypes(include=['number']).columns
for col in columnas_numericas:
    i = data[col]
    mean = np.mean(data[col])
    std = np.std(data[col])
    list1 = []
    for v in i:
        z = (v - mean) / std
        if z > threshold:
            if v not in list1:
                list1.append(v)
    list1.sort()
    outlier[i.name] = list1

print("The following are the outliers in the data:")
for key, value in outlier.items():
    print("\n", key, ":", value)

The following are the outliers in the data:

 CreditScore : []

 Age : [71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 88, 92]

 Tenure : []

 Balance : []

 NumOfProducts : [4]

 HasCrCard : []

 IsActiveMember : []

 EstimatedSalary : []

 Exited : []

Outlier Treatment

# functions to treat outliers by flooring and capping


def treat_outliers(df, col):
    """
    Treats outliers in a variable

    df: dataframe
    col: dataframe column
    """
    Q1 = df[col].quantile(0.25)  # 25th quantile
    Q3 = df[col].quantile(0.75)  # 75th quantile
    IQR = Q3 - Q1
    Lower_Whisker = Q1 - 1.5 * IQR
    Upper_Whisker = Q3 + 1.5 * IQR

    # all the values smaller than Lower_Whisker will be assigned the value of Lower_Whisker
    # all the values greater than Upper_Whisker will be assigned the value of Upper_Whisker
    df[col] = np.clip(df[col], Lower_Whisker, Upper_Whisker)

    return df


def treat_outliers_all(df, col_list):
    """
    Treat outliers in a list of variables

    df: dataframe
    col_list: list of dataframe columns
    """
    for c in col_list:
        df = treat_outliers(df, c)

    return df

new_data = treat_outliers_all(data, ['Age'])

Checking the outliers

threshold = 3
outlier = {}
columnas_numericas = new_data.select_dtypes(include=['number']).columns
for col in columnas_numericas:
    i = new_data[col]
    mean = np.mean(new_data[col])
    std = np.std(new_data[col])
    list1 = []
    for v in i:
        z = (v - mean) / std
        if z > threshold:
            if v not in list1:
                list1.append(v)
    list1.sort()
    outlier[i.name] = list1

print("The following are the outliers in the data:")
for key, value in outlier.items():
    print("\n", key, ":", value)

The following are the outliers in the data:

 CreditScore : []

 Age : []

 Tenure : []

 Balance : []

 NumOfProducts : [4]

 HasCrCard : []

 IsActiveMember : []

 EstimatedSalary : []

 Exited : []

Preparing data for modeling

# getting the features and labels and finally splitting the test and train data.
X = new_data.drop('Exited', axis=1)
Y = new_data['Exited']

# Splitting the dataset into the Training and Testing set.
X_big, X_test, y_big, y_test = train_test_split(
    X, Y, test_size=0.1, random_state=5, shuffle=True)

# Splitting the dataset into the Training and validation set.
X_train, X_val, y_train, y_val = train_test_split(
    X_big, y_big, test_size=0.3, random_state=5, shuffle=True)

X_train.head()

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
6222	582	France	Male	31	2	0.00	2	1	1	33747.03
4085	520	France	Male	34	4	134007.90	1	1	1	193209.11
6328	790	Spain	Female	41	7	109508.68	1	0	0	86776.38
2594	537	Germany	Female	47	6	103163.35	1	1	0	16259.64
9987	606	Spain	Male	30	8	180307.73	2	1	1	1914.41

# Scaling the data
columns_to_scale = ['CreditScore', 'Age', 'Tenure',
                    'Balance', 'NumOfProducts', 'EstimatedSalary']
sc = StandardScaler()
temp = sc.fit(X_train[columns_to_scale])
X_train[columns_to_scale] = temp.transform(X_train[columns_to_scale])
X_test[columns_to_scale] = temp.transform(X_test[columns_to_scale])
X_val[columns_to_scale] = temp.transform(X_val[columns_to_scale])

# Using pd.get dummies to encode the categorical columns in the data
cat_data = ['Geography', 'Gender']
X_train = pd.get_dummies(X_train, columns=cat_data, drop_first=True)
X_test = pd.get_dummies(X_test, columns=cat_data, drop_first=True)
X_val = pd.get_dummies(X_val, columns=cat_data, drop_first=True)

# Checking the shape
print(X_train.shape)
print(X_val.shape)
print(X_test.shape)
print(y_train.shape)
print(y_val.shape)
print(y_test.shape)

(6300, 11)
(2700, 11)
(1000, 11)
(6300,)
(2700,)
(1000,)

# Checking the head()
print('X_train \n')
print(X_train.head())
print('\n y_train \n')
print(y_train.head())

X_train 

      CreditScore       Age    Tenure   Balance  NumOfProducts  HasCrCard  \
6222    -0.705093 -0.786348 -1.038141 -1.230161       0.823097          1   
4085    -1.346339 -0.476666 -0.346669  0.924816      -0.902531          1   
6328     1.446183  0.245927  0.690539  0.530844      -0.902531          0   
2594    -1.170513  0.865293  0.344803  0.428805      -0.902531          1   
9987    -0.456869 -0.889576  1.036275  1.669362       0.823097          1   

      IsActiveMember  EstimatedSalary  Geography_Germany  Geography_Spain  \
6222               1        -1.158081                  0                0   
4085               1         1.620116                  0                0   
6328               0        -0.234188                  0                1   
2594               0        -1.462752                  1                0   
9987               1        -1.712679                  0                1   

      Gender_Male  
6222            1  
4085            1  
6328            0  
2594            0  
9987            1  

 y_train 

6222    0
4085    0
6328    0
2594    1
9987    0
Name: Exited, dtype: int64

Model Building

Model Evaluation Criterion

It's important to consider the potential consequences of making wrong predictions. Two common prediction errors are:

False Positive: Predicting that a customer is exiting (Class 1) when they are not actually exiting (Class 0).

False Negative: Predicting that a customer is not exiting (Class 0) when they are actually exiting (Class 1).

Among these two errors, the more important one for the bank to minimize is the False Negative. Here's why:

Importance of Minimizing False Negatives:

Predicting that a customer is not exiting when they are actually leaving the bank can have significant financial consequences. In such cases, the bank may miss the opportunity to take proactive measures to retain those customers. These customers might be sensitive or dissatisfied for various reasons, and if their issues are not addressed promptly, they could leave, resulting in revenue loss for the bank.

Reducing False Negatives (Increasing Recall):

To reduce the risk of false negatives and increase Recall, which is the proportion of actual positives (churn cases) correctly predicted by the model, the bank can consider the following strategies:

Threshold Adjustment: By adjusting the classification threshold for the churn prediction model, you can increase sensitivity (recall) at the cost of specificity (precision). Lowering the threshold will classify more cases as positive (Class 1), which can help capture more true positives.

Feature Engineering: Carefully select and engineer features that are strong indicators of potential churn. Feature selection and extraction techniques can help improve the model's ability to identify at-risk customers.

Data Augmentation: Augment the dataset with additional data if possible. More data can help the model better capture the complexities of customer behavior and churn patterns.

Balancing the Dataset: If the dataset is imbalanced (i.e., there are far more non-churn cases than churn cases), consider techniques like oversampling the minority class (churn) or using synthetic data generation methods to balance the classes.

In summary, while both false positives and false negatives have their implications, in the context of customer churn prediction for a bank, minimizing false negatives (increasing Recall) is typically more critical. This ensures that the bank can proactively retain customers who are actually at risk of leaving, ultimately reducing revenue loss and maintaining customer loyalty.

Model Building: Neural Network

def make_confusion_matrix(cf,
                          group_names=None,
                          categories='auto',
                          count=True,
                          percent=True,
                          cbar=True,
                          xyticks=True,
                          xyplotlabels=True,
                          sum_stats=True,
                          figsize=None,
                          cmap='Blues',
                          title=None):
    '''
    This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
    Arguments
    '''

    # CODE TO GENERATE TEXT INSIDE EACH SQUARE
    blanks = ['' for i in range(cf.size)]

    if group_names and len(group_names) == cf.size:
        group_labels = ["{}\n".format(value) for value in group_names]
    else:
        group_labels = blanks

    if count:
        group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
    else:
        group_counts = blanks

    if percent:
        group_percentages = ["{0:.2%}".format(
            value) for value in cf.flatten()/np.sum(cf)]
    else:
        group_percentages = blanks

    box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(
        group_labels, group_counts, group_percentages)]
    box_labels = np.asarray(box_labels).reshape(cf.shape[0], cf.shape[1])

    # CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
    if sum_stats:
        # Accuracy is sum of diagonal divided by total observations
        accuracy = np.trace(cf) / float(np.sum(cf))

        # if it is a binary confusion matrix, show some more stats
        if len(cf) == 2:
            # Metrics for Binary Confusion Matrices
            precision = cf[1, 1] / sum(cf[:, 1])
            recall = cf[1, 1] / sum(cf[1, :])
            f1_score = 2*precision*recall / (precision + recall)
            stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
                accuracy, precision, recall, f1_score)
        else:
            stats_text = "\n\nAccuracy={:0.3f}".format(accuracy)
    else:
        stats_text = ""

    # SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
    if figsize == None:
        # Get default figure size if not set
        figsize = plt.rcParams.get('figure.figsize')

    if xyticks == False:
        # Do not show categories if xyticks is False
        categories = False

    # MAKE THE HEATMAP VISUALIZATION
    plt.figure(figsize=figsize)
    sns.heatmap(cf, annot=box_labels, fmt="", cmap=cmap, cbar=cbar,
                xticklabels=categories, yticklabels=categories)

    if xyplotlabels:
        plt.ylabel('True label')
        plt.xlabel('Predicted label' + stats_text)
    else:
        plt.xlabel(stats_text)

    if title:
        plt.title(title)

def graphs_and_report(history, predicted):
    # Plot training & validation loss values
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('Model loss')
    plt.ylabel('Loss')
    plt.xlabel('Epoch')
    plt.legend(['Train', 'Test'], loc='upper left')
    plt.show()

    # Plot training & validation accuracy values
    plt.plot(history.history['accuracy'])
    plt.plot(history.history['val_accuracy'])
    plt.title('Model accuracy')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['Train', 'Test'], loc='upper left')
    plt.show()

Model 1

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

# Initializing the neural network model using the Sequential API
model1 = Sequential()

# Adding the input layer with 11 neurons and ReLU activation function
model1.add(Dense(11, activation='relu',
                 kernel_initializer='he_uniform', input_shape=(11,)))

# Adding the first hidden layer with 64 neurons and ReLU activation function
model1.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))

# Adding the second hidden layer with 32 neurons and ReLU activation function
model1.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))

# Adding the output layer with one neuron and sigmoid activation function
model1.add(Dense(1, activation='sigmoid'))

# Setting Stochastic Gradient Descent (SGD) as the optimizer with a learning rate of 1e-3
sgd = optimizers.SGD(learning_rate=1e-3)

# Compiling the model with binary cross-entropy loss and accuracy as the evaluation metric
model1.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])

# Printing a summary of the model architecture
model1.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 11)                132       
                                                                 
 dense_1 (Dense)             (None, 64)                768       
                                                                 
 dense_2 (Dense)             (None, 32)                2080      
                                                                 
 dense_3 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 3,013
Trainable params: 3,013
Non-trainable params: 0
_________________________________________________________________

# Training model1 on the training data (X_train, y_train) with validation data (X_val, y_val)
# for 100 epochs with a batch size of 32. The training history is stored in the 'history1' variable.
history1 = model1.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, batch_size=32)

Epoch 1/100
197/197 [==============================] - 3s 8ms/step - loss: 0.7272 - accuracy: 0.5530 - val_loss: 0.5709 - val_accuracy: 0.7563
Epoch 2/100
197/197 [==============================] - 1s 6ms/step - loss: 0.5405 - accuracy: 0.7795 - val_loss: 0.5268 - val_accuracy: 0.7848
Epoch 3/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5156 - accuracy: 0.7933 - val_loss: 0.5154 - val_accuracy: 0.7915
Epoch 4/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5059 - accuracy: 0.7943 - val_loss: 0.5088 - val_accuracy: 0.7933
Epoch 5/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4992 - accuracy: 0.7949 - val_loss: 0.5033 - val_accuracy: 0.7948
Epoch 6/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4935 - accuracy: 0.7944 - val_loss: 0.4983 - val_accuracy: 0.7941
Epoch 7/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4885 - accuracy: 0.7952 - val_loss: 0.4940 - val_accuracy: 0.7937
Epoch 8/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4840 - accuracy: 0.7948 - val_loss: 0.4901 - val_accuracy: 0.7937
Epoch 9/100
197/197 [==============================] - 2s 9ms/step - loss: 0.4800 - accuracy: 0.7949 - val_loss: 0.4866 - val_accuracy: 0.7941
Epoch 10/100
197/197 [==============================] - 2s 12ms/step - loss: 0.4764 - accuracy: 0.7948 - val_loss: 0.4836 - val_accuracy: 0.7937
Epoch 11/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4732 - accuracy: 0.7949 - val_loss: 0.4806 - val_accuracy: 0.7937
Epoch 12/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4702 - accuracy: 0.7956 - val_loss: 0.4780 - val_accuracy: 0.7944
Epoch 13/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4674 - accuracy: 0.7960 - val_loss: 0.4756 - val_accuracy: 0.7959
Epoch 14/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4650 - accuracy: 0.7968 - val_loss: 0.4732 - val_accuracy: 0.7967
Epoch 15/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4626 - accuracy: 0.7987 - val_loss: 0.4711 - val_accuracy: 0.7967
Epoch 16/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4604 - accuracy: 0.7992 - val_loss: 0.4690 - val_accuracy: 0.7967
Epoch 17/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4584 - accuracy: 0.8006 - val_loss: 0.4672 - val_accuracy: 0.7974
Epoch 18/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4565 - accuracy: 0.8016 - val_loss: 0.4656 - val_accuracy: 0.7981
Epoch 19/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4547 - accuracy: 0.8022 - val_loss: 0.4639 - val_accuracy: 0.7985
Epoch 20/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4530 - accuracy: 0.8035 - val_loss: 0.4624 - val_accuracy: 0.7981
Epoch 21/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4514 - accuracy: 0.8049 - val_loss: 0.4610 - val_accuracy: 0.7985
Epoch 22/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4499 - accuracy: 0.8059 - val_loss: 0.4595 - val_accuracy: 0.7978
Epoch 23/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4484 - accuracy: 0.8062 - val_loss: 0.4580 - val_accuracy: 0.7981
Epoch 24/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4470 - accuracy: 0.8060 - val_loss: 0.4569 - val_accuracy: 0.7970
Epoch 25/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4457 - accuracy: 0.8073 - val_loss: 0.4555 - val_accuracy: 0.7970
Epoch 26/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4444 - accuracy: 0.8076 - val_loss: 0.4542 - val_accuracy: 0.7967
Epoch 27/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4432 - accuracy: 0.8073 - val_loss: 0.4531 - val_accuracy: 0.7967
Epoch 28/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4419 - accuracy: 0.8075 - val_loss: 0.4519 - val_accuracy: 0.7970
Epoch 29/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4408 - accuracy: 0.8078 - val_loss: 0.4507 - val_accuracy: 0.7967
Epoch 30/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4396 - accuracy: 0.8094 - val_loss: 0.4496 - val_accuracy: 0.7956
Epoch 31/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4386 - accuracy: 0.8098 - val_loss: 0.4485 - val_accuracy: 0.7944
Epoch 32/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4375 - accuracy: 0.8103 - val_loss: 0.4474 - val_accuracy: 0.7944
Epoch 33/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4365 - accuracy: 0.8105 - val_loss: 0.4464 - val_accuracy: 0.7944
Epoch 34/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4355 - accuracy: 0.8108 - val_loss: 0.4454 - val_accuracy: 0.7937
Epoch 35/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4345 - accuracy: 0.8121 - val_loss: 0.4445 - val_accuracy: 0.7933
Epoch 36/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4336 - accuracy: 0.8121 - val_loss: 0.4434 - val_accuracy: 0.7933
Epoch 37/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4326 - accuracy: 0.8127 - val_loss: 0.4427 - val_accuracy: 0.7948
Epoch 38/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4318 - accuracy: 0.8129 - val_loss: 0.4417 - val_accuracy: 0.7956
Epoch 39/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4309 - accuracy: 0.8129 - val_loss: 0.4406 - val_accuracy: 0.7985
Epoch 40/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4301 - accuracy: 0.8141 - val_loss: 0.4398 - val_accuracy: 0.7993
Epoch 41/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4292 - accuracy: 0.8135 - val_loss: 0.4389 - val_accuracy: 0.8000
Epoch 42/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4284 - accuracy: 0.8138 - val_loss: 0.4380 - val_accuracy: 0.8004
Epoch 43/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4276 - accuracy: 0.8144 - val_loss: 0.4373 - val_accuracy: 0.7993
Epoch 44/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4268 - accuracy: 0.8140 - val_loss: 0.4365 - val_accuracy: 0.8000
Epoch 45/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4260 - accuracy: 0.8135 - val_loss: 0.4356 - val_accuracy: 0.8004
Epoch 46/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4252 - accuracy: 0.8143 - val_loss: 0.4347 - val_accuracy: 0.8011
Epoch 47/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4245 - accuracy: 0.8156 - val_loss: 0.4340 - val_accuracy: 0.8015
Epoch 48/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4238 - accuracy: 0.8159 - val_loss: 0.4332 - val_accuracy: 0.8022
Epoch 49/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4230 - accuracy: 0.8167 - val_loss: 0.4326 - val_accuracy: 0.8026
Epoch 50/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4223 - accuracy: 0.8162 - val_loss: 0.4318 - val_accuracy: 0.8030
Epoch 51/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4216 - accuracy: 0.8163 - val_loss: 0.4310 - val_accuracy: 0.8052
Epoch 52/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4209 - accuracy: 0.8167 - val_loss: 0.4303 - val_accuracy: 0.8052
Epoch 53/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4202 - accuracy: 0.8184 - val_loss: 0.4297 - val_accuracy: 0.8048
Epoch 54/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4196 - accuracy: 0.8181 - val_loss: 0.4291 - val_accuracy: 0.8059
Epoch 55/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4189 - accuracy: 0.8181 - val_loss: 0.4283 - val_accuracy: 0.8059
Epoch 56/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4183 - accuracy: 0.8190 - val_loss: 0.4276 - val_accuracy: 0.8059
Epoch 57/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4176 - accuracy: 0.8183 - val_loss: 0.4269 - val_accuracy: 0.8052
Epoch 58/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4171 - accuracy: 0.8183 - val_loss: 0.4263 - val_accuracy: 0.8067
Epoch 59/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4165 - accuracy: 0.8179 - val_loss: 0.4258 - val_accuracy: 0.8067
Epoch 60/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4159 - accuracy: 0.8181 - val_loss: 0.4251 - val_accuracy: 0.8070
Epoch 61/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4153 - accuracy: 0.8178 - val_loss: 0.4246 - val_accuracy: 0.8074
Epoch 62/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4147 - accuracy: 0.8183 - val_loss: 0.4240 - val_accuracy: 0.8070
Epoch 63/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4141 - accuracy: 0.8194 - val_loss: 0.4234 - val_accuracy: 0.8067
Epoch 64/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4135 - accuracy: 0.8181 - val_loss: 0.4227 - val_accuracy: 0.8067
Epoch 65/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4130 - accuracy: 0.8197 - val_loss: 0.4221 - val_accuracy: 0.8070
Epoch 66/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4124 - accuracy: 0.8195 - val_loss: 0.4216 - val_accuracy: 0.8074
Epoch 67/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4119 - accuracy: 0.8205 - val_loss: 0.4211 - val_accuracy: 0.8078
Epoch 68/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4113 - accuracy: 0.8203 - val_loss: 0.4205 - val_accuracy: 0.8081
Epoch 69/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4108 - accuracy: 0.8202 - val_loss: 0.4201 - val_accuracy: 0.8089
Epoch 70/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4102 - accuracy: 0.8203 - val_loss: 0.4195 - val_accuracy: 0.8100
Epoch 71/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4097 - accuracy: 0.8203 - val_loss: 0.4191 - val_accuracy: 0.8107
Epoch 72/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4092 - accuracy: 0.8211 - val_loss: 0.4185 - val_accuracy: 0.8107
Epoch 73/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4087 - accuracy: 0.8219 - val_loss: 0.4180 - val_accuracy: 0.8122
Epoch 74/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4082 - accuracy: 0.8219 - val_loss: 0.4174 - val_accuracy: 0.8126
Epoch 75/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4076 - accuracy: 0.8222 - val_loss: 0.4170 - val_accuracy: 0.8130
Epoch 76/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4071 - accuracy: 0.8222 - val_loss: 0.4165 - val_accuracy: 0.8141
Epoch 77/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4066 - accuracy: 0.8227 - val_loss: 0.4160 - val_accuracy: 0.8141
Epoch 78/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4061 - accuracy: 0.8232 - val_loss: 0.4155 - val_accuracy: 0.8137
Epoch 79/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4056 - accuracy: 0.8233 - val_loss: 0.4149 - val_accuracy: 0.8159
Epoch 80/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4051 - accuracy: 0.8243 - val_loss: 0.4145 - val_accuracy: 0.8156
Epoch 81/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4046 - accuracy: 0.8238 - val_loss: 0.4140 - val_accuracy: 0.8170
Epoch 82/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4041 - accuracy: 0.8251 - val_loss: 0.4134 - val_accuracy: 0.8178
Epoch 83/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4037 - accuracy: 0.8244 - val_loss: 0.4131 - val_accuracy: 0.8174
Epoch 84/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4032 - accuracy: 0.8251 - val_loss: 0.4126 - val_accuracy: 0.8174
Epoch 85/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4027 - accuracy: 0.8251 - val_loss: 0.4121 - val_accuracy: 0.8178
Epoch 86/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4022 - accuracy: 0.8252 - val_loss: 0.4116 - val_accuracy: 0.8174
Epoch 87/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4018 - accuracy: 0.8262 - val_loss: 0.4113 - val_accuracy: 0.8178
Epoch 88/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4013 - accuracy: 0.8262 - val_loss: 0.4108 - val_accuracy: 0.8181
Epoch 89/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4008 - accuracy: 0.8260 - val_loss: 0.4102 - val_accuracy: 0.8185
Epoch 90/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4004 - accuracy: 0.8263 - val_loss: 0.4098 - val_accuracy: 0.8185
Epoch 91/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3999 - accuracy: 0.8267 - val_loss: 0.4094 - val_accuracy: 0.8193
Epoch 92/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3994 - accuracy: 0.8268 - val_loss: 0.4088 - val_accuracy: 0.8185
Epoch 93/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3990 - accuracy: 0.8267 - val_loss: 0.4084 - val_accuracy: 0.8196
Epoch 94/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3985 - accuracy: 0.8270 - val_loss: 0.4081 - val_accuracy: 0.8200
Epoch 95/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3981 - accuracy: 0.8271 - val_loss: 0.4076 - val_accuracy: 0.8204
Epoch 96/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3976 - accuracy: 0.8276 - val_loss: 0.4072 - val_accuracy: 0.8211
Epoch 97/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3972 - accuracy: 0.8279 - val_loss: 0.4067 - val_accuracy: 0.8211
Epoch 98/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3967 - accuracy: 0.8283 - val_loss: 0.4063 - val_accuracy: 0.8215
Epoch 99/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3963 - accuracy: 0.8284 - val_loss: 0.4059 - val_accuracy: 0.8211
Epoch 100/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3958 - accuracy: 0.8292 - val_loss: 0.4055 - val_accuracy: 0.8215

# Predicting target values (y_pred1) using 'model1' on validation data (X_val).
y_pred1 = model1.predict(X_val)


# Let's predict using default threshold
y_pred1 = (y_pred1 > 0.5)

graphs_and_report(history1, y_pred1)

85/85 [==============================] - 0s 2ms/step

# Classification report
cr1 = metrics.classification_report(y_val, y_pred1,digits=3)
print(cr1)

              precision    recall  f1-score   support

           0      0.839     0.959     0.895      2147
           1      0.645     0.286     0.396       553

    accuracy                          0.821      2700
   macro avg      0.742     0.623     0.646      2700
weighted avg      0.799     0.821     0.793      2700

# Confusion matrix
cm1 = confusion_matrix(y_val, y_pred1)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm1,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Observations

Precision (0): 0.839

• Precision measures the accuracy of positive predictions for class 0. In this case, it indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.959

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.895

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.645

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.286

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.396

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Model 2

def best_threshold(model):
    # predict probabilities
    yhat = model.predict(X_train)
    # keep probabilities for the positive outcome only
    yhat = yhat[:, 0]
    # calculate roc curves
    fpr, tpr, thresholds = roc_curve(y_train, yhat)
    # calculate the g-mean for each threshold
    gmeans = np.sqrt(tpr * (1-fpr))
    # locate the index of the largest g-mean
    ix = np.argmax(gmeans)
    print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix]))
    # plot the roc curve for the model
    plt.plot([0, 1], [0, 1], linestyle='--', label='No Skill')
    plt.plot(fpr, tpr, marker='.', label='Logistic')
    plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
    # axis labels
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.legend()
    return thresholds[ix]
    # show the plot
    plt.show()

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

# Initializing the neural network model using the Sequential API
model2 = Sequential()

# Adding the input layer with 64 neurons and ReLU activation function
model2.add(Dense(64, activation='relu',
                 kernel_initializer='he_uniform', input_shape=(11,)))

# Adding the second hidden layer with 32 neurons and ReLU activation function
model2.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))

# Adding the output layer with one neuron and sigmoid activation function
model2.add(Dense(1, activation='sigmoid'))

# Setting Stochastic Gradient Descent (SGD) as the optimizer with a learning rate of 1e-3
sgd = optimizers.SGD(learning_rate=1e-3)

# Compiling the model with binary cross-entropy loss and accuracy as the evaluation metric
model2.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])

# Printing a summary of the model architecture
model2.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 64)                768       
                                                                 
 dense_1 (Dense)             (None, 32)                2080      
                                                                 
 dense_2 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 2,881
Trainable params: 2,881
Non-trainable params: 0
_________________________________________________________________

# Training 'model2' on the training data (X_train, y_train) with validation data (X_val, y_val)
# for 100 epochs with a batch size of 32. The training history is stored in the 'history2' variable.
history2 = model2.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, batch_size=32)

Epoch 1/100
197/197 [==============================] - 2s 6ms/step - loss: 0.6530 - accuracy: 0.6527 - val_loss: 0.6188 - val_accuracy: 0.6922
Epoch 2/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5890 - accuracy: 0.7230 - val_loss: 0.5788 - val_accuracy: 0.7381
Epoch 3/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5588 - accuracy: 0.7559 - val_loss: 0.5544 - val_accuracy: 0.7667
Epoch 4/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5390 - accuracy: 0.7740 - val_loss: 0.5364 - val_accuracy: 0.7759
Epoch 5/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5241 - accuracy: 0.7835 - val_loss: 0.5220 - val_accuracy: 0.7841
Epoch 6/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5121 - accuracy: 0.7871 - val_loss: 0.5101 - val_accuracy: 0.7889
Epoch 7/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5022 - accuracy: 0.7917 - val_loss: 0.5001 - val_accuracy: 0.7893
Epoch 8/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4938 - accuracy: 0.7924 - val_loss: 0.4916 - val_accuracy: 0.7889
Epoch 9/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4867 - accuracy: 0.7937 - val_loss: 0.4843 - val_accuracy: 0.7904
Epoch 10/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4807 - accuracy: 0.7946 - val_loss: 0.4780 - val_accuracy: 0.7926
Epoch 11/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4754 - accuracy: 0.7938 - val_loss: 0.4725 - val_accuracy: 0.7926
Epoch 12/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4708 - accuracy: 0.7943 - val_loss: 0.4677 - val_accuracy: 0.7952
Epoch 13/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4667 - accuracy: 0.7954 - val_loss: 0.4634 - val_accuracy: 0.7974
Epoch 14/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4631 - accuracy: 0.7976 - val_loss: 0.4597 - val_accuracy: 0.7985
Epoch 15/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4600 - accuracy: 0.7990 - val_loss: 0.4563 - val_accuracy: 0.8004
Epoch 16/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4571 - accuracy: 0.8017 - val_loss: 0.4533 - val_accuracy: 0.8033
Epoch 17/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4546 - accuracy: 0.8024 - val_loss: 0.4506 - val_accuracy: 0.8022
Epoch 18/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4522 - accuracy: 0.8040 - val_loss: 0.4481 - val_accuracy: 0.8048
Epoch 19/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4501 - accuracy: 0.8054 - val_loss: 0.4459 - val_accuracy: 0.8059
Epoch 20/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4481 - accuracy: 0.8054 - val_loss: 0.4438 - val_accuracy: 0.8067
Epoch 21/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4462 - accuracy: 0.8049 - val_loss: 0.4419 - val_accuracy: 0.8081
Epoch 22/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4445 - accuracy: 0.8049 - val_loss: 0.4401 - val_accuracy: 0.8093
Epoch 23/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4429 - accuracy: 0.8051 - val_loss: 0.4385 - val_accuracy: 0.8081
Epoch 24/100
197/197 [==============================] - 2s 10ms/step - loss: 0.4413 - accuracy: 0.8060 - val_loss: 0.4369 - val_accuracy: 0.8096
Epoch 25/100
197/197 [==============================] - 2s 11ms/step - loss: 0.4399 - accuracy: 0.8070 - val_loss: 0.4355 - val_accuracy: 0.8104
Epoch 26/100
197/197 [==============================] - 2s 12ms/step - loss: 0.4385 - accuracy: 0.8078 - val_loss: 0.4341 - val_accuracy: 0.8107
Epoch 27/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4372 - accuracy: 0.8070 - val_loss: 0.4328 - val_accuracy: 0.8122
Epoch 28/100
197/197 [==============================] - 2s 9ms/step - loss: 0.4359 - accuracy: 0.8075 - val_loss: 0.4315 - val_accuracy: 0.8126
Epoch 29/100
197/197 [==============================] - 2s 10ms/step - loss: 0.4347 - accuracy: 0.8089 - val_loss: 0.4303 - val_accuracy: 0.8133
Epoch 30/100
197/197 [==============================] - 2s 9ms/step - loss: 0.4335 - accuracy: 0.8089 - val_loss: 0.4291 - val_accuracy: 0.8137
Epoch 31/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4323 - accuracy: 0.8097 - val_loss: 0.4281 - val_accuracy: 0.8141
Epoch 32/100
197/197 [==============================] - 2s 11ms/step - loss: 0.4312 - accuracy: 0.8098 - val_loss: 0.4270 - val_accuracy: 0.8148
Epoch 33/100
197/197 [==============================] - 2s 12ms/step - loss: 0.4302 - accuracy: 0.8103 - val_loss: 0.4260 - val_accuracy: 0.8152
Epoch 34/100
197/197 [==============================] - 2s 11ms/step - loss: 0.4291 - accuracy: 0.8114 - val_loss: 0.4250 - val_accuracy: 0.8156
Epoch 35/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4281 - accuracy: 0.8135 - val_loss: 0.4240 - val_accuracy: 0.8156
Epoch 36/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4271 - accuracy: 0.8140 - val_loss: 0.4231 - val_accuracy: 0.8159
Epoch 37/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4262 - accuracy: 0.8143 - val_loss: 0.4222 - val_accuracy: 0.8159
Epoch 38/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4252 - accuracy: 0.8144 - val_loss: 0.4214 - val_accuracy: 0.8170
Epoch 39/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4243 - accuracy: 0.8148 - val_loss: 0.4205 - val_accuracy: 0.8174
Epoch 40/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4234 - accuracy: 0.8163 - val_loss: 0.4197 - val_accuracy: 0.8178
Epoch 41/100
197/197 [==============================] - 2s 9ms/step - loss: 0.4225 - accuracy: 0.8165 - val_loss: 0.4189 - val_accuracy: 0.8181
Epoch 42/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4216 - accuracy: 0.8170 - val_loss: 0.4181 - val_accuracy: 0.8181
Epoch 43/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4208 - accuracy: 0.8167 - val_loss: 0.4173 - val_accuracy: 0.8185
Epoch 44/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4199 - accuracy: 0.8176 - val_loss: 0.4166 - val_accuracy: 0.8185
Epoch 45/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4191 - accuracy: 0.8181 - val_loss: 0.4158 - val_accuracy: 0.8181
Epoch 46/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4183 - accuracy: 0.8190 - val_loss: 0.4151 - val_accuracy: 0.8181
Epoch 47/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4175 - accuracy: 0.8189 - val_loss: 0.4144 - val_accuracy: 0.8189
Epoch 48/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4167 - accuracy: 0.8195 - val_loss: 0.4137 - val_accuracy: 0.8200
Epoch 49/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4159 - accuracy: 0.8203 - val_loss: 0.4130 - val_accuracy: 0.8204
Epoch 50/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4151 - accuracy: 0.8206 - val_loss: 0.4123 - val_accuracy: 0.8230
Epoch 51/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4144 - accuracy: 0.8216 - val_loss: 0.4117 - val_accuracy: 0.8241
Epoch 52/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4136 - accuracy: 0.8225 - val_loss: 0.4110 - val_accuracy: 0.8252
Epoch 53/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4129 - accuracy: 0.8227 - val_loss: 0.4104 - val_accuracy: 0.8252
Epoch 54/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4121 - accuracy: 0.8227 - val_loss: 0.4098 - val_accuracy: 0.8256
Epoch 55/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4115 - accuracy: 0.8238 - val_loss: 0.4091 - val_accuracy: 0.8256
Epoch 56/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4107 - accuracy: 0.8241 - val_loss: 0.4085 - val_accuracy: 0.8259
Epoch 57/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4100 - accuracy: 0.8252 - val_loss: 0.4079 - val_accuracy: 0.8259
Epoch 58/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4094 - accuracy: 0.8249 - val_loss: 0.4073 - val_accuracy: 0.8270
Epoch 59/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4087 - accuracy: 0.8251 - val_loss: 0.4067 - val_accuracy: 0.8278
Epoch 60/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4080 - accuracy: 0.8263 - val_loss: 0.4062 - val_accuracy: 0.8293
Epoch 61/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4073 - accuracy: 0.8262 - val_loss: 0.4056 - val_accuracy: 0.8300
Epoch 62/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4067 - accuracy: 0.8275 - val_loss: 0.4051 - val_accuracy: 0.8300
Epoch 63/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4060 - accuracy: 0.8275 - val_loss: 0.4045 - val_accuracy: 0.8296
Epoch 64/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4053 - accuracy: 0.8275 - val_loss: 0.4039 - val_accuracy: 0.8307
Epoch 65/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4047 - accuracy: 0.8281 - val_loss: 0.4034 - val_accuracy: 0.8304
Epoch 66/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4041 - accuracy: 0.8289 - val_loss: 0.4028 - val_accuracy: 0.8307
Epoch 67/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4035 - accuracy: 0.8290 - val_loss: 0.4023 - val_accuracy: 0.8315
Epoch 68/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4029 - accuracy: 0.8298 - val_loss: 0.4018 - val_accuracy: 0.8322
Epoch 69/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4022 - accuracy: 0.8297 - val_loss: 0.4013 - val_accuracy: 0.8322
Epoch 70/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4017 - accuracy: 0.8302 - val_loss: 0.4008 - val_accuracy: 0.8326
Epoch 71/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4011 - accuracy: 0.8306 - val_loss: 0.4004 - val_accuracy: 0.8333
Epoch 72/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4005 - accuracy: 0.8305 - val_loss: 0.3998 - val_accuracy: 0.8333
Epoch 73/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3999 - accuracy: 0.8314 - val_loss: 0.3994 - val_accuracy: 0.8337
Epoch 74/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3993 - accuracy: 0.8319 - val_loss: 0.3989 - val_accuracy: 0.8344
Epoch 75/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3988 - accuracy: 0.8319 - val_loss: 0.3984 - val_accuracy: 0.8348
Epoch 76/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3982 - accuracy: 0.8322 - val_loss: 0.3980 - val_accuracy: 0.8356
Epoch 77/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3977 - accuracy: 0.8325 - val_loss: 0.3975 - val_accuracy: 0.8363
Epoch 78/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3971 - accuracy: 0.8325 - val_loss: 0.3971 - val_accuracy: 0.8363
Epoch 79/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3965 - accuracy: 0.8329 - val_loss: 0.3966 - val_accuracy: 0.8352
Epoch 80/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3960 - accuracy: 0.8341 - val_loss: 0.3961 - val_accuracy: 0.8356
Epoch 81/100
197/197 [==============================] - 1s 7ms/step - loss: 0.3955 - accuracy: 0.8346 - val_loss: 0.3957 - val_accuracy: 0.8359
Epoch 82/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3950 - accuracy: 0.8341 - val_loss: 0.3953 - val_accuracy: 0.8359
Epoch 83/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3945 - accuracy: 0.8349 - val_loss: 0.3949 - val_accuracy: 0.8356
Epoch 84/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3940 - accuracy: 0.8357 - val_loss: 0.3944 - val_accuracy: 0.8359
Epoch 85/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3935 - accuracy: 0.8356 - val_loss: 0.3940 - val_accuracy: 0.8367
Epoch 86/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3930 - accuracy: 0.8362 - val_loss: 0.3936 - val_accuracy: 0.8370
Epoch 87/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3925 - accuracy: 0.8352 - val_loss: 0.3932 - val_accuracy: 0.8370
Epoch 88/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3920 - accuracy: 0.8357 - val_loss: 0.3928 - val_accuracy: 0.8378
Epoch 89/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3915 - accuracy: 0.8356 - val_loss: 0.3923 - val_accuracy: 0.8370
Epoch 90/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3910 - accuracy: 0.8360 - val_loss: 0.3920 - val_accuracy: 0.8370
Epoch 91/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3906 - accuracy: 0.8367 - val_loss: 0.3916 - val_accuracy: 0.8374
Epoch 92/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3901 - accuracy: 0.8363 - val_loss: 0.3911 - val_accuracy: 0.8374
Epoch 93/100
197/197 [==============================] - 1s 7ms/step - loss: 0.3896 - accuracy: 0.8362 - val_loss: 0.3908 - val_accuracy: 0.8374
Epoch 94/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3892 - accuracy: 0.8370 - val_loss: 0.3904 - val_accuracy: 0.8378
Epoch 95/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3887 - accuracy: 0.8378 - val_loss: 0.3900 - val_accuracy: 0.8374
Epoch 96/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3883 - accuracy: 0.8381 - val_loss: 0.3896 - val_accuracy: 0.8381
Epoch 97/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3878 - accuracy: 0.8375 - val_loss: 0.3893 - val_accuracy: 0.8378
Epoch 98/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3874 - accuracy: 0.8381 - val_loss: 0.3889 - val_accuracy: 0.8367
Epoch 99/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3870 - accuracy: 0.8378 - val_loss: 0.3885 - val_accuracy: 0.8367
Epoch 100/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3865 - accuracy: 0.8386 - val_loss: 0.3882 - val_accuracy: 0.8370

Tune the threshold using ROC-AUC

threshold2 = best_threshold(model2)

197/197 [==============================] - 0s 1ms/step
Best Threshold=0.191838, G-Mean=0.736

Loss and accuracy

# Predicting target values (y_pred2) using 'model2' on validation data (X_val)
y_pred2 = model2.predict(X_val)

# Applying a threshold to binary classification predictions
y_pred2 = (y_pred2 > threshold2)

graphs_and_report(history2, y_pred2)

85/85 [==============================] - 0s 2ms/step

Classification report

cr2 = metrics.classification_report(y_val, y_pred2,digits=3)
print(cr2)

              precision    recall  f1-score   support

           0      0.924     0.705     0.799      2147
           1      0.403     0.774     0.530       553

    accuracy                          0.719      2700
   macro avg      0.663     0.739     0.665      2700
weighted avg      0.817     0.719     0.744      2700

Confusion matrix

cm2 = confusion_matrix(y_val, y_pred2)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm2,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Observations

Precision (0): 0.924

• Precision measures the accuracy of positive predictions for class 0. In this case, it indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.705

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.799

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.403

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.774

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.530

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Model Building: Neural Network model with Adam Optimizer

Model 3

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

# Initializing the neural network model using the Sequential API
model3 = Sequential()

# Adding the input layer with 64 neurons and ReLU activation function
model3.add(Dense(64, activation='relu',
                 kernel_initializer='he_uniform', input_shape=(11,)))

# Adding the second hidden layer with 32 neurons and ReLU activation function
model3.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))

# Adding the output layer with one neuron and sigmoid activation function
model3.add(Dense(1, activation='sigmoid'))

# Initializing the Adam optimizer with a learning rate of 1e-3
adam = optimizers.Adam(learning_rate=1e-3)

# Compiling the model with binary cross-entropy loss and accuracy as the evaluation metric
model3.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])

# Printing a summary of the model architecture
model3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 64)                768       
                                                                 
 dense_1 (Dense)             (None, 32)                2080      
                                                                 
 dense_2 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 2,881
Trainable params: 2,881
Non-trainable params: 0
_________________________________________________________________

history3 = model3.fit(X_train, y_train, validation_data=(
    X_val, y_val), epochs=100, batch_size=32)

Epoch 1/100
197/197 [==============================] - 3s 6ms/step - loss: 0.4662 - accuracy: 0.7948 - val_loss: 0.4069 - val_accuracy: 0.8274
Epoch 2/100
197/197 [==============================] - 2s 8ms/step - loss: 0.3958 - accuracy: 0.8325 - val_loss: 0.3822 - val_accuracy: 0.8381
Epoch 3/100
197/197 [==============================] - 2s 9ms/step - loss: 0.3733 - accuracy: 0.8448 - val_loss: 0.3684 - val_accuracy: 0.8467
Epoch 4/100
197/197 [==============================] - 2s 8ms/step - loss: 0.3599 - accuracy: 0.8495 - val_loss: 0.3617 - val_accuracy: 0.8526
Epoch 5/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3503 - accuracy: 0.8544 - val_loss: 0.3571 - val_accuracy: 0.8589
Epoch 6/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3433 - accuracy: 0.8560 - val_loss: 0.3552 - val_accuracy: 0.8537
Epoch 7/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3396 - accuracy: 0.8578 - val_loss: 0.3551 - val_accuracy: 0.8519
Epoch 8/100
197/197 [==============================] - 2s 10ms/step - loss: 0.3373 - accuracy: 0.8606 - val_loss: 0.3514 - val_accuracy: 0.8563
Epoch 9/100
197/197 [==============================] - 2s 11ms/step - loss: 0.3334 - accuracy: 0.8602 - val_loss: 0.3505 - val_accuracy: 0.8567
Epoch 10/100
197/197 [==============================] - 2s 8ms/step - loss: 0.3305 - accuracy: 0.8603 - val_loss: 0.3573 - val_accuracy: 0.8541
Epoch 11/100
197/197 [==============================] - 2s 8ms/step - loss: 0.3286 - accuracy: 0.8654 - val_loss: 0.3522 - val_accuracy: 0.8552
Epoch 12/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3275 - accuracy: 0.8619 - val_loss: 0.3508 - val_accuracy: 0.8563
Epoch 13/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3226 - accuracy: 0.8640 - val_loss: 0.3521 - val_accuracy: 0.8552
Epoch 14/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3245 - accuracy: 0.8640 - val_loss: 0.3513 - val_accuracy: 0.8544
Epoch 15/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3209 - accuracy: 0.8652 - val_loss: 0.3518 - val_accuracy: 0.8533
Epoch 16/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3182 - accuracy: 0.8654 - val_loss: 0.3510 - val_accuracy: 0.8533
Epoch 17/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3174 - accuracy: 0.8676 - val_loss: 0.3533 - val_accuracy: 0.8533
Epoch 18/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3149 - accuracy: 0.8673 - val_loss: 0.3524 - val_accuracy: 0.8544
Epoch 19/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3139 - accuracy: 0.8702 - val_loss: 0.3518 - val_accuracy: 0.8522
Epoch 20/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3119 - accuracy: 0.8702 - val_loss: 0.3517 - val_accuracy: 0.8500
Epoch 21/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3112 - accuracy: 0.8700 - val_loss: 0.3528 - val_accuracy: 0.8533
Epoch 22/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3090 - accuracy: 0.8711 - val_loss: 0.3543 - val_accuracy: 0.8504
Epoch 23/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3071 - accuracy: 0.8713 - val_loss: 0.3523 - val_accuracy: 0.8507
Epoch 24/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3074 - accuracy: 0.8725 - val_loss: 0.3553 - val_accuracy: 0.8515
Epoch 25/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3036 - accuracy: 0.8705 - val_loss: 0.3587 - val_accuracy: 0.8504
Epoch 26/100
197/197 [==============================] - 1s 7ms/step - loss: 0.3032 - accuracy: 0.8702 - val_loss: 0.3562 - val_accuracy: 0.8511
Epoch 27/100
197/197 [==============================] - 1s 7ms/step - loss: 0.3020 - accuracy: 0.8729 - val_loss: 0.3563 - val_accuracy: 0.8519
Epoch 28/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2996 - accuracy: 0.8732 - val_loss: 0.3575 - val_accuracy: 0.8533
Epoch 29/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2984 - accuracy: 0.8752 - val_loss: 0.3572 - val_accuracy: 0.8530
Epoch 30/100
197/197 [==============================] - 2s 9ms/step - loss: 0.2982 - accuracy: 0.8743 - val_loss: 0.3580 - val_accuracy: 0.8541
Epoch 31/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2950 - accuracy: 0.8759 - val_loss: 0.3636 - val_accuracy: 0.8500
Epoch 32/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2952 - accuracy: 0.8763 - val_loss: 0.3651 - val_accuracy: 0.8481
Epoch 33/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2933 - accuracy: 0.8787 - val_loss: 0.3616 - val_accuracy: 0.8507
Epoch 34/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2916 - accuracy: 0.8787 - val_loss: 0.3595 - val_accuracy: 0.8548
Epoch 35/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2920 - accuracy: 0.8784 - val_loss: 0.3620 - val_accuracy: 0.8522
Epoch 36/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2900 - accuracy: 0.8789 - val_loss: 0.3639 - val_accuracy: 0.8515
Epoch 37/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2905 - accuracy: 0.8802 - val_loss: 0.3648 - val_accuracy: 0.8511
Epoch 38/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2859 - accuracy: 0.8789 - val_loss: 0.3680 - val_accuracy: 0.8481
Epoch 39/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2868 - accuracy: 0.8779 - val_loss: 0.3646 - val_accuracy: 0.8481
Epoch 40/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2846 - accuracy: 0.8792 - val_loss: 0.3645 - val_accuracy: 0.8500
Epoch 41/100
197/197 [==============================] - 2s 9ms/step - loss: 0.2849 - accuracy: 0.8813 - val_loss: 0.3673 - val_accuracy: 0.8489
Epoch 42/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2836 - accuracy: 0.8811 - val_loss: 0.3678 - val_accuracy: 0.8515
Epoch 43/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2819 - accuracy: 0.8832 - val_loss: 0.3688 - val_accuracy: 0.8522
Epoch 44/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2783 - accuracy: 0.8852 - val_loss: 0.3709 - val_accuracy: 0.8515
Epoch 45/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2792 - accuracy: 0.8835 - val_loss: 0.3723 - val_accuracy: 0.8507
Epoch 46/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2785 - accuracy: 0.8830 - val_loss: 0.3726 - val_accuracy: 0.8522
Epoch 47/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2775 - accuracy: 0.8835 - val_loss: 0.3716 - val_accuracy: 0.8530
Epoch 48/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2764 - accuracy: 0.8822 - val_loss: 0.3749 - val_accuracy: 0.8496
Epoch 49/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2748 - accuracy: 0.8835 - val_loss: 0.3716 - val_accuracy: 0.8519
Epoch 50/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2742 - accuracy: 0.8824 - val_loss: 0.3765 - val_accuracy: 0.8489
Epoch 51/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2719 - accuracy: 0.8840 - val_loss: 0.3788 - val_accuracy: 0.8511
Epoch 52/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2712 - accuracy: 0.8854 - val_loss: 0.3745 - val_accuracy: 0.8463
Epoch 53/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2709 - accuracy: 0.8856 - val_loss: 0.3784 - val_accuracy: 0.8548
Epoch 54/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2703 - accuracy: 0.8833 - val_loss: 0.3773 - val_accuracy: 0.8496
Epoch 55/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2701 - accuracy: 0.8838 - val_loss: 0.3817 - val_accuracy: 0.8485
Epoch 56/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2679 - accuracy: 0.8849 - val_loss: 0.3828 - val_accuracy: 0.8467
Epoch 57/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2664 - accuracy: 0.8871 - val_loss: 0.3796 - val_accuracy: 0.8485
Epoch 58/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2661 - accuracy: 0.8860 - val_loss: 0.3860 - val_accuracy: 0.8504
Epoch 59/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2645 - accuracy: 0.8898 - val_loss: 0.3830 - val_accuracy: 0.8419
Epoch 60/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2641 - accuracy: 0.8873 - val_loss: 0.3784 - val_accuracy: 0.8485
Epoch 61/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2637 - accuracy: 0.8902 - val_loss: 0.3867 - val_accuracy: 0.8470
Epoch 62/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2619 - accuracy: 0.8895 - val_loss: 0.3888 - val_accuracy: 0.8448
Epoch 63/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2587 - accuracy: 0.8906 - val_loss: 0.3891 - val_accuracy: 0.8474
Epoch 64/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2584 - accuracy: 0.8913 - val_loss: 0.4061 - val_accuracy: 0.8278
Epoch 65/100
197/197 [==============================] - 2s 9ms/step - loss: 0.2607 - accuracy: 0.8875 - val_loss: 0.3874 - val_accuracy: 0.8463
Epoch 66/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2572 - accuracy: 0.8900 - val_loss: 0.3871 - val_accuracy: 0.8515
Epoch 67/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2567 - accuracy: 0.8890 - val_loss: 0.3866 - val_accuracy: 0.8463
Epoch 68/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2549 - accuracy: 0.8929 - val_loss: 0.3872 - val_accuracy: 0.8515
Epoch 69/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2552 - accuracy: 0.8919 - val_loss: 0.3934 - val_accuracy: 0.8456
Epoch 70/100
197/197 [==============================] - 2s 10ms/step - loss: 0.2556 - accuracy: 0.8902 - val_loss: 0.3916 - val_accuracy: 0.8519
Epoch 71/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2536 - accuracy: 0.8922 - val_loss: 0.3992 - val_accuracy: 0.8511
Epoch 72/100
197/197 [==============================] - 2s 11ms/step - loss: 0.2513 - accuracy: 0.8922 - val_loss: 0.4001 - val_accuracy: 0.8415
Epoch 73/100
197/197 [==============================] - 2s 10ms/step - loss: 0.2531 - accuracy: 0.8932 - val_loss: 0.3949 - val_accuracy: 0.8474
Epoch 74/100
197/197 [==============================] - 2s 10ms/step - loss: 0.2507 - accuracy: 0.8932 - val_loss: 0.3961 - val_accuracy: 0.8452
Epoch 75/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2502 - accuracy: 0.8910 - val_loss: 0.4033 - val_accuracy: 0.8519
Epoch 76/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2492 - accuracy: 0.8929 - val_loss: 0.3925 - val_accuracy: 0.8500
Epoch 77/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2476 - accuracy: 0.8949 - val_loss: 0.3974 - val_accuracy: 0.8478
Epoch 78/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2474 - accuracy: 0.8970 - val_loss: 0.4020 - val_accuracy: 0.8437
Epoch 79/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2450 - accuracy: 0.8956 - val_loss: 0.3984 - val_accuracy: 0.8474
Epoch 80/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2466 - accuracy: 0.8948 - val_loss: 0.4000 - val_accuracy: 0.8481
Epoch 81/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2438 - accuracy: 0.8946 - val_loss: 0.3999 - val_accuracy: 0.8530
Epoch 82/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2416 - accuracy: 0.8981 - val_loss: 0.4025 - val_accuracy: 0.8463
Epoch 83/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2422 - accuracy: 0.8948 - val_loss: 0.4030 - val_accuracy: 0.8537
Epoch 84/100
197/197 [==============================] - 1s 6ms/step - loss: 0.2407 - accuracy: 0.9000 - val_loss: 0.4050 - val_accuracy: 0.8515
Epoch 85/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2403 - accuracy: 0.8967 - val_loss: 0.4028 - val_accuracy: 0.8467
Epoch 86/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2381 - accuracy: 0.9010 - val_loss: 0.4144 - val_accuracy: 0.8344
Epoch 87/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2363 - accuracy: 0.8987 - val_loss: 0.4056 - val_accuracy: 0.8474
Epoch 88/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2377 - accuracy: 0.8962 - val_loss: 0.4122 - val_accuracy: 0.8519
Epoch 89/100
197/197 [==============================] - 1s 5ms/step - loss: 0.2357 - accuracy: 0.8983 - val_loss: 0.4178 - val_accuracy: 0.8378
Epoch 90/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2360 - accuracy: 0.8973 - val_loss: 0.4110 - val_accuracy: 0.8463
Epoch 91/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2337 - accuracy: 0.9002 - val_loss: 0.4221 - val_accuracy: 0.8348
Epoch 92/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2335 - accuracy: 0.8989 - val_loss: 0.4145 - val_accuracy: 0.8444
Epoch 93/100
197/197 [==============================] - 3s 14ms/step - loss: 0.2324 - accuracy: 0.9013 - val_loss: 0.4280 - val_accuracy: 0.8322
Epoch 94/100
197/197 [==============================] - 2s 12ms/step - loss: 0.2333 - accuracy: 0.9003 - val_loss: 0.4152 - val_accuracy: 0.8430
Epoch 95/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2300 - accuracy: 0.9046 - val_loss: 0.4303 - val_accuracy: 0.8281
Epoch 96/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2301 - accuracy: 0.9017 - val_loss: 0.4169 - val_accuracy: 0.8456
Epoch 97/100
197/197 [==============================] - 2s 8ms/step - loss: 0.2303 - accuracy: 0.9029 - val_loss: 0.4196 - val_accuracy: 0.8393
Epoch 98/100
197/197 [==============================] - 1s 8ms/step - loss: 0.2279 - accuracy: 0.9022 - val_loss: 0.4202 - val_accuracy: 0.8393
Epoch 99/100
197/197 [==============================] - 1s 7ms/step - loss: 0.2272 - accuracy: 0.9022 - val_loss: 0.4218 - val_accuracy: 0.8430
Epoch 100/100
197/197 [==============================] - 1s 4ms/step - loss: 0.2280 - accuracy: 0.9011 - val_loss: 0.4259 - val_accuracy: 0.8374

Tune the threshold using ROC-AUC

threshold3 = best_threshold(model3)

197/197 [==============================] - 0s 2ms/step
Best Threshold=0.247884, G-Mean=0.879

Loss and accuracy

# Predicting target values (y_pred3) using 'model3' on the validation data (X_val).
y_pred3 = model3.predict(X_val)

# Applying a threshold to the predictions.
y_pred3 = (y_pred3 > threshold3)

graphs_and_report(history3, y_pred3)

85/85 [==============================] - 0s 2ms/step

Classification report

# Classification report
cr3 = metrics.classification_report(y_val, y_pred3,digits=3)
print(cr3)

              precision    recall  f1-score   support

           0      0.911     0.821     0.863      2147
           1      0.497     0.687     0.577       553

    accuracy                          0.793      2700
   macro avg      0.704     0.754     0.720      2700
weighted avg      0.826     0.793     0.805      2700

Confusion matrix

cm3 = confusion_matrix(y_val, y_pred3)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm3,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Classification Report:

Precision (0): 0.911

• Precision measures the accuracy of positive predictions for class 0. In this case, it indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.821

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.863

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.497

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.687

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.577

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Model Improvement: Neural Network model with Dropout

Model 4

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

# Initializing the neural network model 'model4'
model4 = Sequential()

# Adding the input layer with 64 neurons and ReLU as the activation function
model4.add(Dense(64, activation='relu', kernel_initializer='he_uniform', input_shape=(11,)))

# Adding a dropout layer with a dropout rate of 20% to prevent overfitting
model4.add(Dropout(0.2))

# Adding the second hidden layer with 32 neurons and ReLU as the activation function
model4.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))

# Adding another dropout layer with a dropout rate of 20%
model4.add(Dropout(0.2))

# Adding the output layer with one neuron and sigmoid as the activation function
model4.add(Dense(1, activation='sigmoid'))

# Defining the stochastic gradient descent (SGD) optimizer with a learning rate of 1e-3
sgd = optimizers.SGD(learning_rate=1e-3)

# Compiling the model using binary cross-entropy loss and accuracy as a metric
model4.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])

# Printing a summary of the model architecture
model4.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 64)                768       
                                                                 
 dropout (Dropout)           (None, 64)                0         
                                                                 
 dense_1 (Dense)             (None, 32)                2080      
                                                                 
 dropout_1 (Dropout)         (None, 32)                0         
                                                                 
 dense_2 (Dense)             (None, 1)                 33        
                                                                 
=================================================================
Total params: 2,881
Trainable params: 2,881
Non-trainable params: 0
_________________________________________________________________

Adding early stop

# Define early stopping criteria to prevent overfitting
early_stop = callbacks.EarlyStopping(
    monitor='val_loss', min_delta=0.001, patience=5)

# Train 'model4' on the training data (X_train, y_train) with validation data (X_val, y_val)
# for a maximum of 100 epochs with a batch size of 32.
# Early stopping will be applied based on validation loss, with a patience of 5 epochs.
history4 = model4.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, batch_size=32, callbacks=[early_stop])

Epoch 1/100
197/197 [==============================] - 3s 9ms/step - loss: 0.6922 - accuracy: 0.6349 - val_loss: 0.6169 - val_accuracy: 0.7000
Epoch 2/100
197/197 [==============================] - 1s 7ms/step - loss: 0.6298 - accuracy: 0.7000 - val_loss: 0.5786 - val_accuracy: 0.7422
Epoch 3/100
197/197 [==============================] - 1s 7ms/step - loss: 0.5914 - accuracy: 0.7322 - val_loss: 0.5560 - val_accuracy: 0.7693
Epoch 4/100
197/197 [==============================] - 2s 10ms/step - loss: 0.5757 - accuracy: 0.7517 - val_loss: 0.5393 - val_accuracy: 0.7811
Epoch 5/100
197/197 [==============================] - 2s 9ms/step - loss: 0.5632 - accuracy: 0.7627 - val_loss: 0.5260 - val_accuracy: 0.7856
Epoch 6/100
197/197 [==============================] - 1s 7ms/step - loss: 0.5505 - accuracy: 0.7724 - val_loss: 0.5154 - val_accuracy: 0.7911
Epoch 7/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5306 - accuracy: 0.7771 - val_loss: 0.5064 - val_accuracy: 0.7911
Epoch 8/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5295 - accuracy: 0.7773 - val_loss: 0.4984 - val_accuracy: 0.7919
Epoch 9/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5336 - accuracy: 0.7727 - val_loss: 0.4919 - val_accuracy: 0.7922
Epoch 10/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5115 - accuracy: 0.7868 - val_loss: 0.4864 - val_accuracy: 0.7933
Epoch 11/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5137 - accuracy: 0.7827 - val_loss: 0.4814 - val_accuracy: 0.7933
Epoch 12/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5068 - accuracy: 0.7856 - val_loss: 0.4769 - val_accuracy: 0.7930
Epoch 13/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5012 - accuracy: 0.7860 - val_loss: 0.4728 - val_accuracy: 0.7948
Epoch 14/100
197/197 [==============================] - 1s 5ms/step - loss: 0.5022 - accuracy: 0.7819 - val_loss: 0.4693 - val_accuracy: 0.7944
Epoch 15/100
197/197 [==============================] - 1s 4ms/step - loss: 0.5000 - accuracy: 0.7886 - val_loss: 0.4662 - val_accuracy: 0.7963
Epoch 16/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4941 - accuracy: 0.7894 - val_loss: 0.4635 - val_accuracy: 0.7978
Epoch 17/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4908 - accuracy: 0.7932 - val_loss: 0.4610 - val_accuracy: 0.7985
Epoch 18/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4864 - accuracy: 0.7951 - val_loss: 0.4587 - val_accuracy: 0.7993
Epoch 19/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4846 - accuracy: 0.7943 - val_loss: 0.4566 - val_accuracy: 0.7996
Epoch 20/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4815 - accuracy: 0.7970 - val_loss: 0.4547 - val_accuracy: 0.8000
Epoch 21/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4799 - accuracy: 0.7917 - val_loss: 0.4529 - val_accuracy: 0.8011
Epoch 22/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4813 - accuracy: 0.7938 - val_loss: 0.4511 - val_accuracy: 0.8015
Epoch 23/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4871 - accuracy: 0.7929 - val_loss: 0.4496 - val_accuracy: 0.8026
Epoch 24/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4804 - accuracy: 0.7929 - val_loss: 0.4481 - val_accuracy: 0.8030
Epoch 25/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4777 - accuracy: 0.7941 - val_loss: 0.4467 - val_accuracy: 0.8041
Epoch 26/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4713 - accuracy: 0.7990 - val_loss: 0.4453 - val_accuracy: 0.8048
Epoch 27/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4726 - accuracy: 0.7944 - val_loss: 0.4441 - val_accuracy: 0.8056
Epoch 28/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4730 - accuracy: 0.7933 - val_loss: 0.4430 - val_accuracy: 0.8052
Epoch 29/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4713 - accuracy: 0.7978 - val_loss: 0.4419 - val_accuracy: 0.8059
Epoch 30/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4690 - accuracy: 0.7963 - val_loss: 0.4409 - val_accuracy: 0.8067
Epoch 31/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4688 - accuracy: 0.7968 - val_loss: 0.4400 - val_accuracy: 0.8070
Epoch 32/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4709 - accuracy: 0.8005 - val_loss: 0.4391 - val_accuracy: 0.8067
Epoch 33/100
197/197 [==============================] - 2s 8ms/step - loss: 0.4658 - accuracy: 0.7963 - val_loss: 0.4382 - val_accuracy: 0.8085
Epoch 34/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4610 - accuracy: 0.7995 - val_loss: 0.4373 - val_accuracy: 0.8081
Epoch 35/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4737 - accuracy: 0.7989 - val_loss: 0.4365 - val_accuracy: 0.8096
Epoch 36/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4655 - accuracy: 0.7997 - val_loss: 0.4357 - val_accuracy: 0.8096
Epoch 37/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4629 - accuracy: 0.7978 - val_loss: 0.4350 - val_accuracy: 0.8100
Epoch 38/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4618 - accuracy: 0.7984 - val_loss: 0.4343 - val_accuracy: 0.8100
Epoch 39/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4662 - accuracy: 0.8041 - val_loss: 0.4336 - val_accuracy: 0.8115
Epoch 40/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4619 - accuracy: 0.8030 - val_loss: 0.4330 - val_accuracy: 0.8119
Epoch 41/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4615 - accuracy: 0.7973 - val_loss: 0.4324 - val_accuracy: 0.8126
Epoch 42/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4571 - accuracy: 0.8003 - val_loss: 0.4316 - val_accuracy: 0.8126
Epoch 43/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4625 - accuracy: 0.7997 - val_loss: 0.4311 - val_accuracy: 0.8130
Epoch 44/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4581 - accuracy: 0.8046 - val_loss: 0.4305 - val_accuracy: 0.8130
Epoch 45/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4555 - accuracy: 0.8011 - val_loss: 0.4299 - val_accuracy: 0.8126
Epoch 46/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4564 - accuracy: 0.8044 - val_loss: 0.4294 - val_accuracy: 0.8133
Epoch 47/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4574 - accuracy: 0.7989 - val_loss: 0.4288 - val_accuracy: 0.8137
Epoch 48/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4618 - accuracy: 0.7983 - val_loss: 0.4284 - val_accuracy: 0.8141
Epoch 49/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4568 - accuracy: 0.8038 - val_loss: 0.4279 - val_accuracy: 0.8141
Epoch 50/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4623 - accuracy: 0.8002 - val_loss: 0.4275 - val_accuracy: 0.8144
Epoch 51/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4555 - accuracy: 0.8006 - val_loss: 0.4270 - val_accuracy: 0.8144
Epoch 52/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4612 - accuracy: 0.8013 - val_loss: 0.4266 - val_accuracy: 0.8137
Epoch 53/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4563 - accuracy: 0.8032 - val_loss: 0.4261 - val_accuracy: 0.8148
Epoch 54/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4521 - accuracy: 0.8037 - val_loss: 0.4257 - val_accuracy: 0.8148
Epoch 55/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4562 - accuracy: 0.8027 - val_loss: 0.4253 - val_accuracy: 0.8144
Epoch 56/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4533 - accuracy: 0.8016 - val_loss: 0.4248 - val_accuracy: 0.8144
Epoch 57/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4556 - accuracy: 0.8035 - val_loss: 0.4245 - val_accuracy: 0.8163
Epoch 58/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4491 - accuracy: 0.8070 - val_loss: 0.4240 - val_accuracy: 0.8159
Epoch 59/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4517 - accuracy: 0.8067 - val_loss: 0.4236 - val_accuracy: 0.8148
Epoch 60/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4514 - accuracy: 0.8016 - val_loss: 0.4232 - val_accuracy: 0.8156
Epoch 61/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4516 - accuracy: 0.8027 - val_loss: 0.4228 - val_accuracy: 0.8148
Epoch 62/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4484 - accuracy: 0.8067 - val_loss: 0.4223 - val_accuracy: 0.8152
Epoch 63/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4509 - accuracy: 0.8054 - val_loss: 0.4220 - val_accuracy: 0.8152
Epoch 64/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4500 - accuracy: 0.8078 - val_loss: 0.4217 - val_accuracy: 0.8152
Epoch 65/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4448 - accuracy: 0.8103 - val_loss: 0.4212 - val_accuracy: 0.8152
Epoch 66/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4513 - accuracy: 0.8067 - val_loss: 0.4209 - val_accuracy: 0.8156
Epoch 67/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4473 - accuracy: 0.8046 - val_loss: 0.4205 - val_accuracy: 0.8159
Epoch 68/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4483 - accuracy: 0.7995 - val_loss: 0.4202 - val_accuracy: 0.8163
Epoch 69/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4458 - accuracy: 0.8063 - val_loss: 0.4199 - val_accuracy: 0.8167
Epoch 70/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4486 - accuracy: 0.8038 - val_loss: 0.4195 - val_accuracy: 0.8163
Epoch 71/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4476 - accuracy: 0.8076 - val_loss: 0.4191 - val_accuracy: 0.8163
Epoch 72/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4451 - accuracy: 0.8054 - val_loss: 0.4188 - val_accuracy: 0.8163
Epoch 73/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4470 - accuracy: 0.8043 - val_loss: 0.4186 - val_accuracy: 0.8163
Epoch 74/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4449 - accuracy: 0.8084 - val_loss: 0.4183 - val_accuracy: 0.8163
Epoch 75/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4452 - accuracy: 0.8060 - val_loss: 0.4180 - val_accuracy: 0.8170
Epoch 76/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4374 - accuracy: 0.8075 - val_loss: 0.4176 - val_accuracy: 0.8167
Epoch 77/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4462 - accuracy: 0.8073 - val_loss: 0.4173 - val_accuracy: 0.8174
Epoch 78/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4422 - accuracy: 0.8052 - val_loss: 0.4169 - val_accuracy: 0.8174
Epoch 79/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4438 - accuracy: 0.8098 - val_loss: 0.4166 - val_accuracy: 0.8170
Epoch 80/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4461 - accuracy: 0.8073 - val_loss: 0.4164 - val_accuracy: 0.8178
Epoch 81/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4460 - accuracy: 0.8057 - val_loss: 0.4161 - val_accuracy: 0.8181
Epoch 82/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4434 - accuracy: 0.8078 - val_loss: 0.4158 - val_accuracy: 0.8185
Epoch 83/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4425 - accuracy: 0.8083 - val_loss: 0.4155 - val_accuracy: 0.8185
Epoch 84/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4425 - accuracy: 0.8070 - val_loss: 0.4152 - val_accuracy: 0.8185
Epoch 85/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4398 - accuracy: 0.8117 - val_loss: 0.4149 - val_accuracy: 0.8181
Epoch 86/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4423 - accuracy: 0.8079 - val_loss: 0.4146 - val_accuracy: 0.8181
Epoch 87/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4438 - accuracy: 0.8090 - val_loss: 0.4144 - val_accuracy: 0.8185
Epoch 88/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4370 - accuracy: 0.8083 - val_loss: 0.4141 - val_accuracy: 0.8181
Epoch 89/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4448 - accuracy: 0.8067 - val_loss: 0.4139 - val_accuracy: 0.8193
Epoch 90/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4409 - accuracy: 0.8086 - val_loss: 0.4137 - val_accuracy: 0.8196
Epoch 91/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4385 - accuracy: 0.8068 - val_loss: 0.4134 - val_accuracy: 0.8200
Epoch 92/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4428 - accuracy: 0.8108 - val_loss: 0.4132 - val_accuracy: 0.8204
Epoch 93/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4417 - accuracy: 0.8067 - val_loss: 0.4130 - val_accuracy: 0.8207
Epoch 94/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4426 - accuracy: 0.8098 - val_loss: 0.4128 - val_accuracy: 0.8211
Epoch 95/100
197/197 [==============================] - 1s 5ms/step - loss: 0.4366 - accuracy: 0.8098 - val_loss: 0.4125 - val_accuracy: 0.8222
Epoch 96/100
197/197 [==============================] - 1s 4ms/step - loss: 0.4389 - accuracy: 0.8111 - val_loss: 0.4123 - val_accuracy: 0.8219
Epoch 97/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4416 - accuracy: 0.8054 - val_loss: 0.4120 - val_accuracy: 0.8222
Epoch 98/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4421 - accuracy: 0.8117 - val_loss: 0.4119 - val_accuracy: 0.8204
Epoch 99/100
197/197 [==============================] - 1s 7ms/step - loss: 0.4354 - accuracy: 0.8097 - val_loss: 0.4116 - val_accuracy: 0.8219
Epoch 100/100
197/197 [==============================] - 1s 6ms/step - loss: 0.4338 - accuracy: 0.8113 - val_loss: 0.4112 - val_accuracy: 0.8215

Tune the threshold using ROC-AUC

threshold4 = best_threshold(model4)

197/197 [==============================] - 2s 6ms/step
Best Threshold=0.224817, G-Mean=0.715

Loss and accuracy

# Predicting target values (y_pred4) using 'model4' on the validation data (X_val).
y_pred4 = model4.predict(X_val)

# Applying a threshold to the predictions.
y_pred4 = (y_pred4 > threshold4)

graphs_and_report(history4, y_pred4)

85/85 [==============================] - 0s 2ms/step

Classification report

cr4 = metrics.classification_report(y_val, y_pred4)
print(cr4)

              precision    recall  f1-score   support

           0       0.92      0.73      0.81      2147
           1       0.42      0.76      0.54       553

    accuracy                           0.73      2700
   macro avg       0.67      0.74      0.68      2700
weighted avg       0.82      0.73      0.76      2700

Confusion matrix

cm4 = confusion_matrix(y_val, y_pred4)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm4,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Observations

Precision (0): 0.92

• Precision measures the accuracy of positive predictions for class 0. A high value indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.73

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.81

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.42

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.76

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.54

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Model Improvement: Neural Network model with Hyperparameter tuning

Model 5

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

def create_model(dropout_rate=0.0, lr=0.001, layer_1=64, layer_2=32):
    """
    Create a neural network model with customizable dropout rates, learning rate, and layer sizes.

    Parameters:
    - dropout_rate: The dropout rate applied to hidden layers.
    - lr: The learning rate for the optimizer.
    - layer_1: Number of neurons in the first hidden layer.
    - layer_2: Number of neurons in the second hidden layer.

    Returns:
    A compiled Keras model.
    """

    # Initializing the neural network model
    model = Sequential()

    # Adding the input layer with ReLU activation and He uniform initialization
    model.add(Dense(layer_1, kernel_initializer='he_uniform', activation='relu', input_dim=X_train.shape[1]))

    # Adding dropout to the input layer with a customizable dropout rate
    model.add(Dropout(dropout_rate))

    # Adding the hidden layer with ReLU activation and He uniform initialization
    model.add(Dense(layer_2, kernel_initializer='he_uniform', activation='relu'))

    # Adding dropout to the hidden layer with a customizable dropout rate
    model.add(Dropout(dropout_rate))

    # Adding the output layer with sigmoid activation for binary classification
    model.add(Dense(1, activation='sigmoid'))

    # Initializing the optimizer (Stochastic Gradient Descent with the specified learning rate)
    optimizer = tf.keras.optimizers.SGD(learning_rate=lr)

    # Compiling the model with binary cross-entropy as the loss and accuracy as a metric
    model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

    return model

# Creating a KerasClassifier based on the custom create_model function
keras_estimator = KerasClassifier(build_fn=create_model, verbose=1)

# Defining the hyperparameter grid for grid search
param_grid = {
    "batch_size": [32, 64, 128],
    "lr": [0.01, 0.001, 0.1],
    "dropout_rate": [0.0, 0.2, 0.5]
}

# Number of splits for k-fold cross-validation
kfold_splits = 3

# Initializing the GridSearchCV with the Keras estimator, hyperparameter grid, and cross-validation settings
grid = GridSearchCV(estimator=keras_estimator, verbose=1, cv=kfold_splits, param_grid=param_grid, n_jobs=-1)

# Performing grid search with cross-validation on the training data and validation data
grid_result = grid.fit(X_train, y_train, validation_data=(X_val, y_val), verbose=1)

# Printing the best hyperparameters and corresponding best score
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

# Extracting and storing mean test scores, standard deviations, and parameter combinations
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

Fitting 3 folds for each of 27 candidates, totalling 81 fits
197/197 [==============================] - 2s 5ms/step - loss: 0.4331 - accuracy: 0.8110 - val_loss: 0.3942 - val_accuracy: 0.8326
Best: 0.825126 using {'batch_size': 32, 'dropout_rate': 0.0, 'lr': 0.1}

# Creating 'model5' with the best hyperparameters from grid search
model5 = create_model(lr=grid_result.best_params_['lr'], dropout_rate=grid_result.best_params_['dropout_rate'])

# Printing a summary of the architecture of 'model5'
model5.summary()

Model: "sequential_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_15 (Dense)            (None, 64)                768       
                                                                 
 dropout_7 (Dropout)         (None, 64)                0         
                                                                 
 dense_16 (Dense)            (None, 32)                2080      
                                                                 
 dropout_8 (Dropout)         (None, 32)                0         
                                                                 
 dense_17 (Dense)            (None, 1)                 33        
                                                                 
=================================================================
Total params: 2,881
Trainable params: 2,881
Non-trainable params: 0
_________________________________________________________________

# Training 'model5' on the training data (X_train, y_train) with validation data (X_val, y_val)
# for a maximum of 100 epochs using the best batch size obtained from grid search.
# Early stopping is applied based on validation loss, as defined by the 'early_stop' callback.
history5 = model5.fit(X_train, y_train, validation_data=(X_val, y_val),
                      epochs=100, batch_size=grid_result.best_params_['batch_size'], callbacks=[early_stop])

Epoch 1/100
197/197 [==============================] - 2s 6ms/step - loss: 0.4411 - accuracy: 0.8100 - val_loss: 0.4027 - val_accuracy: 0.8256
Epoch 2/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3806 - accuracy: 0.8349 - val_loss: 0.3742 - val_accuracy: 0.8448
Epoch 3/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3623 - accuracy: 0.8465 - val_loss: 0.3688 - val_accuracy: 0.8500
Epoch 4/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3539 - accuracy: 0.8517 - val_loss: 0.3755 - val_accuracy: 0.8470
Epoch 5/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3486 - accuracy: 0.8525 - val_loss: 0.3531 - val_accuracy: 0.8578
Epoch 6/100
197/197 [==============================] - 1s 6ms/step - loss: 0.3436 - accuracy: 0.8548 - val_loss: 0.3592 - val_accuracy: 0.8544
Epoch 7/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3406 - accuracy: 0.8587 - val_loss: 0.3610 - val_accuracy: 0.8544
Epoch 8/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3375 - accuracy: 0.8630 - val_loss: 0.3545 - val_accuracy: 0.8556
Epoch 9/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3350 - accuracy: 0.8570 - val_loss: 0.3489 - val_accuracy: 0.8589
Epoch 10/100
197/197 [==============================] - 1s 4ms/step - loss: 0.3316 - accuracy: 0.8633 - val_loss: 0.3505 - val_accuracy: 0.8593
Epoch 11/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3305 - accuracy: 0.8625 - val_loss: 0.3518 - val_accuracy: 0.8574
Epoch 12/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3287 - accuracy: 0.8656 - val_loss: 0.3516 - val_accuracy: 0.8593
Epoch 13/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3253 - accuracy: 0.8652 - val_loss: 0.3562 - val_accuracy: 0.8607
Epoch 14/100
197/197 [==============================] - 1s 5ms/step - loss: 0.3243 - accuracy: 0.8651 - val_loss: 0.3512 - val_accuracy: 0.8619

Tune the threshold using ROC-AUC

threshold5 = best_threshold(model5)

197/197 [==============================] - 1s 2ms/step
Best Threshold=0.216372, G-Mean=0.803

Loss and accuracy

# Predicting target values (y_pred5) using 'model5' on the validation data (X_val).
y_pred5 = model5.predict(X_val)

# Applying a threshold to the predictions.
y_pred5 = (y_pred5 > threshold5)
graphs_and_report(history5, y_pred5)

85/85 [==============================] - 0s 2ms/step

Classification report

cr5 = metrics.classification_report(y_val, y_pred5,digits=3)
print(cr5)

              precision    recall  f1-score   support

           0      0.922     0.816     0.866      2147
           1      0.506     0.732     0.598       553

    accuracy                          0.799      2700
   macro avg      0.714     0.774     0.732      2700
weighted avg      0.837     0.799     0.811      2700

Confusion matrix

cm5 = confusion_matrix(y_val, y_pred5)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm5,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Observations

Precision (0): 0.922

• Precision measures the accuracy of positive predictions for class 0. A high value indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.816

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.866

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.506

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.732

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.598

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Model Improvement: Neural Network model with balanced data

Model 6

# Applying SMOTE (Synthetic Minority Over-sampling Technique) to the training data
sm = SMOTE(random_state=42)
X_train, y_train = sm.fit_resample(X_train, y_train)

# Printing the shape of the training data after upsampling
print('After UpSampling, the shape of train_X: {}'.format(X_train.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train.shape))

After UpSampling, the shape of train_X: (10044, 11)
After UpSampling, the shape of train_y: (10044,) 

# Printing unique values and their counts in the 'y_train' array
print(np.unique(y_train, return_counts=True))

(array([0, 1]), array([5022, 5022]))

import random
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
random.seed(42)
tf.random.set_seed(42)

# Initializing the neural network model 'model6'
model6 = Sequential()

# Adding the input layer with 32 neurons and ReLU activation, using He uniform initialization,
# and specifying the input shape.
model6.add(Dense(32, activation='relu', kernel_initializer='he_uniform', input_shape=(11,)))

# Adding a dropout layer with a dropout rate of 20% to prevent overfitting.
model6.add(Dropout(0.2))

# Adding the second hidden layer with 16 neurons, ReLU activation, and He uniform initialization.
model6.add(Dense(16, activation='relu', kernel_initializer='he_uniform'))

# Adding another dropout layer with a dropout rate of 20%.
model6.add(Dropout(0.2))

# Adding the third hidden layer with 8 neurons, ReLU activation, and He uniform initialization.
model6.add(Dense(8, activation='relu', kernel_initializer='he_uniform'))

# Adding the output layer with one neuron and sigmoid activation for binary classification.
model6.add(Dense(1, activation='sigmoid'))

# Initializing the stochastic gradient descent (SGD) optimizer with a learning rate of 1e-3.
sgd = optimizers.SGD(learning_rate=1e-3)

# Compiling the model using binary cross-entropy loss and accuracy as a metric.
model6.compile(optimizer=sgd, loss='binary_crossentropy', metrics=['accuracy'])

# Printing a summary of the model architecture.
model6.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_4 (Dense)             (None, 32)                384       
                                                                 
 dropout_2 (Dropout)         (None, 32)                0         
                                                                 
 dense_5 (Dense)             (None, 16)                528       
                                                                 
 dense_6 (Dense)             (None, 8)                 136       
                                                                 
 dense_7 (Dense)             (None, 1)                 9         
                                                                 
=================================================================
Total params: 1,057
Trainable params: 1,057
Non-trainable params: 0
_________________________________________________________________

# Training 'model6' on the training data (X_train, y_train) with validation data (X_val, y_val)
# for a maximum of 100 epochs using a batch size of 64.
# Early stopping is applied based on validation loss, as defined by the 'early_stop' callback.
history6 = model6.fit(X_train, y_train, validation_data=(X_val, y_val),
                      epochs=100, batch_size=64, callbacks=[early_stop])

Epoch 1/100
157/157 [==============================] - 1s 9ms/step - loss: 0.6352 - accuracy: 0.6364 - val_loss: 0.5901 - val_accuracy: 0.7330
Epoch 2/100
157/157 [==============================] - 1s 8ms/step - loss: 0.6323 - accuracy: 0.6412 - val_loss: 0.5882 - val_accuracy: 0.7348
Epoch 3/100
157/157 [==============================] - 1s 8ms/step - loss: 0.6314 - accuracy: 0.6438 - val_loss: 0.5854 - val_accuracy: 0.7389
Epoch 4/100
157/157 [==============================] - 1s 7ms/step - loss: 0.6253 - accuracy: 0.6484 - val_loss: 0.5854 - val_accuracy: 0.7396
Epoch 5/100
157/157 [==============================] - 1s 8ms/step - loss: 0.6257 - accuracy: 0.6482 - val_loss: 0.5833 - val_accuracy: 0.7411
Epoch 6/100
157/157 [==============================] - 2s 10ms/step - loss: 0.6273 - accuracy: 0.6475 - val_loss: 0.5816 - val_accuracy: 0.7415
Epoch 7/100
157/157 [==============================] - 1s 9ms/step - loss: 0.6181 - accuracy: 0.6554 - val_loss: 0.5791 - val_accuracy: 0.7452
Epoch 8/100
157/157 [==============================] - 1s 7ms/step - loss: 0.6189 - accuracy: 0.6560 - val_loss: 0.5764 - val_accuracy: 0.7456
Epoch 9/100
157/157 [==============================] - 1s 6ms/step - loss: 0.6177 - accuracy: 0.6564 - val_loss: 0.5751 - val_accuracy: 0.7474
Epoch 10/100
157/157 [==============================] - 1s 4ms/step - loss: 0.6121 - accuracy: 0.6591 - val_loss: 0.5743 - val_accuracy: 0.7474
Epoch 11/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6115 - accuracy: 0.6617 - val_loss: 0.5716 - val_accuracy: 0.7463
Epoch 12/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6108 - accuracy: 0.6648 - val_loss: 0.5691 - val_accuracy: 0.7452
Epoch 13/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6110 - accuracy: 0.6642 - val_loss: 0.5685 - val_accuracy: 0.7459
Epoch 14/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6089 - accuracy: 0.6654 - val_loss: 0.5667 - val_accuracy: 0.7467
Epoch 15/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6056 - accuracy: 0.6669 - val_loss: 0.5654 - val_accuracy: 0.7481
Epoch 16/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6058 - accuracy: 0.6674 - val_loss: 0.5626 - val_accuracy: 0.7496
Epoch 17/100
157/157 [==============================] - 1s 5ms/step - loss: 0.6037 - accuracy: 0.6728 - val_loss: 0.5613 - val_accuracy: 0.7496
Epoch 18/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5977 - accuracy: 0.6782 - val_loss: 0.5597 - val_accuracy: 0.7500
Epoch 19/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5995 - accuracy: 0.6744 - val_loss: 0.5580 - val_accuracy: 0.7511
Epoch 20/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5965 - accuracy: 0.6826 - val_loss: 0.5565 - val_accuracy: 0.7507
Epoch 21/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5940 - accuracy: 0.6754 - val_loss: 0.5550 - val_accuracy: 0.7500
Epoch 22/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5948 - accuracy: 0.6783 - val_loss: 0.5523 - val_accuracy: 0.7522
Epoch 23/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5957 - accuracy: 0.6776 - val_loss: 0.5497 - val_accuracy: 0.7522
Epoch 24/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5920 - accuracy: 0.6858 - val_loss: 0.5490 - val_accuracy: 0.7522
Epoch 25/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5869 - accuracy: 0.6871 - val_loss: 0.5484 - val_accuracy: 0.7537
Epoch 26/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5894 - accuracy: 0.6885 - val_loss: 0.5469 - val_accuracy: 0.7537
Epoch 27/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5847 - accuracy: 0.6908 - val_loss: 0.5457 - val_accuracy: 0.7533
Epoch 28/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5857 - accuracy: 0.6856 - val_loss: 0.5457 - val_accuracy: 0.7541
Epoch 29/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5846 - accuracy: 0.6846 - val_loss: 0.5444 - val_accuracy: 0.7541
Epoch 30/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5832 - accuracy: 0.6900 - val_loss: 0.5425 - val_accuracy: 0.7537
Epoch 31/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5803 - accuracy: 0.6967 - val_loss: 0.5392 - val_accuracy: 0.7559
Epoch 32/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5837 - accuracy: 0.6877 - val_loss: 0.5386 - val_accuracy: 0.7559
Epoch 33/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5819 - accuracy: 0.6882 - val_loss: 0.5383 - val_accuracy: 0.7559
Epoch 34/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5805 - accuracy: 0.6894 - val_loss: 0.5392 - val_accuracy: 0.7552
Epoch 35/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5834 - accuracy: 0.6892 - val_loss: 0.5379 - val_accuracy: 0.7570
Epoch 36/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5760 - accuracy: 0.6945 - val_loss: 0.5354 - val_accuracy: 0.7581
Epoch 37/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5765 - accuracy: 0.7001 - val_loss: 0.5333 - val_accuracy: 0.7578
Epoch 38/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5769 - accuracy: 0.6961 - val_loss: 0.5335 - val_accuracy: 0.7570
Epoch 39/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5738 - accuracy: 0.6993 - val_loss: 0.5318 - val_accuracy: 0.7578
Epoch 40/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5735 - accuracy: 0.7001 - val_loss: 0.5303 - val_accuracy: 0.7581
Epoch 41/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5754 - accuracy: 0.6966 - val_loss: 0.5306 - val_accuracy: 0.7574
Epoch 42/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5724 - accuracy: 0.6969 - val_loss: 0.5296 - val_accuracy: 0.7574
Epoch 43/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5667 - accuracy: 0.7045 - val_loss: 0.5281 - val_accuracy: 0.7574
Epoch 44/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5681 - accuracy: 0.7024 - val_loss: 0.5283 - val_accuracy: 0.7556
Epoch 45/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5703 - accuracy: 0.6976 - val_loss: 0.5272 - val_accuracy: 0.7559
Epoch 46/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5705 - accuracy: 0.7047 - val_loss: 0.5269 - val_accuracy: 0.7570
Epoch 47/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5598 - accuracy: 0.7107 - val_loss: 0.5254 - val_accuracy: 0.7556
Epoch 48/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5690 - accuracy: 0.7016 - val_loss: 0.5239 - val_accuracy: 0.7567
Epoch 49/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5630 - accuracy: 0.7055 - val_loss: 0.5228 - val_accuracy: 0.7570
Epoch 50/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5629 - accuracy: 0.7048 - val_loss: 0.5213 - val_accuracy: 0.7574
Epoch 51/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5623 - accuracy: 0.7061 - val_loss: 0.5203 - val_accuracy: 0.7581
Epoch 52/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5613 - accuracy: 0.7129 - val_loss: 0.5196 - val_accuracy: 0.7585
Epoch 53/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5599 - accuracy: 0.7128 - val_loss: 0.5185 - val_accuracy: 0.7585
Epoch 54/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5608 - accuracy: 0.7072 - val_loss: 0.5174 - val_accuracy: 0.7600
Epoch 55/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5539 - accuracy: 0.7124 - val_loss: 0.5161 - val_accuracy: 0.7619
Epoch 56/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5576 - accuracy: 0.7084 - val_loss: 0.5154 - val_accuracy: 0.7626
Epoch 57/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5527 - accuracy: 0.7161 - val_loss: 0.5128 - val_accuracy: 0.7637
Epoch 58/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5566 - accuracy: 0.7091 - val_loss: 0.5114 - val_accuracy: 0.7648
Epoch 59/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5579 - accuracy: 0.7151 - val_loss: 0.5108 - val_accuracy: 0.7648
Epoch 60/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5526 - accuracy: 0.7131 - val_loss: 0.5111 - val_accuracy: 0.7648
Epoch 61/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5550 - accuracy: 0.7124 - val_loss: 0.5108 - val_accuracy: 0.7644
Epoch 62/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5537 - accuracy: 0.7111 - val_loss: 0.5090 - val_accuracy: 0.7644
Epoch 63/100
157/157 [==============================] - 1s 4ms/step - loss: 0.5502 - accuracy: 0.7095 - val_loss: 0.5089 - val_accuracy: 0.7630
Epoch 64/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5529 - accuracy: 0.7130 - val_loss: 0.5094 - val_accuracy: 0.7626
Epoch 65/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5454 - accuracy: 0.7210 - val_loss: 0.5065 - val_accuracy: 0.7652
Epoch 66/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5487 - accuracy: 0.7125 - val_loss: 0.5056 - val_accuracy: 0.7644
Epoch 67/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5471 - accuracy: 0.7168 - val_loss: 0.5054 - val_accuracy: 0.7644
Epoch 68/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5429 - accuracy: 0.7193 - val_loss: 0.5038 - val_accuracy: 0.7656
Epoch 69/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5437 - accuracy: 0.7198 - val_loss: 0.5043 - val_accuracy: 0.7648
Epoch 70/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5410 - accuracy: 0.7220 - val_loss: 0.5037 - val_accuracy: 0.7648
Epoch 71/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5479 - accuracy: 0.7151 - val_loss: 0.5030 - val_accuracy: 0.7663
Epoch 72/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5455 - accuracy: 0.7217 - val_loss: 0.5038 - val_accuracy: 0.7656
Epoch 73/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5446 - accuracy: 0.7221 - val_loss: 0.5007 - val_accuracy: 0.7681
Epoch 74/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5435 - accuracy: 0.7205 - val_loss: 0.5014 - val_accuracy: 0.7685
Epoch 75/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5438 - accuracy: 0.7226 - val_loss: 0.4995 - val_accuracy: 0.7696
Epoch 76/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5441 - accuracy: 0.7174 - val_loss: 0.4991 - val_accuracy: 0.7689
Epoch 77/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5465 - accuracy: 0.7171 - val_loss: 0.4972 - val_accuracy: 0.7719
Epoch 78/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5392 - accuracy: 0.7264 - val_loss: 0.4972 - val_accuracy: 0.7707
Epoch 79/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5373 - accuracy: 0.7208 - val_loss: 0.4956 - val_accuracy: 0.7726
Epoch 80/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5395 - accuracy: 0.7162 - val_loss: 0.4967 - val_accuracy: 0.7711
Epoch 81/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5368 - accuracy: 0.7248 - val_loss: 0.4954 - val_accuracy: 0.7733
Epoch 82/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5381 - accuracy: 0.7245 - val_loss: 0.4954 - val_accuracy: 0.7726
Epoch 83/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5371 - accuracy: 0.7212 - val_loss: 0.4941 - val_accuracy: 0.7730
Epoch 84/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5345 - accuracy: 0.7245 - val_loss: 0.4938 - val_accuracy: 0.7730
Epoch 85/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5363 - accuracy: 0.7237 - val_loss: 0.4923 - val_accuracy: 0.7726
Epoch 86/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5357 - accuracy: 0.7209 - val_loss: 0.4922 - val_accuracy: 0.7726
Epoch 87/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5358 - accuracy: 0.7269 - val_loss: 0.4918 - val_accuracy: 0.7726
Epoch 88/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5343 - accuracy: 0.7250 - val_loss: 0.4906 - val_accuracy: 0.7719
Epoch 89/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5319 - accuracy: 0.7313 - val_loss: 0.4903 - val_accuracy: 0.7722
Epoch 90/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5328 - accuracy: 0.7299 - val_loss: 0.4909 - val_accuracy: 0.7719
Epoch 91/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5348 - accuracy: 0.7233 - val_loss: 0.4903 - val_accuracy: 0.7715
Epoch 92/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5298 - accuracy: 0.7284 - val_loss: 0.4889 - val_accuracy: 0.7719
Epoch 93/100
157/157 [==============================] - 1s 6ms/step - loss: 0.5299 - accuracy: 0.7262 - val_loss: 0.4885 - val_accuracy: 0.7730
Epoch 94/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5311 - accuracy: 0.7267 - val_loss: 0.4885 - val_accuracy: 0.7726
Epoch 95/100
157/157 [==============================] - 1s 7ms/step - loss: 0.5304 - accuracy: 0.7274 - val_loss: 0.4875 - val_accuracy: 0.7741
Epoch 96/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5297 - accuracy: 0.7274 - val_loss: 0.4858 - val_accuracy: 0.7748
Epoch 97/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5315 - accuracy: 0.7284 - val_loss: 0.4876 - val_accuracy: 0.7737
Epoch 98/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5275 - accuracy: 0.7303 - val_loss: 0.4851 - val_accuracy: 0.7756
Epoch 99/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5223 - accuracy: 0.7325 - val_loss: 0.4828 - val_accuracy: 0.7770
Epoch 100/100
157/157 [==============================] - 1s 5ms/step - loss: 0.5248 - accuracy: 0.7313 - val_loss: 0.4829 - val_accuracy: 0.7767

Tune the threshold using ROC-AUC

threshold6 = best_threshold(model6)

314/314 [==============================] - 1s 2ms/step
Best Threshold=0.468450, G-Mean=0.765

Loss and accuracy

# Predicting target values (y_pred6) using 'model6' on the validation data (X_val).
y_pred6 = model6.predict(X_val)

# Applying a threshold to the predictions.
y_pred6 = (y_pred6 > threshold6)

graphs_and_report(history6, y_pred6)

85/85 [==============================] - 0s 2ms/step

Classification report

cr6 = metrics.classification_report(y_val, y_pred6,digits=3)
print(cr6)

              precision    recall  f1-score   support

           0      0.915     0.768     0.835      2147
           1      0.445     0.722     0.550       553

    accuracy                          0.759      2700
   macro avg      0.680     0.745     0.693      2700
weighted avg      0.818     0.759     0.777      2700

Confusion matrix

cm6 = confusion_matrix(y_val, y_pred6)
labels = ['True Negative', 'False Positive', 'False Negative', 'True Positive']
categories = ['Not_Exited', 'Exited']
make_confusion_matrix(cm6,
                      group_names=labels,
                      categories=categories,
                      cmap='Blues')

Observations

Precision (0): 0.915

• Precision measures the accuracy of positive predictions for class 0. In this case, it indicates that when the model predicts class 0, it's usually correct.

Recall (0): 0.768

• Recall (or Sensitivity) measures the ability of the model to correctly identify actual class 0 instances. It tells us what proportion of actual class 0 cases were correctly predicted.

F1-Score (0): 0.835

• The F1-Score is the harmonic mean of precision and recall for class 0. It provides a balance between precision and recall for this class.

Precision (1): 0.445

• Precision for class 1. Indicates how accurate the model's positive predictions are for class 1.

Recall (1): 0.722

• Recall for class 1. Measures the ability to correctly identify actual class 1 instances.

F1-Score (1): 0.550

• F1-Score for class 1. Similar to class 0, it balances precision and recall for class 1.

Final Model

for i in range(1,7):
  print(f'Model {i}'+'-'*20)
###  print(eval(f'cr{i}'))

Model 1--------------------
              precision    recall  f1-score   support

           0      0.839     0.959     0.895      2147
           1      0.645     0.286     0.396       553

    accuracy                          0.821      2700
   macro avg      0.742     0.623     0.646      2700
weighted avg      0.799     0.821     0.793      2700

Model 2--------------------
              precision    recall  f1-score   support

           0      0.924     0.705     0.799      2147
           1      0.403     0.774     0.530       553

    accuracy                          0.719      2700
   macro avg      0.663     0.739     0.665      2700
weighted avg      0.817     0.719     0.744      2700

Model 3--------------------
              precision    recall  f1-score   support

           0      0.911     0.821     0.863      2147
           1      0.497     0.687     0.577       553

    accuracy                          0.793      2700
   macro avg      0.704     0.754     0.720      2700
weighted avg      0.826     0.793     0.805      2700

Model 4--------------------
              precision    recall  f1-score   support

           0       0.92      0.73      0.81      2147
           1       0.42      0.76      0.54       553

    accuracy                           0.73      2700
   macro avg       0.67      0.74      0.68      2700
weighted avg       0.82      0.73      0.76      2700

Model 5--------------------
              precision    recall  f1-score   support

           0      0.922     0.816     0.866      2147
           1      0.506     0.732     0.598       553

    accuracy                          0.799      2700
   macro avg      0.714     0.774     0.732      2700
weighted avg      0.837     0.799     0.811      2700

Model 6--------------------
              precision    recall  f1-score   support

           0      0.915     0.768     0.835      2147
           1      0.445     0.722     0.550       553

    accuracy                          0.759      2700
   macro avg      0.680     0.745     0.693      2700
weighted avg      0.818     0.759     0.777      2700

Observations

Evaluating the models:

Model 1: It has a high precision for class 0 but a low recall for class 1, which suggests it may not be the best choice if you want to correctly identify class 1 instances.

Model 2: It has a higher recall for class 1 compared to Model 1, but its precision for class 1 is quite low. It also has a lower overall accuracy than Model 1.

Model 3: This model has a good balance between precision and recall for both classes, making it a reasonable choice. It also has a decent overall accuracy.

Model 4: While it has a high recall for class 1, its precision is low. The overall accuracy is also lower compared to other models.

Model 5: Model 5 shows a decent balance between precision and recall for both classes, along with a good overall accuracy.

Model 6: Model 6 has a higher recall for class 1, but its precision for class 1 is lower. The overall accuracy is also lower compared to some other models.

Conclusion

Based on the metrics and the priority of maximizing recall for class 1 (identifying positive cases), Model 5 might be a preferable choice. It has a higher recall for class 1 (0.687) compared to Model 3, which could be crucial if the goal is to minimize false negatives, even though it has slightly lower precision (0.497).

Model 5 offers a good balance between recall and overall accuracy.

Actionable Insights and Recommendations

What recommedations would you suggest to the bank?

Customer Segmentation:

• Segment customers based on age groups, account balances, and credit scores. Tailor marketing and service strategies for each segment. For example, offer retirement planning services for older customers and financial education for those with lower balances or credit scores.

Retention Programs:

• Develop customer retention programs that incentivize and reward active members. These programs could include loyalty rewards, discounts, or special offers for long-standing customers who actively use the bank's services.

Cross-Selling:

• Implement targeted cross-selling strategies to increase the number of products per customer. Identify complementary financial products and services to offer based on individual customer profiles. Ensure that the cross-selling approach is customer-centric and adds value.

Customer Feedback:

• Collect regular feedback from customers to understand their needs, concerns, and reasons for potential churn. Use surveys, focus groups, and digital feedback channels to gather insights.

Enhanced Customer Experience:

• Continuously improve the overall customer experience. Ensure that online and mobile banking platforms are user-friendly and provide seamless services. Address any issues related to account management, transactions, or customer support promptly.

---