Practical: RNN vs CNN with Fashion-MNIST¶

Anastasia Giachanou, Tina Shahedi

Machine Learning with Python - Utrecht Summer School

In this practical, we'll explore and understand the application of advanced deep learning models, including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), using the Fashion-MNIST dataset.

In [1]:
!pip install scikeras[tensorflow] > /dev/null 2>&1     # gpu compute platform
!pip install scikeras[tensorflow-cpu] > /dev/null 2>&1
!pip install scikeras > /dev/null 2>&1
!pip install pydot graphviz > /dev/null 2>&1
In [2]:
from scikeras.wrappers import KerasClassifier
In [5]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
import seaborn as sns
import random
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import plot_model
from tensorflow.keras.layers import Dense, Flatten, BatchNormalization, Dropout, Conv2D, MaxPooling2D, LSTM
from tensorflow.keras.regularizers import l2
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from tensorflow.keras.optimizers import *
from tensorflow.keras.optimizers import Adam, SGD, RMSprop
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import classification_report, confusion_matrix

Run the following lines to prepare the data for the models (the code was also used in the previous practical session).

In [4]:
# Load the dataset
fashion_mnist = tf.keras.datasets.fashion_mnist
(sample_images, sample_labels), (test_images, test_labels) = fashion_mnist.load_data()

# Set a random seed for reproducibility
np.random.seed(100)

# Randomly choose 15,000 indices from the range of train_images length
indices = np.random.choice(sample_images.shape[0], 15000, replace=False)

# Use these indices to sample images and labels
train_images = sample_images[indices]
train_labels = sample_labels[indices]

# Now sample_images and sample_labels contain your 15,000 samples
print("train_images shape:", train_images.shape)
print("train_labels shape:", train_labels.shape)

# Flatten the image data and convert it to a DataFrame
# The images are reshaped from 28x28 to 784 per image
train_images_flattened = train_images.reshape(train_images.shape[0], -1)
test_images_flattened = test_images.reshape(test_images.shape[0], -1)

# Create DataFrames
train_df = pd.DataFrame(train_images_flattened)
test_df = pd.DataFrame(test_images_flattened)

# Add labels to the DataFrames
train_df['label'] = train_labels
test_df['label'] = test_labels

# Normalize the pixel values to be between 0 and 1
X_train= train_images.astype('float32') / 255.0
X_test = test_images.astype('float32') / 255.0

# One-hot encode the labels
y_train = tf.keras.utils.to_categorical(train_labels, num_classes=10)
y_test = tf.keras.utils.to_categorical(test_labels, num_classes=10)

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
29515/29515 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
26421880/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
5148/5148 ━━━━━━━━━━━━━━━━━━━━ 0s 1us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
4422102/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
train_images shape: (15000, 28, 28)
train_labels shape: (15000,)

Lets refresh our memory regarding the structure and characteristics of the dataset.

In [6]:
fig, ax = plt.subplots(6, 6, figsize=(8, 8))
fig.suptitle('Fashion images and labels', fontsize=14)
ax = ax.ravel()  # This line ensuring ax is a flat array

for i in range(36):
    sample_n = random.randint(0, X_train.shape[0] - 1)
    ax[i].imshow((X_train[sample_n]).reshape(28, 28), cmap='Greys')
    ax[i].get_xaxis().set_visible(False)
    ax[i].get_yaxis().set_visible(False)
    label_index = np.argmax(y_train[sample_n])
    ax[i].set_title(label_index, fontsize=12)


plt.subplots_adjust(hspace=0.3)

As we can see, each training example is assigned to one of the following labels:

  1. T-shirt/top
  2. Trouser
  3. Pullover
  4. Dress
  5. Coat
  6. Sandal
  7. Shirt
  8. Sneaker
  9. Bag
  10. Ankle boot

Let's begin!¶

Recurrent Neural Networks¶

A Recurrent Neural Network (RNN) is typically used for sequential data like text. However, it can be adapted for image data such as the Fashion MNIST dataset by processing rows or columns of pixels as sequences. This allows the RNN to capture spatial information over sequences of the image. Long-short term memory (LSTMs), a specialized kind of RNN, are particularly effective for this purpose. They selectively remember patterns over long sequences, which can be useful for learning the nuances of fashion item images.

1. Reshape the X_train and X_test variables.

In [7]:
# Reshape for RNN input

x_train_rnn = X_train.reshape(X_train.shape[0], 28, 28)
x_test_rnn = X_test.reshape(X_test.shape[0], 28, 28)

2. Build a Sequential RNN model for Fashion-MNIST with Keras. Start with the Sequential() constructos and then add an LSTM layer of 64 units, followed by a Dropout layer of 0,25. Then repeat this LSTM-Dropout sequence with an LSTM layer of 32 units. End with a Dense layer of 16 units (relu activation) and a final Dense layer of 10 units (softmax activation) for classification. Compile with Adam optimizer, using categorical_crossentropy as the loss function.

In [9]:
# Define the RNN model architecture
model_rnn = Sequential()

# Adding LSTM layers and dropout to avoid overfitting
# Assuming X_train_shape is (num_samples, 28, 28)
model_rnn.add(LSTM(64, input_shape=(28, 28), activation='relu', return_sequences=True))
model_rnn.add(Dropout(0.25))
model_rnn.add(LSTM(32, activation='relu'))
model_rnn.add(Dropout(0.25))
model_rnn.add(Dense(16, activation='relu'))
model_rnn.add(Dropout(0.25))
model_rnn.add(Dense(10, activation='softmax'))

# Compile the model with the optimizer and define the loss and metrics
model_rnn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

3. Output the summary of the model and visualize the architecture of RNN model by using the plot_model function from keras.utils.

In [10]:
model_rnn.summary()
plot_model(model_rnn, to_file='model_rnn.png', show_shapes=True, show_layer_names=True, dpi=66)

Model: "sequential_1"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ lstm_2 (LSTM)                        │ (None, 28, 64)              │          23,808 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_3 (Dropout)                  │ (None, 28, 64)              │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ lstm_3 (LSTM)                        │ (None, 32)                  │          12,416 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_4 (Dropout)                  │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 16)                  │             528 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_5 (Dropout)                  │ (None, 16)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_3 (Dense)                      │ (None, 10)                  │             170 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 36,922 (144.23 KB)

 Trainable params: 36,922 (144.23 KB)

 Non-trainable params: 0 (0.00 B)
Out[10]:

4. Now it's time to train! Train (with fit()) the RNN model for 5 epochs and a batch size of 64, using X_train and y_train, and validate using X_test and y_test. Save the training history.

In [11]:
# Train the model
history = model_rnn.fit(
    x_train_rnn, y_train,
    epochs=5,
    batch_size=64,
    validation_data=(x_test_rnn, y_test))

tf.keras.backend.clear_session()

Epoch 1/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 24s 74ms/step - accuracy: 0.2720 - loss: 1.9128 - val_accuracy: 0.6557 - val_loss: 1.0647
Epoch 2/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 16s 54ms/step - accuracy: 0.5938 - loss: 1.1501 - val_accuracy: 0.7392 - val_loss: 0.7673
Epoch 3/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 46ms/step - accuracy: 0.6726 - loss: 0.9127 - val_accuracy: 0.7563 - val_loss: 0.6956
Epoch 4/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 26s 68ms/step - accuracy: 0.7233 - loss: 0.7844 - val_accuracy: 0.7894 - val_loss: 0.5784
Epoch 5/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 82ms/step - accuracy: 0.7507 - loss: 0.7057 - val_accuracy: 0.7794 - val_loss: 0.6058

5. Plot the model's accuracy and loss over epochs. Ypu can use the same function as in the previous practical

In [12]:
def plot_training_history(history):

    plt.figure(figsize=(12, 5))

    # Plotting accuracy
    plt.subplot(1, 2, 1)
    plt.plot(history.history['accuracy'], label='Training Accuracy')
    plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
    plt.title('Model Accuracy')
    plt.xlabel('Epoch')
    plt.ylabel('Accuracy')
    plt.legend(loc='lower right')

    # Plotting loss
    plt.subplot(1, 2, 2)
    plt.plot(history.history['loss'], label='Training Loss')
    plt.plot(history.history['val_loss'], label='Validation Loss')
    plt.title('Model Loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.legend(loc='upper right')

    plt.show()

plot_training_history(history)

6. Calculate the RNN model's accuracy on the training and test datasets.

In [13]:
# Evaluate the model on test dataset
loss, accuracy = model_rnn.evaluate(x_train_rnn, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model_rnn.evaluate(x_test_rnn, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))

Training Accuracy: 0.7912
Testing Accuracy:  0.7794

7. Train the RNN model for 10 epochs and compare the results.

In [14]:
# Define the RNN model architecture
model_rnn = Sequential()

# Adding LSTM layers and dropout to avoid overfitting
model_rnn.add(LSTM(64, input_shape=(28, 28), activation='relu', return_sequences=True))
model_rnn.add(Dropout(0.25))
model_rnn.add(LSTM(32, activation='relu'))
model_rnn.add(Dropout(0.25))
model_rnn.add(Dense(16, activation='relu'))
model_rnn.add(Dropout(0.25))
model_rnn.add(Dense(10, activation='softmax'))


# Compile the model with the optimizer and define the loss and metrics
model_rnn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Train the model
history = model_rnn.fit(
    x_train_rnn, y_train,
    epochs=10,
    batch_size=64,
    validation_data=(x_test_rnn, y_test))

tf.keras.backend.clear_session()

/usr/local/lib/python3.10/dist-packages/keras/src/layers/rnn/rnn.py:204: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(**kwargs)

Epoch 1/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 62ms/step - accuracy: 0.2940 - loss: 1.9034 - val_accuracy: 0.6368 - val_loss: 0.9610
Epoch 2/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 18s 50ms/step - accuracy: 0.5467 - loss: 1.1532 - val_accuracy: 0.7083 - val_loss: 0.7830
Epoch 3/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 15s 63ms/step - accuracy: 0.6273 - loss: 0.9682 - val_accuracy: 0.7412 - val_loss: 0.6960
Epoch 4/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 17s 50ms/step - accuracy: 0.6893 - loss: 0.8160 - val_accuracy: 0.7433 - val_loss: 0.6516
Epoch 5/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 49ms/step - accuracy: 0.7098 - loss: 0.7682 - val_accuracy: 0.7647 - val_loss: 0.6031
Epoch 6/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 21s 49ms/step - accuracy: 0.7394 - loss: 0.7073 - val_accuracy: 0.7838 - val_loss: 0.5553
Epoch 7/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 23s 60ms/step - accuracy: 0.7506 - loss: 0.6760 - val_accuracy: 0.7888 - val_loss: 0.5615
Epoch 8/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 55ms/step - accuracy: 0.7587 - loss: 0.6730 - val_accuracy: 0.7918 - val_loss: 0.5518
Epoch 9/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 49ms/step - accuracy: 0.7713 - loss: 0.6269 - val_accuracy: 0.8205 - val_loss: 0.4966
Epoch 10/10
235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 45ms/step - accuracy: 0.7833 - loss: 0.5947 - val_accuracy: 0.8169 - val_loss: 0.5220

8. Plot the results of the model with the 10 epochs.

In [15]:
plot_training_history(history)

Convolutional neural networks¶

Convolutional Neural Networks (CNNs), often referred to as convnets, have significantly transformed the landscape of machine learning, especially in image classification and computer vision. Their ability to autonomously extract and learn features from images has set new benchmarks in how machines interpret visual information.

9. Reshape Fashion MNIST images to include a channel dimension for CNN input, converting 2D arrays (28x28) into 3D arrays (28x28x1).

In [16]:
x_train_cnn = X_train.reshape(-1, 28, 28, 1)  # Add channel dimension for training set
x_test_cnn = X_test.reshape(-1, 28, 28, 1)   # Add channel dimension for test set

10. Build a CNN model with Keras for the Fashion-MNIST dataset. Reshape images for Conv2D compatibility, add a MaxPooling2D (pool size 2) and a Dropout layer (rate 0.25), flatten for dense layers, then use a 32-unit Dense layer and a 10-unit 'softmax' output layer. Compile with Adam optimizer and categorical_crossentropy.

In [18]:
# Define the model architecture
cnn_model = Sequential([
    Conv2D(filters=64, kernel_size=3, activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D(pool_size=2),
    Dropout(0.25),
    Flatten(),
    Dense(32, activation='relu'),
    Dropout(0.25),
    Dense(10, activation='softmax')
])

# Compile the model
cnn_model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['accuracy'])

11. Print the summary and use the plot_model function from keras.utils to create a visual representation of your CNN model's architecture.

In [19]:
cnn_model.summary()
#Visualize the CNN Model Architecture
plot_model(cnn_model, to_file='cnn_model.png', show_shapes=True, dpi=66)

Model: "sequential_1"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ conv2d_1 (Conv2D)                    │ (None, 26, 26, 64)          │             640 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ max_pooling2d_1 (MaxPooling2D)       │ (None, 13, 13, 64)          │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_2 (Dropout)                  │ (None, 13, 13, 64)          │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ flatten_1 (Flatten)                  │ (None, 10816)               │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 32)                  │         346,144 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_3 (Dropout)                  │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_3 (Dense)                      │ (None, 10)                  │             330 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 347,114 (1.32 MB)

 Trainable params: 347,114 (1.32 MB)

 Non-trainable params: 0 (0.00 B)
Out[19]:

12. Fit the model in 5 epochs with a batch size of 64. Save the training process in a variable named history.

In [20]:
# Train the model
history = cnn_model.fit(x_train_cnn, y_train, epochs=5, batch_size=64, validation_data=(x_test_cnn, y_test))
tf.keras.backend.clear_session()

Epoch 1/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 76ms/step - accuracy: 0.5874 - loss: 1.1778 - val_accuracy: 0.8041 - val_loss: 0.5260
Epoch 2/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 76ms/step - accuracy: 0.7824 - loss: 0.6100 - val_accuracy: 0.8353 - val_loss: 0.4638
Epoch 3/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 15s 65ms/step - accuracy: 0.8219 - loss: 0.4917 - val_accuracy: 0.8456 - val_loss: 0.4194
Epoch 4/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 63ms/step - accuracy: 0.8405 - loss: 0.4455 - val_accuracy: 0.8630 - val_loss: 0.3834
Epoch 5/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 15s 62ms/step - accuracy: 0.8466 - loss: 0.4200 - val_accuracy: 0.8716 - val_loss: 0.3642

13. Evaluate the accuracy of the model on the training and test data.

In [21]:
# Evaluate the model on the reshaped training dataset
loss, accuracy = cnn_model.evaluate(x_train_cnn, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
print('training loss: {:.4f}'.format(loss))

# Evaluate the model on the reshaped test dataset
loss, accuracy = cnn_model.evaluate(x_test_cnn, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
print('testing loss: {:.4f}'.format(loss))

Training Accuracy: 0.8955
training loss: 0.2878
Testing Accuracy:  0.8716
testing loss: 0.3642

Hyperparameter Optimization¶

14. Define a function named create_model that constructs a CNN model for the Fashion MNIST dataset. The function should accept the number of filters, kernel size, and dense layer size (embedding size) as input arguments. Utilize the structure of your prior CNN model, ensuring these parameters dynamically define the model's convolutional layers and dense layer accordingly.

In [22]:
def create_model(num_filters, kernel_size, dense_size):
    model = Sequential([
        Conv2D(filters=num_filters, kernel_size=kernel_size, activation='relu', input_shape=(28, 28, 1)),
        MaxPooling2D(pool_size=2),
        Dropout(0.2),
        Flatten(),
        Dense(dense_size, activation='relu'),
        Dense(10, activation='softmax')  # Output layer for 10 classes
    ])

    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

15. Use a Python dictionary to define a set of hyperparameters for your CNN model, including the number of filters, kernel size, and dense layer size. Implement this in your create_model function, allowing for dynamic model configuration.

In [23]:
param_grid = dict(num_filters=[16, 32, 64],
                  kernel_size=[3, 5, 7],
                  dense_size=[16, 32])

16. Use the KerasClassifier from scikeras and set model to create_model function. Set epochs = 5 and batch_size=64

In [28]:
model = KerasClassifier(model=create_model,
                  epochs = 5,
                  batch_size=64,
                  num_filters = 32,
                  kernel_size = 3,
                  dense_size = 32,
                  verbose=True)

17. Use RandomizedSearchCV together with the KerasClassifier model you created and the predefined hyperparameter grid. Set it up for 5-fold cross-validation. Set n_iter to 3 (we do that so the fitting part will not take a lot of time during the practical.)

In [29]:
grid = RandomizedSearchCV(
    estimator=model,
    param_distributions=param_grid,
    n_iter=3,
    cv=5,
    verbose=2,
    n_jobs=1  # Here is where n_jobs should be set
)

18. Fit the RandomizedSearchCV instance with x_train_cnn and y_train, initiating the search for the best hyperparameters.

In [30]:
grid_result = grid.fit(x_train_cnn, y_train)

Fitting 5 folds for each of 3 candidates, totalling 15 fits
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 58ms/step - accuracy: 0.6621 - loss: 1.0075
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 68ms/step - accuracy: 0.8398 - loss: 0.4585
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 18s 57ms/step - accuracy: 0.8612 - loss: 0.3889
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 58ms/step - accuracy: 0.8799 - loss: 0.3380
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 48ms/step - accuracy: 0.8913 - loss: 0.3175
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 20ms/step
[CV] END .......dense_size=32, kernel_size=3, num_filters=64; total time= 1.3min

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

Epoch 1/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 54ms/step - accuracy: 0.6431 - loss: 1.0179
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 56ms/step - accuracy: 0.8290 - loss: 0.4721
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 57ms/step - accuracy: 0.8605 - loss: 0.3944
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 58ms/step - accuracy: 0.8723 - loss: 0.3546
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 58ms/step - accuracy: 0.8872 - loss: 0.3152
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 12ms/step
[CV] END .......dense_size=32, kernel_size=3, num_filters=64; total time= 1.4min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 57ms/step - accuracy: 0.6709 - loss: 0.9716
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 56ms/step - accuracy: 0.8419 - loss: 0.4493
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 51ms/step - accuracy: 0.8673 - loss: 0.3810
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 56ms/step - accuracy: 0.8823 - loss: 0.3251
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 58ms/step - accuracy: 0.8934 - loss: 0.2956
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 12ms/step
[CV] END .......dense_size=32, kernel_size=3, num_filters=64; total time= 1.1min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 56ms/step - accuracy: 0.6399 - loss: 1.0399
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 58ms/step - accuracy: 0.8367 - loss: 0.4599
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 57ms/step - accuracy: 0.8700 - loss: 0.3595
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 55ms/step - accuracy: 0.8892 - loss: 0.3151
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 57ms/step - accuracy: 0.8988 - loss: 0.2879
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 12ms/step
[CV] END .......dense_size=32, kernel_size=3, num_filters=64; total time= 1.6min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 57ms/step - accuracy: 0.6557 - loss: 1.0070
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 19s 52ms/step - accuracy: 0.8460 - loss: 0.4326
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 56ms/step - accuracy: 0.8646 - loss: 0.3731
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 57ms/step - accuracy: 0.8846 - loss: 0.3265
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 19s 51ms/step - accuracy: 0.9011 - loss: 0.2880
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 12ms/step
[CV] END .......dense_size=32, kernel_size=3, num_filters=64; total time= 1.2min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 41ms/step - accuracy: 0.5636 - loss: 1.2130
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 6s 33ms/step - accuracy: 0.8057 - loss: 0.5402
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 33ms/step - accuracy: 0.8331 - loss: 0.4689
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 43ms/step - accuracy: 0.8477 - loss: 0.4267
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 6s 32ms/step - accuracy: 0.8577 - loss: 0.3962
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
[CV] END .......dense_size=16, kernel_size=5, num_filters=32; total time=  44.5s
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.5854 - loss: 1.2120
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 42ms/step - accuracy: 0.8014 - loss: 0.5460
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 36ms/step - accuracy: 0.8258 - loss: 0.4771
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 32ms/step - accuracy: 0.8477 - loss: 0.4312
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 33ms/step - accuracy: 0.8644 - loss: 0.3889
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
[CV] END .......dense_size=16, kernel_size=5, num_filters=32; total time=  45.0s
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 42ms/step - accuracy: 0.5498 - loss: 1.3692
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 35ms/step - accuracy: 0.8058 - loss: 0.5377
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 33ms/step - accuracy: 0.8321 - loss: 0.4798
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 41ms/step - accuracy: 0.8473 - loss: 0.4295
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 6s 33ms/step - accuracy: 0.8533 - loss: 0.3957
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
[CV] END .......dense_size=16, kernel_size=5, num_filters=32; total time=  46.7s
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.5895 - loss: 1.2630
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 33ms/step - accuracy: 0.8152 - loss: 0.5238
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 33ms/step - accuracy: 0.8513 - loss: 0.4340
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 41ms/step - accuracy: 0.8621 - loss: 0.3801
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 9s 33ms/step - accuracy: 0.8791 - loss: 0.3521
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
[CV] END .......dense_size=16, kernel_size=5, num_filters=32; total time=  49.8s
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.4957 - loss: 1.3898
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 8s 41ms/step - accuracy: 0.8005 - loss: 0.5812
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 42ms/step - accuracy: 0.8383 - loss: 0.4635
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 6s 33ms/step - accuracy: 0.8516 - loss: 0.4148
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 37ms/step - accuracy: 0.8658 - loss: 0.3850
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step
[CV] END .......dense_size=16, kernel_size=5, num_filters=32; total time=  43.7s
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 63ms/step - accuracy: 0.4709 - loss: 1.3980
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 63ms/step - accuracy: 0.7866 - loss: 0.6026
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 62ms/step - accuracy: 0.8321 - loss: 0.4744
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 60ms/step - accuracy: 0.8510 - loss: 0.4244
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 63ms/step - accuracy: 0.8631 - loss: 0.3822
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step
[CV] END .......dense_size=16, kernel_size=7, num_filters=64; total time= 1.3min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 63ms/step - accuracy: 0.5566 - loss: 1.2603
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 64ms/step - accuracy: 0.7993 - loss: 0.5443
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 59ms/step - accuracy: 0.8371 - loss: 0.4528
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 58ms/step - accuracy: 0.8556 - loss: 0.4074
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 22s 64ms/step - accuracy: 0.8675 - loss: 0.3742
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step
[CV] END .......dense_size=16, kernel_size=7, num_filters=64; total time= 1.2min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 62ms/step - accuracy: 0.4659 - loss: 1.4846
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 61ms/step - accuracy: 0.7750 - loss: 0.6309
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 10s 54ms/step - accuracy: 0.8248 - loss: 0.4856
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 62ms/step - accuracy: 0.8489 - loss: 0.4257
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 62ms/step - accuracy: 0.8548 - loss: 0.4029
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step
[CV] END .......dense_size=16, kernel_size=7, num_filters=64; total time= 1.1min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 62ms/step - accuracy: 0.6034 - loss: 1.1342
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 63ms/step - accuracy: 0.8241 - loss: 0.5024
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 11s 56ms/step - accuracy: 0.8500 - loss: 0.4244
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 22s 63ms/step - accuracy: 0.8658 - loss: 0.3703
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 62ms/step - accuracy: 0.8736 - loss: 0.3495
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 14ms/step
[CV] END .......dense_size=16, kernel_size=7, num_filters=64; total time= 1.5min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

188/188 ━━━━━━━━━━━━━━━━━━━━ 13s 64ms/step - accuracy: 0.3407 - loss: 1.7258
Epoch 2/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 63ms/step - accuracy: 0.6713 - loss: 0.8642
Epoch 3/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 12s 63ms/step - accuracy: 0.8078 - loss: 0.5349
Epoch 4/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 20s 60ms/step - accuracy: 0.8423 - loss: 0.4519
Epoch 5/5
188/188 ━━━━━━━━━━━━━━━━━━━━ 21s 63ms/step - accuracy: 0.8600 - loss: 0.4015
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 14ms/step
[CV] END .......dense_size=16, kernel_size=7, num_filters=64; total time= 1.5min
Epoch 1/5

/usr/local/lib/python3.10/dist-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

235/235 ━━━━━━━━━━━━━━━━━━━━ 19s 74ms/step - accuracy: 0.6703 - loss: 0.9373
Epoch 2/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 13s 54ms/step - accuracy: 0.8559 - loss: 0.4058
Epoch 3/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 21s 56ms/step - accuracy: 0.8824 - loss: 0.3457
Epoch 4/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 20s 52ms/step - accuracy: 0.8902 - loss: 0.3106
Epoch 5/5
235/235 ━━━━━━━━━━━━━━━━━━━━ 12s 51ms/step - accuracy: 0.9048 - loss: 0.2677

19. Identify and note the best score and hyperparameters obtained from the RandomizedSearchCV results.

In [31]:
print(grid_result.best_score_)
print(grid_result.best_params_)

0.8768
{'num_filters': 64, 'kernel_size': 3, 'dense_size': 32}

20. Evaluate the tuned model on the test set (X_test, y_test) to measure its final performance.

In [32]:
test_accuracy = grid.score(x_test_cnn, y_test)
test_accuracy

157/157 ━━━━━━━━━━━━━━━━━━━━ 2s 12ms/step
Out[32]:
0.8796

Predict¶

21. Predict the labels for the test dataset using the trained model.

In [38]:
# Predict
predictions = grid_result.predict(x_test_cnn)

157/157 ━━━━━━━━━━━━━━━━━━━━ 3s 19ms/step

Evaluating Model¶

22. Evaluate the performance of the model using a classification report and confusion matrix.

In [40]:
# Convert predictions and labels from one-hot to labels
y_pred = np.argmax(predictions, axis=1)
y_true = np.argmax(y_test, axis=1)

# Generate a classification report
print(classification_report(y_true, y_pred, target_names=fashion_labels))

# Optional: Confusion matrix
conf_matrix = confusion_matrix(y_true, y_pred)

# Plotting the confusion matrix
plt.figure(figsize=(12, 10))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=fashion_labels, yticklabels=fashion_labels)
plt.title('Confusion Matrix')
plt.xlabel('Predicted Labels')
plt.ylabel('True Labels')
plt.show()

              precision    recall  f1-score   support

 T-shirt/top       0.85      0.81      0.83      1000
     Trouser       0.98      0.96      0.97      1000
    Pullover       0.89      0.71      0.79      1000
       Dress       0.83      0.93      0.88      1000
        Coat       0.78      0.83      0.81      1000
      Sandal       0.94      0.98      0.96      1000
       Shirt       0.67      0.71      0.69      1000
     Sneaker       0.94      0.94      0.94      1000
         Bag       0.97      0.96      0.96      1000
  Ankle boot       0.97      0.95      0.96      1000

    accuracy                           0.88     10000
   macro avg       0.88      0.88      0.88     10000
weighted avg       0.88      0.88      0.88     10000

Optional. Visualize predictions of a model on the Fashion MNIST dataset by plotting images with predicted and true labels, highlighting correct and incorrect predictions.

In [39]:
def plot_image(i, predictions_array, true_label, img):
    true_label, img = np.argmax(true_label), img.reshape(28, 28)
    plt.grid(False)
    plt.xticks([])
    plt.yticks([])

    plt.imshow(img, cmap=plt.cm.binary)

    predicted_label = np.argmax(predictions_array)
    if predicted_label == true_label:
        color = 'blue'
    else:
        color = 'red'

    plt.xlabel("{} {:2.0f}% ({})".format(fashion_labels[predicted_label],
                                         100*np.max(predictions_array),
                                         fashion_labels[true_label]),
               color=color)

# Mapping classes to labels for Fashion MNIST
fashion_labels = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
                  "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]

# Plotting a few predictions
num_rows = 4
num_cols = 4
num_images = num_rows*num_cols
plt.figure(figsize=(1.5*1.7*num_cols, 2*num_rows))
for i in range(num_images):
    plt.subplot(num_rows, 2*num_cols, 2*i+1)
    plot_image(i, predictions[i], y_test[i], x_test_cnn[i])
plt.tight_layout()
plt.show()

End of practical!