Cross Entropy Loss

Introduction

The Cross Entropy Loss is a widely used measure to evaluate the difference between two probability distributions. In the context of neural networks, it is primarily used to assess the performance of classification models, particularly in multi-class classification tasks.

Mathematical Formula

The cross entropy loss function is mathematically defined as follows:

\[L(y, \hat{y}) = -\sum_{i=1}^{N} y_i \log(\hat{y}_i)\]

where:

• \(y\) represents the vector of true labels (a probability distribution where only one class is assigned a value of 1, and the others are 0).

• \(\hat{y}\) represents the vector of model predictions (the predicted probabilities for each class).

• \(N\) is the number of classes.

Meaning and Concept Behind Cross Entropy

The central idea behind Cross Entropy is to quantify the “disagreement” between the true and predicted probability distributions. If the probabilities predicted by the model are close to the true probabilities (the labels), the loss will be low. Conversely, if the predictions are far from the true labels, the loss will be high. This measure encourages the model to refine its predictions to better match the observed truths.

Interpretation: A cross entropy loss of zero means the model has perfectly predicted the correct class for all examples, while a high loss indicates that the model often assigned low probabilities to the correct classes.

Usage Example

Here’s a simple Python example demonstrating how to calculate the Cosine Similarity between two vectors using the distancia package:

from distancia import CrossEntropy

# Exemple de données
y_true = [[1, 2], [0, 1]]  # Étiquettes réelles
y_pred = [[[0.1, 0.2, 0.7], [0.3, 0.4, 0.3]],
      [[0.5, 0.2, 0.3], [0.6, 0.3, 0.1]]]  # Prédictions du modèle

# Créer une instance de CrossEntropy
loss_fn = CrossEntropy()

# Calculer la perte
loss_value = loss_fn(y_true, y_pred)
print(f'Cross Entropy Loss: {loss_value}')

>>>Cross Entropy Loss: -0.2534601641198856

Here is a second example of more developed neural classification

#source : https://keras.io/examples/nlp/text_classification_with_transformer/

!pip install distancia

import keras
from keras import ops
from keras import layers
from distancia import *

class TransformerBlock(layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):
        super().__init__()
        self.att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)
        self.ffn = keras.Sequential(
        [layers.Dense(ff_dim, activation="relu"), layers.Dense(embed_dim),]
        )
        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = layers.Dropout(rate)
        self.dropout2 = layers.Dropout(rate)

    def call(self, inputs):
        attn_output = self.att(inputs, inputs)
        attn_output = self.dropout1(attn_output)
        out1 = self.layernorm1(inputs + attn_output)
        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output)
        return self.layernorm2(out1 + ffn_output)

class TokenAndPositionEmbedding(layers.Layer):
    def __init__(self, maxlen, vocab_size, embed_dim):
        super().__init__()
        self.token_emb = layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)
        self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=embed_dim)

    def call(self, x):
        maxlen = ops.shape(x)[-1]
        positions = ops.arange(start=0, stop=maxlen, step=1)
        positions = self.pos_emb(positions)
        x = self.token_emb(x)
        return x + positions
import numpy as np
import tensorflow as tf
from tensorflow import keras



vocab_size = 20000  # Only consider the top 20k words
maxlen = 200  # Only consider the first 200 words of each movie review
(x_train, y_train), (x_val, y_val) = keras.datasets.imdb.load_data(num_words=vocab_size)
print(len(x_train), "Training sequences")
print(len(x_val), "Validation sequences")

x_train = keras.utils.pad_sequences(x_train, maxlen=maxlen)
x_val = keras.utils.pad_sequences(x_val, maxlen=maxlen)

embed_dim = 32  # Embedding size for each token
num_heads = 2  # Number of attention heads
ff_dim = 32  # Hidden layer size in feed forward network inside transformer

inputs = layers.Input(shape=(maxlen,))
embedding_layer = TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim)
x = embedding_layer(inputs)
transformer_block = TransformerBlock(embed_dim, num_heads, ff_dim)
x = transformer_block(x)
x = layers.GlobalAveragePooling1D()(x)
x = layers.Dropout(0.1)(x)
x = layers.Dense(20, activation="relu")(x)
x = layers.Dropout(0.1)(x)
outputs = layers.Dense(2, activation="softmax")(x)

model = keras.Model(inputs=inputs, outputs=outputs)

loss_crossentropy=CrossEntropyLoss()
model.compile(optimizer="adam", loss=custom_loss_crossentropy, metrics=["accuracy"])
history = model.fit(x_train, y_train, batch_size=32, epochs=3, validation_data=(x_val, y_val))

import matplotlib.pyplot as plt

# Visualisation des courbes de précision
plt.figure(figsize=(12, 6))

plt.plot(history.history['accuracy'], label='Cross Entropy')
plt.title('Accuracy Distancia')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()

plt.show()

History and Context

Cross entropy has its roots in information theory, introduced by Claude Shannon in 1948. In this context, entropy measures the uncertainty of an information source. Cross entropy, on the other hand, measures the divergence between two probability distributions, thus linking concepts from information theory to modern machine learning.

The use of cross entropy as a loss function gained popularity with the development of artificial neural networks and is now one of the standard loss functions for classification tasks.

Academic Reference

For a deeper understanding, you can refer to the foundational work by Claude Shannon on information theory Shannon[1]:

[1]

C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423, 1948. doi:10.1002/j.1538-7305.1948.tb01338.x.

Conclusion

The Cross Entropy Loss is an essential loss function for classification models in machine learning. It guides the model by providing an error signal based on the divergence between the true and predicted distributions. Understanding and using this loss function is fundamental to developing effective and accurate classification models.