Cost Functions and Distances - 8. Sequence and Structured Prediction Losses

8. Sequence and Structured Prediction Losses

Sequence and structured prediction losses are used when the output is a sequence or a structured object (e.g., a sentence, a parse tree, a segmentation mask). These losses must account for the dependencies between the output elements. The most common sequence losses are Connectionist Temporal Classification (CTC), Sequence Cross-Entropy, and CRF Loss.

8.1 Connectionist Temporal Classification (CTC)

Connectionist Temporal Classification (CTC) is a loss function used for sequence labeling tasks where the alignment between the input and output sequences is unknown. It is widely used in speech recognition and handwriting recognition. CTC defines a loss that marginalizes over all possible alignments:

where $\pi$ is an alignment path, $B$ is the function that removes repeated and blank tokens, and $T$ is the length of the input sequence.

Mathematical properties: CTC is a powerful loss for sequence labeling tasks where the alignment is unknown. It is used in speech recognition, handwriting recognition, and other sequence-to-sequence tasks. The CTC loss is differentiable and can be optimized with gradient descent.

Real-World Applications: Speech recognition (CTC is widely used in end-to-end Automatic Speech Recognition (ASR) systems), handwriting recognition (transcribing handwritten text from images), keyword spotting (CTC loss is used with triplet loss to learn word embeddings for keyword spotting), and multilingual ASR (CTC-DRO addresses language disparities in speech recognition across multiple languages).

8.2 Sequence Cross-Entropy (Teacher Forcing)

Sequence Cross-Entropy (also called Teacher Forcing) is the standard loss for sequence generation tasks. The model predicts the next token in the sequence given the previous tokens, and the loss is the cross-entropy between the predicted distribution and the true token:

During training, the model is fed the ground-truth previous tokens (teacher forcing).

Mathematical properties: Sequence cross-entropy is the standard loss for training autoregressive models. It is simple, differentiable, and effective.

Real-World Applications: Machine translation (the standard loss for training encoder-decoder models like Transformers), text generation (used in language models like GPT), and image captioning (training models to generate descriptions of images).

8.3 CRF Loss

The Conditional Random Field (CRF) Loss is used for structured prediction tasks where the output has a known structure, such as a sequence of labels in named entity recognition or a segmentation mask. The CRF loss models the conditional probability of the output given the input and encourages the output to respect the structural constraints.

Mathematical properties: The CRF loss is a structured prediction loss that captures the dependencies between output labels. It is used in tasks where the output has a known structure, such as part-of-speech tagging, named entity recognition, and semantic segmentation. The CRF loss is differentiable and can be optimized with gradient descent.

Real-World Applications: Named entity recognition (CRF loss is used in BERT-BiGRU-CRF and BiLSTM-CRF models to capture long-distance dependencies and enhance entity boundary recognition accuracy), part-of-speech tagging (capturing dependencies between tags in a sequence), and semantic segmentation (used with CRFs as a post-processing step or as a differentiable loss).

8.4 Connectionist Temporal Classification (CTC) vs. Cross-Entropy

The choice between CTC and cross-entropy depends on the task. CTC is preferred when the alignment between the input and output is unknown, as in speech recognition, where the length of the input spectrogram is much longer than the output text. Cross-entropy is preferred when the alignment is known, as in machine translation, where the input and output sequences have a known correspondence.

Applications

• Speech recognition (CTC).
• Machine translation (Sequence Cross-Entropy).
• Named entity recognition (CRF Loss).
• Handwriting recognition (CTC).

Strengths and limitations

Table 10: Strengths and limitations of sequence losses. CTC handles unknown alignments (e.g., speech) but assumes monotonic alignment. Sequence cross‑entropy (teacher forcing) is simple and effective but can cause exposure bias during inference. CRF loss captures output dependencies but is computationally heavy for long sequences.