A survey of recent LLM Research Papers

The rapidly evolving field of Large Language Models (LLMs) has seen a surge of new research recently. With the AI community constantly exploring new possibilities, it can be challenging to stay updated on the latest developments. In this post, I’m sharing my key takeaways from a deep dive into recent LLM research, highlighting some of the emerging trends.

I have categorized the papers into six themes: LLM Augmentation, LLM Pretraining, LLM Post-training, LLM Alignment, LLM Auditing and LLM Compression. We will review recent papers from each of these themes.

Theme 1: LLM Augmentation

LLM Augmentation is the process of expanding knowledge of LLMs with access to external tools or data. We will look at a couple of recent papers related to Augmentation below.

1) Retrieval-Augmented Generation for Large Language Models: A Survey

LLMs demonstrate significant capabilities but face challenges such as hallucination, outdated knowledge, and nontransparent reasoning processes. Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases. This paper does a thorough and systematic review of the state-of-the-art RAG.

Benefits of RAG

• Enhances the accuracy and credibility of the models
• Reduces hallucinations by grounding responses in retrieved, factual data
• Allows for continuous knowledge updates and integration of domain-specific information.
  

Steps involved

The figure below shows an example of RAG applied to Question Answering. The steps involved are:

• Indexing: offline step with external data: cleaning, segmentation/chunking, vector representation, index storking text chunks and vector representation as key-value pairs
  
• Retrieval: convert user input to vector representation, compute similarity score between query vector and vectorized chunks within the indexed corpus, prioritize and retrieve the top K chunks
• Generation: posed query and selected documents are synthesized into a coherent prompt to which the LLM responds

The paper categorizes RAG into three paradigms: Naive RAG, Advanced RAG, and Modular RAG. It also explores the key components of RAG—retriever, generator, and augmentation methods—along with their optimizations, and highlights the latest research for each component.

2) LLM Augmented LLMs: Expanding Capabilities through Composition

This paper by Google asks the key question – can we augment LLMs with other LLMs? i.e. is it possible to combine a general-purpose “anchor” model with a specialized model to create new abilities? For example, could the code understanding ability of one model be merged with the language generation skill of another to facilitate code-to-text generation?

A typical approach for this problem involves further pre-training or fine-tuning the anchor model on the data that was originally used to train the augmenting model. However, this is not always practical since training large models is computationally expensive. To mitigate this, authors propose CALM (Composition to Augment Language Models) which:

• Augments an anchor LLM (m_B) with new capabilities with a specialized augmenting (smaller) model (m_A).
• Introduces cross-attention between models to “compose” their representations (learn from each other’s representations) 
• Enables new capabilities not achievable by either models individually

The figure below illustrates two augmenting models (m_A) with different capabilities: low-resource languages (left), and code (right).

• Example 1: Low Resource Language (left)
  • Compose a large anchor language model m_B with a smaller augmenting model m_A that has been pre-trained on low-resource languages
  • Outcome: the combined language model is able to perform translation and math-word problem solving tasks presented in these low-resource languages.

• Example 2: Code (right) 
  • LLMs (m_B) could often lack the specific knowledge of code syntax due to a skewed representation of code data in their pre-training corpora. 
  • Conversely, small models (m_A) trained specifically on code data could exhibit a good understanding of code syntax, but they may lack broad world knowledge and reasoning.
  • CALM can enable best of both worlds by combining capabilities of these models.

The goal of CALM is to learn a composition m_A⊕B = f(m_A, m_B, Θ_C, D_C) to achieve some joint task C. The weights of m_A and m_B are frozen. Θ_C are the additional trainable parameters introduced to learn the composition, while D_C is the set of examples that are used to learn this composition. The main benefits of CALM are:

• CALM is effective in scenarios where there’s a need to leverage specialized knowledge stored in different models. For instance, a foundational LLM can be augmented with models containing proprietary data or expertise.
  
• Ability to merge distinct knowledge from multiple models without the need to update the individual models. Can apply to diverse domains and settings.

Theme 2: LLM Pretraining

Scaling Data-Constrained Language Models

Scaling language models usually means increasing both their size and the amount of training data. However, as the amount of text data available online becomes a limiting factor, the authors explore how to scale models when data is limited.

The authors conducted experiments with varying amounts of data repetition and compute resources, using up to 900 billion training tokens and 9 billion parameter models. They found that with limited data, training with up to 4 epochs of repeated data yields negligible changes to
loss compared to having unique data. However, adding more compute resources has diminishing returns when data is highly repeated.

The authors propose a new rule for optimizing compute usage based on the diminishing value of repeated tokens and excess parameters. They also tested methods to handle data scarcity, such as adding code data to the training set or removing common filters. Check out this video for a more thorough deep dive into this paper.

Theme 3: LLM Post-training

1) Unlimiformer: Long-Range Transformers with Unlimited Length Input

Transformer models have been limited to bounded input lengths, because of their need to attend to every token in the input (context window). In this work, the authors propose Unlimiformer: a novel method to extend pretrained encoder-decoder transformers to handle virtually unlimited input lengths.

Unlimiformer uses a k-nearest-neighbor (kNN) index to manage cross-attention, allowing each decoder layer to attend to selected tokens from the entire input sequence rather than being restricted to a fixed window. This approach enables processing of extremely long documents, such as those found in book summarization tasks, without truncation.

The paper highlights Unlimiformer’s effectiveness across various benchmarks, showcasing its superior performance in summarizing lengthy documents compared to other long-range transformers.

2) Evaluating and Inducing Personality in Pre-trained Language Models

Evaluating LLMs’ Personality

LLMs are trained on numerous textual corpora and have the potential to exhibit various personalities. The authors evaluate LLMs’ personality using their “Machine Personality Inventory” (MPI) Dataset for five key traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. This is a multiple-choice question answering examples where the LLM must choose how accurately statements such as “You love to help others” describe it, and each statement is associated with a personality trait.

Do LLMs exhibit a consistent personality? The authors find that only the largest models exhibit consistent personality traits that are similar to those of humans: GPT-3.5 175B and Alpaca 7B attain human-level internal consistency across all five factors. In comparison, other vanilla models with fewer parameters lack stable personalities.

Inducing LLMs’ Personality

The authors introduce the Personality Prompting method to induce LLMs with a certain personality in a controllable manner. 3 steps:

• Given a trait, construct Naive prompt
• The naive prompt is transformed into a keyword prompt by utilizing trait descriptive words 
• Using chain-of-thought prompting method, self-prompt the target LLM to generate short descriptive sentences of people with these traits invoking its internal knowledge

With Extraversion as the target trait, they facilitate the transformation of the intuitive naive prompt into a collection of keywords. These words accurately convey the personality traits of an extraverted individual, more specific and understandable for LLMs. Next, a keyword prompt leveraging these feature words is constructed and passed to LLMs to initiate a brief description of Extraversion as the personality prompt. While human-designed prompts are empirical or rely on trial and error, this methof takes advantage of LLMs’ internal knowledge of Extraversion and is, therefore, more suited for the model.

The authors show that compared with following two baselines: Naive Prompting and Words Auto Prompting, Personality Prompting does a better job and the induced personality is generally more stable than neutral in terms of internal consistence.

Theme 4: LLM Alignment

LLM Alignment refers to the process of ensuring that large language models (LLMs) act in ways that are safe, ethical, and aligned with human values and expectations.

Reinforcement Learning from Human Feedback (RLHF) is a popular technique for LLM Alignment, where the LLM learns directly from human feedback. However, RLHF has several limitations – the paper Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback discusses this in detail. The first two papers below propose approaches which improve upon RLHF.

1) Direct Preference Optimization: Your Language Model is Secretly a Reward Model

This paper from Stanford has been out for a few months, but it’s significant enough that I wanted to include it in this post. The paper proposed Direct Preference Optimization (DPO) for LLM Alignment.

Traditional methods for fine-tuning language models with human feedback involve two steps: first, creating a reward model based on human preferences for different responses, and then using reinforcement learning (RL) to adjust the model to maximize this reward.

In contrast, Direct Preference Optimization (DPO) simplifies this process by directly finding the best policy to match human preferences using a straightforward classification task. This approach fits an implicit reward model and allows the optimal policy to be extracted directly.

The authors show that DPO is often a better approach than RLHF, since the responses generated by DPO align more closely with user preferences. This video does a great job of explaining DPO in more detail.

2) Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Reinforcement Learning from Human Feedback (RLHF) uses human feedback to optimize learning in LLMs. However, traditional RLHF conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference. To mitigate this, the paper proposes Fine-grained RLHF, which provides a reward after every sentence is generated; and incorporates multiple reward models associated with different feedback types.

The figure above shows a comparison of Reinforcement Learning with Preference Based RLHF and Fine-Grained RLHF. In Preference Based RLHF, we collect human preferences on the overall quality of LM outputs. In Fine-Grained RLHF:

• the authors ask annotators to mark which part of an output contains what type(s) of errors.
• they then train a fine-grained reward model for each type of error and optimize LM against these reward models.
• they provide a relevance reward and a factuality reward after each sentence is generated. 
• there is also a holistic information completeness reward after the whole text is generated

The authors show that Fine-Grained RLHF outperforms Preference RLHF – it leads to generations that are much more factually correct and contains more complete information, and generates fewer irrelevance, repetition, and incoherence errors.

3) Large Language Model Alignment: A Survey

The paper presents various methodologies and challenges related to aligning LLMs with human values. It categorizes alignment methods into outer and inner alignment. Outer alignment focuses on aligning the model’s training objectives with human values through techniques like reinforcement learning from human feedback (RLHF). Inner alignment ensures that the model’s internal optimization processes are consistent with the with the intended goals.

The paper also discusses mechanistic interpretability, which aims to understand the internal workings of LLMs to improve alignment. Additionally, it discusses potential risks like the model’s susceptibility to adversarial attacks.

The authors emphasize the importance of developing robust evaluation methodologies to assess alignment effectiveness and suggest future research directions, including automated alignment, dynamic evaluation, and bridging the gap between LLM and broader AI alignment research.

Theme 5: LLM Auditing

Detecting Pretraining Data from Large Language Models

In this paper, the authors study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model
was trained on the provided text?

In this regard, the paper presents MIN-K% PROB, a simple and effective method that can detect whether a large language model (e.g., GPT-3) was pretrained on the provided text without knowing the pretraining data. It also introduces the WIKIMIA benchmark, which distinguishes between seen and unseen data using Wikipedia events created before and after model training.

To determine whether a text X is in the pretraining data of a LLM such as GPT, MIN-K% PROB first gets the probability for each token in X, selects the k% tokens with minimum probabilities and calculates their average log likelihood. If the average log likelihood is high, the text is likely in the pretraining data.

The experiments in the paper demonstrate that MIN-K% PROB achieves a 7.4% improvement on WIKIMIA over previous methods. The authors apply MIN-K% PROB to three real-world scenarios,
copyrighted book detection, detecting dataset contamination and privacy auditing, and find it a consistently effective solution.

Theme 6: LLM Compression

A Survey on Model Compression for Large Language Models

LLMs formidable size and computational demands present significant challenges for practical deployment, especially in resource-constrained environments. The field of model compression has emerged as a pivotal research area to alleviate these limitations. This paper presents a comprehensive survey that navigates the landscape of model compression techniques tailored specifically for LLMs. Within each of these techniques, the authors highlight recent advancements and innovative approaches

LLM Pruning: A powerful technique to reduce the size or complexity of a model by removing unnecessary or redundant components or parameters. Structured pruning removes connections or hierarchical structures based on specific rules while preserving the overall network structure. Unstructured pruning prunes individual parameters, resulting in an irregular sparse structure.

LLM Knowledge Distillation:  Aimed at improving model performance and generalization by transferring knowledge from a complex model, referred to as the teacher model, to a simpler counterpart known as the student model.

LLM Quantization: LLMs like GPT-3 typically store their parameters as floating-point values: each parameter occupies two bytes, leading to a model the size of GPT-3 requiring hundreds of gigabytes of memory. Quantization, a compression technique, converts these parameters into single-byte or smaller integers, significantly reducing the size of an LLM.

LLM Low-Rank Factorization: Low-Rank Factorization is a model compression technique that aims to approximate a given weight matrix by decomposing it into two or more smaller matrices with significantly lower dimensions. In the field of LLM research, low-rank factorization has been widely adopted to fine-tune LLMs efficiently.

Other Interesting Papers

Specializing Smaller Language Models towards Multi-Step Reasoning, Fu et al., 2023

The surprising ability of LLMs to perform well on complex reasoning with only few-shot chain-of-thought prompts is believed to emerge only in very large-scale models. The authors show that such abilities can be distilled down from GPT-3.5 (≥ 175B) to T5 variants (≤ 11B) through model specialization, to specialize the model’s ability towards a target task.

Small models (commonly viewed as smaller than 10B) have limited model capacity, but if we specialize their capacity towards a target task, the model can achieve decent performance improvements.

Fine-Tuning Language Models with Just Forward Passes, Malladi et al., 2023

LLM fine-tuning process typically involves feeding the task-specific dataset to the pre-trained model and adjusting its parameters through backpropagation. As LMs grow in size, backpropagation requires a large amount of memory. 

Zeroth-order (ZO) methods can in-principle estimate gradients using only two forward passes but are theorized to be quite slow for optimizing large models. This paper proposes a memory-efficient zeroth-order optimizer (MeZO) as a more memory-efficient version of a classic zeroth-order optimizer.