First Hallucination-Free LLM; Fine-Tune or Retrieval; Privacy Issues in LLMs; New Embedding Model by Google; What Resilience Means and More.

Editor's Paper Recommendations

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: The ability of large language models (LLMs) to respond to various questions across various domains demonstrates that they contain significant factual information within their pre-trained weights. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. This study compares two common approaches: fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while fine-tuning offers some improvement, RAG consistently outperforms it for existing knowledge encountered during training and entirely new knowledge. Moreover, LLMs struggle to learn new factual information through fine-tuning. Exposing them to numerous variations of the same fact during training could alleviate this problem.

Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer: Large language models (LLMs) are empowered with broad knowledge, but task-specific performance is often suboptimal. It necessitates fine-tuning LLMs with task-specific data, which may be inaccessible due to privacy concerns. This paper proposes a novel approach to enhance LLMs with smaller language models (SLMs) trained on clients using their private task-specific data. We suggest CrossLM as a way for LLMs and SLMs to improve each other. In this model, the SLMs encourage the LLM to produce task-specific, high-quality data, improving both the LLM and the SLM. We evaluate CrossLM using publicly accessible language models across benchmark tasks. These results show that CrossLM improves the task-specific performance of both the SLMs on clients and the LLM on the cloud server while keeping the LLM's ability to generalize.

Privacy Issues in Large Language Models: A Survey: This is the first survey of the active area of AI research focusing on privacy issues in Large Language Models (LLMs). Specifically, we focus on work that red-teams models to highlight privacy risks, attempts to build privacy into the training or inference process, enables efficient data deletion from trained models to comply with existing privacy regulations, and mitigates copyright issues. We summarize technical research that develops algorithms, proves theorems, and runs empirical evaluations. While an extensive body of legal and policy work addresses these challenges from a different angle, that is not the focus of our survey. Nevertheless, these works and recent legal developments inform how these technical problems are formalized, so we discuss them briefly in Section 1. While we have made our best effort to include all the relevant work, we may have missed some recent work due to the fast-moving nature of this research. If we still need to include some of your work, please get in touch with us, and we will attempt to keep this survey relatively up-to-date. We maintain a repository with the list of papers covered in this survey and any relevant code publicly available at this https URL.

--

Are you looking to advertise a product, job opening, or event to an audience of over 40,000 AI researchers and engineers? Feel free to contact us on LinkedIn to explore your options.

Enjoy the newsletter? Help us make it bigger and better by sharing it with colleagues and friends.

--

Industry Insights

• Compress GPT-4, and Claude prompts with LLMLingua-2
• Introducing the First Hallucination-Free LLM
• Google Unveils Gecko, a Versatile Text Embedding Model Distilled from Large Language Models
• New ReaLM AI by Apple on-device model

Growth Zone

What Resilience Means and Why It Matters

Expert Advice