HighlightCurated by Aramai EditorialarXiv

LSTM: A Search Space Odyssey

Presents the first large-scale analysis of eight LSTM variants on three tasks, finding the forget gate and output activation most critical.

Since the LSTM's 1995 inception, many variants have become state-of-the-art, raising interest in which components matter. This paper reports the first large-scale comparison of eight LSTM variants on speech recognition, handwriting recognition, and polyphonic music modeling. Hyperparameters were tuned per task by random search and ranked by functional ANOVA, over 5400 runs (~15 years of CPU time). No variant significantly beats the standard LSTM; the forget gate and output activation are its most critical components, and its hyperparameters are largely independent.

Based on: LSTM: A Search Space Odyssey · IEEE Transactions on Neural Networks and Learning Systems

HighlightCurated by Aramai EditorialarXiv

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

A tutorial that formally derives RNN and LSTM equations from differential equations, justifies unrolling, and proposes a generalized Vanilla LSTM.

LSTM networks are widely covered, but most articles state inference formulas axiomatically, omit training formulas, and present RNN 'unrolling' without justification. This tutorial explains essential RNN and LSTM fundamentals in one document. Drawing on signal processing, it formally derives the canonical RNN from differential equations and proves a statement yielding the unrolling technique. It then transforms the RNN into a Vanilla LSTM through logical arguments, provides all governing equations, and introduces extensions producing the most general LSTM variant to date.

Based on: Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network · Physica A: Statistical Mechanics and its Applications

HighlightCurated by Aramai EditorialarXiv

On the Opportunities and Risks of Foundation Models

A comprehensive report characterizing foundation models—their capabilities, technical principles, applications, and societal impact.

This report characterizes 'foundation models'—models like BERT, DALL-E, and GPT-3 trained on broad data at scale and adaptable to many downstream tasks. It surveys their opportunities and risks: capabilities (language, vision, robotics, reasoning), technical principles (architectures, training, data), applications (law, healthcare, education), and societal impact (inequity, misuse, environmental effects). Built on deep and transfer learning, their scale yields emergent capabilities and drives homogenization, whose inherited defects demand caution.

Based on: On the Opportunities and Risks of Foundation Models · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Parameter-Efficient Transfer Learning for NLP

Introduces adapter modules that add few trainable parameters per task, enabling parameter-efficient transfer learning for NLP.

Fine-tuning large pre-trained models is effective for NLP transfer but parameter-inefficient, requiring a full new model per task. The authors propose transfer via adapter modules, which add only a few trainable parameters per task while keeping the original network fixed, yielding compact, extensible models with high parameter sharing. Transferring BERT to 26 text classification tasks, including GLUE, adapters reach within 0.4% of full fine-tuning while adding only 3.6% of parameters per task, versus 100% for fine-tuning.

Based on: Parameter-Efficient Transfer Learning for NLP · International Conference on Machine Learning

HighlightCurated by Aramai EditorialConference on Fairness, Accountability and Transparency

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

An examination of ever-larger language models, weighing their risks and recommending cost-aware, well-documented, stakeholder-driven alternatives.

The paper steps back from three years of ever-larger English language models such as BERT, GPT-2/3, and Switch-C, which pushed benchmark state of the art through architecture and sheer size. It asks how big is too big and what risks the technology poses, plus paths to mitigate them. The authors recommend weighing environmental and financial costs first, curating and documenting datasets rather than ingesting everything on the web, running pre-development checks of fit with research goals and stakeholder values, and pursuing directions beyond ever-larger models.

Based on: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 · Conference on Fairness, Accountability and Transparency

HighlightCurated by Aramai EditorialarXiv

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Introduces Mamba, an attention-free selective state space model for linear-time sequence modeling across language, audio, and genomics.

Most foundation models rely on the Transformer's attention module, while subquadratic alternatives like structured state space models (SSMs) have lagged on language. The authors trace this to weak content-based reasoning and let SSM parameters depend on the input, so the model selectively propagates or forgets information per token. A hardware-aware parallel algorithm and a simplified attention- and MLP-free design yield Mamba, which offers fast inference, linear scaling, and state-of-the-art results across modalities; Mamba-3B matches Transformers twice its size.

Based on: Mamba: Linear-Time Sequence Modeling with Selective State Spaces · arXiv.org

HighlightCurated by Aramai EditorialarXiv

Modeling Relational Data with Graph Convolutional Networks

Introduces R-GCNs, graph convolutional networks for multi-relational knowledge bases, applied to link prediction and entity classification.

Knowledge graphs remain incomplete even at their largest (Yago, DBpedia, Wikidata). The authors introduce Relational Graph Convolutional Networks (R-GCNs) for two knowledge base completion tasks: link prediction, recovering missing subject-predicate-object triples, and entity classification, recovering missing attributes. R-GCNs extend graph neural networks to highly multi-relational data. Effective stand-alone for entity classification, an R-GCN encoder also improves factorization models like DistMult, giving a 29.8% gain on FB15k-237 over a decoder-only baseline.

Based on: Modeling Relational Data with Graph Convolutional Networks · Extended Semantic Web Conference

HighlightCurated by Aramai EditorialarXiv

The Power of Scale for Parameter-Efficient Prompt Tuning

Introduces prompt tuning, learning soft prompts via backpropagation to adapt frozen language models, matching full model tuning as scale grows.

The paper explores prompt tuning, a mechanism for learning soft prompts that condition frozen language models for downstream tasks. Unlike GPT-3's discrete text prompts, soft prompts are learned via backpropagation from labeled examples and outperform GPT-3's few-shot learning by a large margin. Ablations with T5 show the method grows more competitive with scale, matching full model tuning once models exceed billions of parameters. Soft prompts also improve robustness to domain transfer and enable efficient prompt ensembling.

Based on: The Power of Scale for Parameter-Efficient Prompt Tuning · Conference on Empirical Methods in Natural Language Processing

HighlightCurated by Aramai EditorialarXiv

RoFormer: Enhanced Transformer with Rotary Position Embedding

Introduces Rotary Position Embedding (RoPE), encoding absolute position via a rotation matrix and adding relative-position dependency in self-attention.

The paper investigates how to integrate positional information into transformer-based language models and proposes Rotary Position Embedding (RoPE). RoPE encodes absolute position with a rotation matrix while incorporating explicit relative-position dependency into self-attention. It offers flexibility in sequence length, decaying inter-token dependency with distance, and compatibility with linear self-attention. Evaluated as RoFormer on long-text classification benchmarks, it consistently outperforms alternatives and is supported by theoretical analysis.

Based on: RoFormer: Enhanced Transformer with Rotary Position Embedding · Neurocomputing

HighlightCurated by Aramai EditorialarXiv

Measuring Mathematical Problem Solving With the MATH Dataset

Introduces MATH, a dataset of 12,500 competition math problems with step-by-step solutions for measuring and teaching mathematical reasoning in ML models.

Mathematical problem solving remains difficult for computers. The authors introduce MATH, a dataset of 12,500 challenging competition math problems, each with a full step-by-step solution usable to teach models to generate derivations and explanations. They also release a large auxiliary pretraining dataset covering math fundamentals. Despite some gains, accuracy stays low even with enormous Transformers, and the authors argue that simply scaling model size and compute is impractical, so new algorithmic advances are likely needed.

Based on: Measuring Mathematical Problem Solving With the MATH Dataset · NeurIPS Datasets and Benchmarks

HighlightCurated by Aramai EditorialarXiv

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Proposes prefix-tuning, a lightweight alternative to fine-tuning that freezes the language model and optimizes continuous task-specific prefix vectors.

Fine-tuning adapts large pretrained language models but modifies all parameters, requiring a full model copy per task. Prefix-tuning instead keeps the model frozen and optimizes a small sequence of continuous, task-specific vectors, the prefix, that later tokens attend to as virtual tokens. Applied to GPT-2 for table-to-text and BART for summarization, it learns only 0.1% of the parameters yet matches full fine-tuning with full data, beats it in low-data settings, and extrapolates better to unseen topics.

Based on: Prefix-Tuning: Optimizing Continuous Prompts for Generation · Annual Meeting of the Association for Computational Linguistics

HighlightCurated by Aramai EditorialarXiv

Longformer: The Long-Document Transformer

Introduces Longformer, a transformer whose attention scales linearly with sequence length to process documents of thousands of tokens.

Standard transformers cannot process long sequences because self-attention scales quadratically with length. Longformer introduces attention that scales linearly, combining local windowed attention with task-motivated global attention as a drop-in replacement for self-attention. It reaches state-of-the-art results on character-level language modeling and, when pretrained and finetuned, consistently outperforms RoBERTa on long-document tasks, with new records on WikiHop and TriviaQA. A Longformer-Encoder-Decoder variant supports generative tasks like arXiv summarization.

Based on: Longformer: The Long-Document Transformer · arXiv.org