HighlightCurated by Aramai Editorial

RoFormer: Enhanced Transformer with Rotary Position Embedding

Introduces Rotary Position Embedding (RoPE), encoding absolute position via a rotation matrix and adding relative-position dependency in self-attention.

The paper investigates how to integrate positional information into transformer-based language models and proposes Rotary Position Embedding (RoPE). RoPE encodes absolute position with a rotation matrix while incorporating explicit relative-position dependency into self-attention. It offers flexibility in sequence length, decaying inter-token dependency with distance, and compatibility with linear self-attention. Evaluated as RoFormer on long-text classification benchmarks, it consistently outperforms alternatives and is supported by theoretical analysis.

Based on: RoFormer: Enhanced Transformer with Rotary Position Embedding · Neurocomputing

HighlightCurated by Aramai Editorial

Book Reviews: Foundations of Statistical Natural Language Processing

The first comprehensive introduction to statistical natural language processing, covering the theory and algorithms needed to build NLP tools.

Statistical approaches to natural language text had become dominant, and this foundational text offers the first comprehensive introduction to statistical natural language processing (NLP). It provides the theory and algorithms needed to build NLP tools, with broad but rigorous coverage of mathematical and linguistic foundations and detailed discussion of statistical methods, letting students and researchers build their own implementations. Covered topics include collocation finding, word sense disambiguation, probabilistic parsing, and information retrieval.

Based on: Book Reviews: Foundations of Statistical Natural Language Processing · International Conference on Computational Logic

HighlightCurated by Aramai Editorial

OrthoMCL: identification of ortholog groups for eukaryotic genomes.

OrthoMCL is a scalable Markov Cluster algorithm-based method for identifying ortholog and paralog groups across multiple eukaryotic genomes.

Identifying orthologous gene groups aids genome annotation, evolutionary studies, and comparative genomics, but prokaryote-oriented methods struggle with eukaryotes, which have many paralogs and incomplete sequences. OrthoMCL scalably constructs orthologous groups across multiple eukaryotic taxa using a Markov Cluster algorithm to group orthologs and paralogs. It matches INPARANOID on two genomes but extends to many species, and agreement with EC annotations suggests high reliability for automated annotation. It was applied to seven genomes and offers a web interface.

Based on: OrthoMCL: identification of ortholog groups for eukaryotic genomes. · Genome Research

HighlightCurated by Aramai Editorial

REVIGO Summarizes and Visualizes Long Lists of Gene Ontology Terms

REVIGO is a web server that summarizes long, redundant Gene Ontology term lists into a representative subset using semantic-similarity clustering.

High-throughput biology is often interpreted by testing for enriched Gene Ontology (GO) categories, but the resulting GO term lists are frequently long and redundant, making interpretation difficult. REVIGO is a web server that summarizes such lists by finding a representative subset of terms using a simple clustering algorithm based on semantic similarity measures. It visualizes this non-redundant set several ways, including multidimensional scaling and graph-based layouts that render semantic relationships, plus treemaps and tag clouds. REVIGO is freely available online.

Based on: REVIGO Summarizes and Visualizes Long Lists of Gene Ontology Terms · PLoS ONE

HighlightCurated by Aramai Editorial

Instant neural graphics primitives with a multiresolution hash encoding

Introduces a multiresolution hash encoding that lets small networks train and render neural graphics primitives orders of magnitude faster.

Neural graphics primitives built from fully connected networks are costly to train and evaluate. This paper reduces that cost with an input encoding that lets a smaller network keep quality while cutting compute and memory operations. A small network is augmented by a multiresolution hash table of trainable feature vectors optimized by SGD, and the multiresolution structure disambiguates hash collisions. Implemented as fully-fused CUDA kernels, it achieves several orders of magnitude speedup: training in seconds and rendering 1920x1080 in tens of milliseconds.

Based on: Instant neural graphics primitives with a multiresolution hash encoding · ACM Transactions on Graphics

HighlightCurated by Aramai Editorial

VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text

Presents VADER, a parsimonious rule-based sentiment analysis model tuned for social media text using a validated lexicon and grammatical intensity rules.

VADER is a simple, rule-based model for general sentiment analysis designed for social media text. The authors build and empirically validate a gold-standard lexicon with sentiment-intensity measures attuned to microblog-like contexts, then combine it with five rules capturing grammatical and syntactic conventions for expressing sentiment. Compared against eleven benchmarks including LIWC, ANEW, SentiWordNet, and ML methods (Naive Bayes, Maximum Entropy, SVM), VADER outperforms individual human raters on tweets (F1 of 0.96 vs 0.84) and generalizes better across contexts.

Based on: VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text · International Conference on Web and Social Media

HighlightCurated by Aramai Editorial

Learning Transferable Features with Deep Adaptation Networks

Proposes Deep Adaptation Networks that match domain distributions in a reproducing kernel Hilbert space for transferable features in domain adaptation.

Deep networks learn transferable features, but transferability drops in higher layers as features grow specific and domain discrepancy rises. The authors propose the Deep Adaptation Network (DAN), generalizing deep CNNs to domain adaptation to reduce dataset bias in task-specific layers. DAN embeds task-specific layers' representations in a reproducing kernel Hilbert space and matches domains' mean embeddings, via optimal multikernel selection. It yields transferable features with statistical guarantees, scales linearly, and sets state of the art on standard benchmarks.

Based on: Learning Transferable Features with Deep Adaptation Networks · International Conference on Machine Learning

HighlightCurated by Aramai Editorial

Context Encoders: Feature Learning by Inpainting

Introduces Context Encoders, a CNN that learns visual features unsupervised by inpainting missing image regions from their surrounding context.

Context Encoders is an unsupervised feature-learning method based on context-driven pixel prediction. A CNN is trained to generate an arbitrary missing image region conditioned on its surroundings, forcing it to understand the whole image and hypothesize plausible content. A combined reconstruction and adversarial loss yields sharper results than reconstruction alone by handling multiple output modes. The learned features capture appearance and semantics, improve CNN pre-training for classification, detection, and segmentation, plus enable semantic inpainting.

Based on: Context Encoders: Feature Learning by Inpainting · Computer Vision and Pattern Recognition

HighlightCurated by Aramai Editorial

On the difficulty of training recurrent neural networks

Analyzes the vanishing and exploding gradient problems in RNNs and proposes gradient norm clipping and a soft constraint as remedies.

Training recurrent neural networks is hampered by two well-known problems, the vanishing and exploding gradients described by Bengio et al. (1994). This paper deepens understanding of these issues by analyzing them from analytical, geometric, and dynamical-systems perspectives. That analysis motivates a simple yet effective solution: a gradient norm clipping strategy for exploding gradients and a soft constraint for vanishing gradients. The authors validate their hypotheses and proposed solutions empirically.

Based on: On the difficulty of training recurrent neural networks · International Conference on Machine Learning

HighlightCurated by Aramai Editorial

Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Presents an incremental Bayesian method that learns generative object-category models from only a few training images, tested across 101 object categories.

Existing methods for learning object categories need thousands of images, are slow, cannot learn incrementally, and ignore prior knowledge. The authors learn categories from just a few images by reusing priors from previously learned, unrelated categories. A generative probabilistic model captures the shape and appearance of a feature constellation, learned incrementally in a Bayesian way. On 101 diverse categories, the incremental method matches a batch Bayesian version on small training sets while being much faster, and both Bayesian methods beat maximum likelihood.

Based on: Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories · 2004 Conference on Computer Vision and Pattern Recognition Workshop

HighlightCurated by Aramai Editorial

Recent advances in convolutional neural networks

Surveys advances in convolutional neural networks: layer design, activations, loss functions, regularization, optimization, and applications.

Deep learning has driven strong performance across visual recognition, speech recognition, and natural language processing, with convolutional neural networks the most extensively studied model. Fueled by growing annotated datasets and stronger GPUs, CNN research has advanced rapidly. This survey broadly reviews recent CNN improvements across layer design, activation functions, loss functions, regularization, optimization, and fast computation. It also introduces applications of CNNs in computer vision, speech, and natural language processing.

Based on: Recent advances in convolutional neural networks · Pattern Recognition

HighlightCurated by Aramai Editorial

Measuring Mathematical Problem Solving With the MATH Dataset

Introduces MATH, a dataset of 12,500 competition math problems with step-by-step solutions for measuring and teaching mathematical reasoning in ML models.

Mathematical problem solving remains difficult for computers. The authors introduce MATH, a dataset of 12,500 challenging competition math problems, each with a full step-by-step solution usable to teach models to generate derivations and explanations. They also release a large auxiliary pretraining dataset covering math fundamentals. Despite some gains, accuracy stays low even with enormous Transformers, and the authors argue that simply scaling model size and compute is impractical, so new algorithmic advances are likely needed.

Based on: Measuring Mathematical Problem Solving With the MATH Dataset · NeurIPS Datasets and Benchmarks