HighlightCurated by Aramai Editorial

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Introduces Mamba, an attention-free selective state space model for linear-time sequence modeling across language, audio, and genomics.

Most foundation models rely on the Transformer's attention module, while subquadratic alternatives like structured state space models (SSMs) have lagged on language. The authors trace this to weak content-based reasoning and let SSM parameters depend on the input, so the model selectively propagates or forgets information per token. A hardware-aware parallel algorithm and a simplified attention- and MLP-free design yield Mamba, which offers fast inference, linear scaling, and state-of-the-art results across modalities; Mamba-3B matches Transformers twice its size.

Based on: Mamba: Linear-Time Sequence Modeling with Selective State Spaces · arXiv.org

HighlightCurated by Aramai Editorial

UMAP: Uniform Manifold Approximation and Projection

Introduces UMAP, a fast, mathematically grounded manifold learning technique for visualization and general non-linear dimension reduction.

UMAP is a dimension reduction technique usable for visualization, much like t-SNE, but also for general non-linear dimension reduction. It rests on a rigorous mathematical foundation yet remains simple to use through a scikit-learn compatible API. UMAP ranks among the fastest available manifold learning implementations, running significantly faster than most t-SNE implementations.

Based on: UMAP: Uniform Manifold Approximation and Projection · Journal of Open Source Software

HighlightCurated by Aramai Editorial

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Introduces CutMix, augmentation cutting and pasting patches between images with area-proportional label mixing, aiding classification and localization.

Regional dropout helps CNN classifiers attend to less discriminative object parts, but existing methods delete informative pixels via black or noise patches, causing information loss. CutMix instead cuts and pastes patches between training images while mixing labels in proportion to patch area, using pixels efficiently while keeping regularization benefits. It beats state-of-the-art augmentation on CIFAR and ImageNet classification and weakly-supervised localization; as a pretrained model it aids Pascal detection and MS-COCO captioning, robustness, and OOD detection.

Based on: CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features · IEEE International Conference on Computer Vision

HighlightCurated by Aramai Editorial

Modeling Relational Data with Graph Convolutional Networks

Introduces R-GCNs, graph convolutional networks for multi-relational knowledge bases, applied to link prediction and entity classification.

Knowledge graphs remain incomplete even at their largest (Yago, DBpedia, Wikidata). The authors introduce Relational Graph Convolutional Networks (R-GCNs) for two knowledge base completion tasks: link prediction, recovering missing subject-predicate-object triples, and entity classification, recovering missing attributes. R-GCNs extend graph neural networks to highly multi-relational data. Effective stand-alone for entity classification, an R-GCN encoder also improves factorization models like DistMult, giving a 29.8% gain on FB15k-237 over a decoder-only baseline.

Based on: Modeling Relational Data with Graph Convolutional Networks · Extended Semantic Web Conference

HighlightCurated by Aramai Editorial

Federated Learning: Challenges, Methods, and Future Directions

Surveys federated learning, which trains models across devices or siloed data while data stays local, outlining challenges, methods and future work.

Federated learning trains statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping the data localized. Training in heterogeneous and potentially massive networks introduces new challenges that require departing from standard approaches to large-scale machine learning, distributed optimization, and privacy-preserving data analysis. This article discusses the characteristics and challenges of federated learning, overviews current approaches, and outlines future directions relevant to many research communities.

Based on: Federated Learning: Challenges, Methods, and Future Directions · IEEE Signal Processing Magazine

HighlightCurated by Aramai Editorial

CLUMPP: a cluster matching and permutation program for dealing with label switching and multimodality in analysis of population structure

Presents CLUMPP, a program with three algorithms to align replicate population-structure clustering runs, resolving label switching and multimodality.

Clustering individuals into populations from multilocus genotypes is useful across genetics, but algorithms like STRUCTURE, BAPS and TESS use stochastic unsupervised methods that can yield distinct solutions across replicate runs of the same data. Differences arise from label switching, the arbitrary labeling of clusters, and from genuine multimodality. The authors describe three algorithms for aligning replicate analyses, implemented in CLUMPP (CLUster Matching and Permutation Program), illustrated on 100 replicate runs of 600 chickens from 20 breeds.

Based on: CLUMPP: a cluster matching and permutation program for dealing with label switching and multimodality in analysis of population structure · Bioinform.

HighlightCurated by Aramai Editorial

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

Proposes TransUNet, combining Transformers and U-Net to use transformers as strong encoders for medical image segmentation.

TransUNet combines Transformers and U-Net for medical image segmentation, addressing U-Net's weak long-range dependency modeling caused by convolution locality. The Transformer encodes tokenized image patches from a CNN feature map as an input sequence to extract global context, while the decoder upsamples these encoded features and combines them with high-resolution CNN feature maps to enable precise localization. The authors argue Transformers are strong encoders for segmentation; TransUNet achieves superior performance on multi-organ and cardiac segmentation.

Based on: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation · arXiv.org

HighlightCurated by Aramai Editorial

Dual Attention Network for Scene Segmentation

Proposes DANet, coupling position and channel self-attention modules on a dilated FCN to reach state-of-the-art scene segmentation.

This paper addresses scene segmentation by capturing contextual dependencies with self-attention. Instead of fusing multi-scale features, the Dual Attention Network (DANet) integrates local features with global dependencies via two attention modules on a dilated FCN. A position attention module aggregates each position as a weighted sum over all positions; a channel attention module emphasizes interdependent channel maps. Summing their outputs improves representation and yields new state of the art on Cityscapes, PASCAL Context and COCO Stuff, at 81.5% mean IoU.

Based on: Dual Attention Network for Scene Segmentation · Computer Vision and Pattern Recognition

HighlightCurated by Aramai Editorial

A Comprehensive Survey on Transfer Learning

A comprehensive survey systematizing transfer learning, reviewing 40+ approaches from data and model perspectives with experiments on 20+ models.

Transfer learning improves target-domain learners by transferring knowledge from different but related source domains, reducing the need for large target datasets. This survey connects and systematizes existing research, summarizing the mechanisms and strategies of transfer learning. It reviews more than 40 representative approaches, especially homogeneous ones, from the perspectives of data and model. Over 20 models are experimented on three datasets—Amazon Reviews, Reuters-21578, and Office-31—showing the importance of selecting appropriate models for each application.

Based on: A Comprehensive Survey on Transfer Learning · Proceedings of the IEEE

HighlightCurated by Aramai Editorial

Flamingo: a Visual Language Model for Few-Shot Learning

Introduces Flamingo, visual language models bridging pretrained vision and language models for in-context few-shot learning on image and video tasks.

Flamingo is a family of Visual Language Models built for rapid adaptation to new tasks from only a few annotated examples. Architectural innovations bridge frozen vision-only and language-only models, handle arbitrarily interleaved images and text, and ingest images or videos. Trained on large-scale multimodal web corpora of interleaved text and images, Flamingo gains in-context few-shot learning. A single model sets new state of the art on many image and video benchmarks just by prompting with task examples, often beating models fine-tuned on far more data.

Based on: Flamingo: a Visual Language Model for Few-Shot Learning · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

Optimal approximations by piecewise smooth functions and associated variational problems

Introduces and studies three variational problems for optimally approximating images by piecewise smooth functions, motivated by computer vision.

This paper introduces and studies three new variational problems motivated by applications to computer vision. A fundamental problem is to appropriately decompose the domain of an image function g(x,y)—the light intensity striking a plane domain when a three-dimensional world is viewed from a point. Because light reflected from distinct objects strikes different regions, the image is generally discontinuous along the boundaries, or edges, between them. The variational problems formalize optimal approximation of such images by piecewise smooth functions over these regions.

Based on: Optimal approximations by piecewise smooth functions and associated variational problems · Semantic Scholar

HighlightCurated by Aramai Editorial

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Introduces the Exponential Linear Unit (ELU), an activation with negative values that speeds deep network training and improves accuracy over ReLUs.

The paper introduces the exponential linear unit (ELU), an activation that speeds up deep network learning and raises accuracy. Like ReLUs, ELUs avoid vanishing gradients via the identity for positive inputs, but their negative values push mean activations toward zero, reducing bias shift and bringing gradients closer to the natural gradient. ELUs saturate to a negative value for small inputs, giving noise-robust deactivation. Experiments show faster learning and better generalization than ReLUs and LReLUs, with top CIFAR results and under 10% single-crop ImageNet error.

Based on: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) · International Conference on Learning Representations