HighlightCurated by Aramai Editorial

Visual Instruction Tuning

Introduces LLaVA, a large multimodal model instruction-tuned on GPT-4-generated language-image data, connecting a vision encoder with an LLM.

Instruction tuning LLMs on machine-generated data improves zero-shot capabilities but is less explored in the multimodal field. This paper makes the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data, and instruction-tunes LLaVA, an end-to-end large multimodal model connecting a vision encoder and an LLM. LLaVA shows strong multimodal chat ability, scoring 85.1% relative to GPT-4 on a synthetic multimodal benchmark, and with GPT-4 achieves state-of-the-art 92.53% on Science QA; data, model, and code are public.

Based on: Visual Instruction Tuning · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Introduces deep compression, a pruning, trained quantization, and Huffman coding pipeline cutting neural network storage 35x-49x with no accuracy loss.

Neural networks are compute- and memory-intensive, making deployment on resource-limited embedded systems difficult. Deep compression is a three-stage pipeline of pruning, trained quantization, and Huffman coding, with retraining to fine-tune remaining connections and quantized centroids; it cuts storage 35x to 49x without accuracy loss. AlexNet shrinks from 240MB to 6.9MB and VGG-16 from 552MB to 11.3MB, letting models fit in on-chip SRAM rather than off-chip DRAM, with 3x-4x layerwise speedup and 3x-7x better energy efficiency.

Based on: Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding · International Conference on Learning Representations

HighlightCurated by Aramai Editorial

Non-local Neural Networks

Presents non-local operations as generic building blocks that capture long-range dependencies by computing responses as weighted sums over all positions.

Convolutional and recurrent operations process one local neighborhood at a time. Inspired by the classical non-local means method, this paper presents non-local operations, a generic family of building blocks that compute the response at a position as a weighted sum of the features at all positions and can be plugged into many computer vision architectures. Without bells and whistles, non-local models compete with or outperform competition winners on Kinetics and Charades video classification, and improve object detection, segmentation, and pose estimation on COCO.

Based on: Non-local Neural Networks · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

HighlightCurated by Aramai Editorial

Overcoming catastrophic forgetting in neural networks

Proposes training neural networks sequentially by selectively slowing learning on weights important to previous tasks, overcoming catastrophic forgetting

Deep neural networks, unlike humans, struggle to learn multiple tasks sequentially, and catastrophic forgetting was widely thought to be an inevitable feature of connectionist models. Inspired by synaptic consolidation in neuroscience, this work trains networks sequentially by selectively slowing learning on the weights important for previous tasks. The approach proves scalable and effective on hand-written digit classification tasks and sequentially learned Atari 2600 games, maintaining expertise on tasks not experienced for a long time.

Based on: Overcoming catastrophic forgetting in neural networks · Proceedings of the National Academy of Sciences of the United States of America

HighlightCurated by Aramai Editorial

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Develops diffusion-based generative models where a forward process slowly destroys data structure and a learned reverse process restores it.

The work targets a central machine learning problem: modeling complex data with distributions that stay flexible yet tractable for learning, sampling, inference, and evaluation. Inspired by non-equilibrium statistical physics, an iterative forward diffusion process slowly destroys structure in the data, and a learned reverse diffusion restores it, yielding a flexible, tractable generative model. The method scales to thousands of layers or time steps, supports conditional and posterior probabilities, and has an open-source reference implementation.

Based on: Deep Unsupervised Learning using Nonequilibrium Thermodynamics · International Conference on Machine Learning

HighlightCurated by Aramai Editorial

Evaluating Large Language Models Trained on Code

Introduces Codex, a GPT model fine-tuned on GitHub code, and HumanEval, a benchmark for functional correctness of programs synthesized from docstrings.

The paper introduces Codex, a GPT language model fine-tuned on public GitHub code, and studies its Python code-writing abilities; a distinct production version powers GitHub Copilot. On HumanEval, a newly released benchmark measuring functional correctness of programs synthesized from docstrings, Codex solves 28.8% of problems versus 0% for GPT-3 and 11.4% for GPT-J, and repeated sampling solves 70.2% with 100 samples per problem. The authors also examine limitations, such as long chains of operations and variable binding, and discuss safety, security, and economic impacts.

Based on: Evaluating Large Language Models Trained on Code · arXiv.org

HighlightCurated by Aramai Editorial

Improved Techniques for Training GANs

Presents new GAN training techniques that achieve state-of-the-art semi-supervised classification and generate images humans find visually realistic.

The authors present new architectural features and training procedures for the GAN framework, focused on semi-supervised learning and generating images humans find visually realistic, rather than maximizing test-data likelihood. The techniques achieve state-of-the-art semi-supervised classification on MNIST, CIFAR-10, and SVHN. A visual Turing test confirms image quality: generated MNIST samples are indistinguishable from real data, CIFAR-10 samples yield a 21.3% human error rate, and ImageNet samples at unprecedented resolution show recognizable class features.

Based on: Improved Techniques for Training GANs · Neural Information Processing Systems

HighlightCurated by Aramai Editorial

YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors

Introduces YOLOv7, a real-time object detector combining trainable bag-of-freebies training tools with a new architecture and compound scaling method.

Real-time object detection is a central computer vision problem, and the authors identify two research topics that emerged from recent architecture and training optimization methods. They address these with a trainable bag-of-freebies solution, combining flexible, efficient training tools with a proposed architecture and compound scaling method. YOLOv7 surpasses all known object detectors in speed and accuracy across 5-120 FPS, reaching 56.8% AP, the highest among real-time detectors at 30 FPS or higher on a V100 GPU; source code is released.

Based on: YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors · Computer Vision and Pattern Recognition

HighlightCurated by Aramai Editorial

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Introduces Batch Normalization, normalizing layer inputs per mini-batch to speed up and stabilize deep neural network training.

Training deep networks is complicated by internal covariate shift, where each layer's input distribution shifts as prior layers update, forcing low learning rates and careful initialization. Batch Normalization addresses this by normalizing layer inputs per mini-batch as part of the architecture, enabling higher learning rates and less careful initialization, sometimes removing the need for Dropout. Applied to a state-of-the-art classifier, it matches accuracy with 14x fewer steps, and a batch-normalized ensemble reaches 4.82% top-5 ImageNet error, beating human raters.

Based on: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift · International Conference on Machine Learning

HighlightCurated by Aramai Editorial

Going deeper with convolutions

Proposes the Inception architecture (GoogLeNet), a 22-layer deep network improving compute efficiency for ImageNet classification and detection.

This paper proposes the Inception deep convolutional architecture, achieving new state-of-the-art results in ILSVRC14 classification and detection. Its key feature is improved computing resource utilization: depth and width increase while the computational budget stays constant, guided by the Hebbian principle and multi-scale processing intuition. The ILSVRC14 submission, a 22-layer network called GoogLeNet, is evaluated on both classification and detection quality.

Based on: Going deeper with convolutions · Computer Vision and Pattern Recognition

HighlightCurated by Aramai Editorial

Learning Transferable Visual Models From Natural Language Supervision

Learns visual representations by predicting image-caption pairings from 400 million web image-text pairs, enabling zero-shot transfer.

Standard vision systems train on fixed predetermined categories, limiting generality without more labeled data. This paper learns directly from raw image text via a simple pre-training task predicting which caption matches which image, scaled to 400 million web image-text pairs. After pre-training, natural language enables zero-shot transfer, benchmarked across 30+ vision datasets spanning OCR, action recognition, and fine-grained classification, often matching supervised baselines, e.g. matching ResNet-50's zero-shot ImageNet accuracy without its training data.

Based on: Learning Transferable Visual Models From Natural Language Supervision · International Conference on Machine Learning

HighlightCurated by Aramai Editorial

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Details the design principles behind PyTorch, a deep learning library combining Pythonic imperative usability with GPU-accelerated speed.

Deep learning frameworks have typically traded off usability against speed. PyTorch is presented as a library showing these goals are compatible, designed to support an imperative, Pythonic programming style where code acts as the model, debugging is easy, and it stays consistent with other scientific computing libraries, while remaining efficient on hardware accelerators like GPUs. The paper details the principles behind PyTorch's implementation and architecture, and demonstrates the efficiency of its subsystems and overall speed on common benchmarks.

Based on: PyTorch: An Imperative Style, High-Performance Deep Learning Library · Neural Information Processing Systems