Home / Research Areas / AI Accelerators
Research Area

AI Accelerators

We study compact and energy-efficient neural network architectures for edge AI, with a particular focus on low-bit quantization and hardware-oriented implementation of MLP-Mixer models.

Overview

Deep neural networks (DNNs) have become the de facto standard in computer vision. Convolutional neural networks (CNNs), including VGGNet and ResNet, use convolutional layers for image recognition, while Vision Transformers (ViTs) apply self-attention mechanisms to visual tasks.

More recently, MLP-Mixer has shown that an architecture mainly composed of multilayer perceptrons can achieve competitive image-classification performance. Because its basic structure consists of fully connected layers and nonlinear activation functions, MLP-Mixer can require less complex computation than attention-based models and is therefore attractive for edge AI.

However, further reductions in computational cost, memory use, latency, and energy consumption are necessary before MLP-Mixer can be efficiently deployed on resource-constrained edge devices.

Research Goal

To develop a hardware-oriented, binarized MLP-Mixer that maintains useful classification accuracy while reducing parameter storage and arithmetic complexity for low-power edge AI accelerators.

Proposed Method Architecture Evaluation Results Keywords

Proposed Method
Binarized MLP-Mixer with Integer-Quantized Activations

We propose a quantized MLP-Mixer in which the trainable weight parameters are represented using one-bit values. Binarizing the parameters substantially reduces the memory required to store the model and replaces costly floating-point operations with simpler hardware operations.

The proposed method starts from an MLP-Mixer defined using conventional 32-bit floating-point values. Layers containing trainable parameters are then binarized to obtain a one-bit-weight MLP-Mixer.

Weight Binarization

The weights of the trainable layers are represented using two discrete values. During inference, this representation allows multiplication-intensive operations to be implemented using simpler arithmetic or logic operations, making the model suitable for specialized AI hardware.

Integer Quantization of Activations

The outputs of the binarized layers are quantized to integer values. Some components, such as nonlinear activation functions and layer-normalization operations, are difficult to quantize directly. To handle these components, identity activation layers are inserted immediately after them, and the activation values are quantized at those points.

Since this work targets hardware implementation, symmetric quantization is employed. Symmetric quantization has a relatively simple numerical representation and requires less computation than more complex asymmetric schemes, making it suitable for edge AI accelerators.

Training with the Straight-Through Estimator

Quantization operations are not directly differentiable. During training, the straight-through estimator (STE) is therefore used to estimate gradients in real-valued form. This enables gradient-based optimization while the forward computation uses binarized weights and quantized activations.

Architecture of the Proposed Binarized MLP-Mixer

The proposed architecture applies one-bit quantization to trainable parameter layers in both token-mixing and channel-mixing blocks. Integer-quantization layers are inserted after structures whose outputs cannot be efficiently quantized internally.

This design separates binary weight computation from activation quantization and makes the required numerical precision explicit at each stage of the network. It therefore provides a practical basis for future FPGA, ASIC, or other edge-oriented hardware implementations.

Architecture of the proposed binarized MLP-Mixer with one-bit weights and integer-quantized activations
Figure 1. Architecture of the proposed binarized MLP-Mixer.

Experimental Evaluation

The effectiveness of the proposed method was first evaluated using a software-defined binarized MLP-Mixer. The experiments examined whether the model could be trained successfully and how the activation bit width affected classification accuracy.

Dataset and Data Split

MNIST was used as the image-classification dataset. Of the 60,000 training samples, 80% were used to optimize the model and the remaining 20% were used for validation after each epoch. The model checkpoint with the lowest validation loss was then evaluated on the test dataset, and Top-1 accuracy was calculated.

Quantization Settings

The model weights were binarized, while the output bit width of each layer was set to 1, 2, 4, or 8 bits. Quantization-aware training (QAT) was performed for each setting so that the model could adapt to the quantization effects during optimization.

Experimental Results

Table 1. Classification accuracy for each activation bit width
Activation Bit WidthTop-1 Accuracy
1 bit0.5640
2 bits0.7487
4 bits0.9080
8 bits0.9253
Experimental setup
Figure 2. Training and validation loss curves for each activation bit width.
Experimental setup
Figure 3. Accuracy curves for each activation bit width.

Key Findings

Training with One-Bit Activations

In contrast to earlier observations, the MLP-Mixer could still be trained when both the weights and layer outputs were represented using one bit, although the resulting accuracy was limited.

Accuracy above 0.90

When the activation bit width was four bits or higher, the binarized MLP-Mixer achieved a test accuracy greater than 0.90.

Hardware–Accuracy Trade-off

The results demonstrate a clear trade-off between activation precision and classification performance, providing a useful basis for selecting bit widths in future hardware designs.

Significance and Future Work

These results indicate that MLP-Mixer can be trained using one-bit weights and low-bit activations. In particular, four-bit activation outputs provide a promising balance between classification accuracy and implementation cost.

Future work includes implementing the proposed architecture on FPGA or ASIC platforms, measuring actual latency and power consumption, reducing the cost of layer normalization and activation functions, and evaluating the method on more complex image datasets.

Keywords

AI Accelerator Edge AI MLP-Mixer Binary Neural Network Weight Binarization Activation Quantization Quantization-Aware Training Straight-Through Estimator Symmetric Quantization

Related Publications

← Back to Research Areas