How is RAW Bayer data converted to RGB565?

A digital camera sensor does not directly produce a full-color image. Instead, it produces what is called RAW Bayer data. This data contains only partial color information because each pixel records only one color component. To display or process the image properly, it must be converted into a standard format such as RGB565, where each pixel contains complete red, green, and blue values in a compressed 16-bit format.

RGB565 means that every pixel contains 5 bits for Red, 6 bits for Green, and 5 bits for Blue, making a total of 16 bits per pixel. Compared to RGB888 (24 bits per pixel), RGB565 reduces memory usage and bandwidth. Because of this, RGB565 is widely used in embedded systems, LCDs, microcontrollers, and low-power vision devices.

The process of converting RAW Bayer data into RGB565 involves multiple image processing stages. These stages are usually performed inside an Image Signal Processor (ISP), GPU pipeline, or dedicated image processing hardware.

Understanding RAW Bayer Data

A camera sensor is covered by a color filter array called the Bayer filter. The most common arrangement is called the RGGB pattern. In this pattern, each group of four pixels contains two green pixels, one red pixel, and one blue pixel.

The reason for having more green pixels is that the human eye is more sensitive to green light. This improves image detail and brightness accuracy.

However, each pixel in the sensor captures only one color value. For example, a red pixel only measures red intensity, not green or blue. This means that RAW Bayer data does not contain full color information for any single pixel.

Because of this limitation, we need a process called demosaicing to reconstruct the missing color values.

What is RGB565?

RGB565 is a standard 16-bit color format used mainly in embedded display systems. In this format, each pixel contains three values:

• Red (5 bits)

• Green (6 bits)

• Blue (5 bits)

Red and Blue values range from 0 to 31

Green values range from 0 to 63.

Green is given 6 bits instead of 5 because the human eye is more sensitive to green color variations.

Unlike RAW Bayer data, RGB565 contains complete color information for every pixel, but with reduced precision compared to RGB888. This makes it suitable for memory-limited systems and real-time display applications.

Why Conversion is Necessary

RAW Bayer data cannot be displayed directly on a monitor because it does not contain full RGB information. It only contains one color component per pixel. Therefore, conversion is necessary to reconstruct the missing color components and generate a complete image.

After reconstructing full RGB values, the data must be compressed into 16-bit RGB565 format. Without this conversion, the image would appear distorted and contain incorrect colors. The conversion process ensures that every pixel has proper red, green, and blue values in a compact format suitable for embedded displays.

Step-by-Step Conversion Process

Black Level Correction

The first step in the pipeline is black level correction. Camera sensors introduce a small offset value even when no light is present. This offset must be removed to ensure accurate brightness levels. The system subtracts a predefined black level value from every pixel. This step improves contrast and ensures that dark areas appear truly black.

Defective Pixel Correction

Some pixels in a sensor may be damaged or behave abnormally. These are called defective pixels. They may always appear bright or dark regardless of the actual scene. In this stage, the system detects such abnormal pixels and replaces their values using neighbouring pixel information. This improves image quality.

Demosaicing

When a camera captures an image, the sensor does not directly capture full RGB color information at every pixel. Instead, it uses the Bayer filter pattern. Because of this arrangement, each pixel records only one color value.

The most common Bayer pattern is RGGB. In a 2×2 block, it is arranged as:

Row 1: Red, Green

Row 2: Green, Blue

There are two green pixels in each block because the human eye is more sensitive to green light. Increasing green sampling improves brightness detail, and overall image quality.

To understand the conversion process, consider a small 3×3 RAW Bayer data example (RGGB pattern):

Each letter represents a pixel that stores only one intensity value of that specific color.

For example:

A Red pixel may store R = 120

A Green pixel may store G = 90

A Blue pixel may store B = 60

However, to display a proper color image, every pixel must contain three values: (R, G, B). Therefore, the missing two color components at each pixel must be estimated. This estimation process is called demosaicing.

Consider a Red pixel with value R = 120. This pixel does not contain Green or Blue values, so they must be calculated from neighboring pixels.

First, to estimate the Green value at this Red pixel, we examine the surrounding Green pixels (above, below, left, and right).

Example nearby Green values: 100, 95, 105, 98

Using bilinear interpolation, we calculate the average:

Green = (100 + 95 + 105 + 98) / 4

Green ≈ 100

Now the pixel becomes:(R, G, B) = (120, 100, ?)

Next, to estimate the Blue value, we check the diagonal neighboring Blue pixels.

Example nearby Blue values: 60, 65, 58, 62

Blue = (60 + 65 + 58 + 62) / 4

Blue ≈ 61

Now the full RGB value for this pixel becomes:(R, G, B) = (120, 100, 61)

This same estimation process is repeated for every pixel in the image until a complete RGB image is formed.

White Balance

Different lighting conditions affect colors differently. Indoor lighting may appear yellow, while outdoor daylight may appear bluish. White balance adjusts the red, green, and blue channels so that white objects appear truly white.

This is done by applying gain factors to each color channel. This ensures natural color reproduction.

Color Correction Matrix

Even after white balance, sensor colors may not perfectly match real-world colors. A color correction matrix (CCM) is applied to adjust the relationship between red, green, and blue channels. This mathematical transformation improves overall color accuracy.

Gamma Correction

The human eye does not respond linearly to brightness. To match human perception, gamma correction is applied. This step adjusts brightness levels using a nonlinear transformation. It enhances details in shadows and mid-tones while maintaining highlights.

RGB565 Formatting

After all corrections are complete, the pixel values are typically available as 8-bit R, G, and B values. These values must now be converted into RGB565 format.

For example, if:

R = 120

G = 100

B = 61

Conversion to RGB565:

Red (5 bits) = 120 ÷ 8 ≈ 15

Green (6 bits) = 100 ÷ 4 ≈ 25

Blue (5 bits) = 61 ÷ 8 ≈ 7

These values are then packed into a single 16-bit number:

RRRRR GGGGGG BBBBB

At this stage, the image is ready for display in a 16-bit framebuffer.

Complete System Architecture Explanation

In a typical embedded vision system, such as those using CMOS cameras with Jetson platforms, the processing flow begins at the image sensor. The CMOS sensor captures light and outputs RAW Bayer data.

This RAW data travels through an interface such as MIPI CSI to the Image Signal Processor. Inside the ISP, the steps of black level correction, defect correction, demosaicing, white balance, color correction, and gamma correction are performed sequentially.

After processing, the ISP outputs RGB565 directly (if configured) or RGB888, which is later converted to RGB565 in the display pipeline. The formatted data is stored in memory and then sent to a display or embedded LCD module.

This architecture ensures real-time processing for applications like industrial displays, handheld devices, automotive dashboards, and embedded GUI systems.

Memory and Bandwidth Considerations

RAW Bayer images typically use 10-bit or 12-bit depth per pixel. RGB565 uses 16 bits per pixel, while RGB888 uses 24 bits per pixel.

For example, in a 1920 × 1080 image:

RGB888 requires significantly more memory than RGB565.RGB565 reduces memory usage by about 33% compared to RGB888.

This reduction decreases:

• Framebuffer size

• Memory bandwidth

• Power consumption

This is very important in embedded systems where memory and processing speed are limited.

Practical Implementation

In software environments such as OpenCV, conversion from RAW Bayer to RGB can be done using built-in demosaicing functions. The final image can then be converted to RGB565 using bit-shifting operations.

In embedded hardware like Jetson systems, the ISP performs most of the conversion automatically. The output format (RGB565) can be selected through driver configuration or display settings.

In high-performance applications, hardware acceleration ensures real-time conversion without dropping frames.

Conclusion

The conversion from RAW Bayer to RGB565 is a multi-stage image processing pipeline that reconstructs full color information from partial sensor data and then compresses it into a 16-bit format.

The key stage is demosaicing, which estimates missing color components. Additional corrections, such as white balance, color correction, and gamma correction, improve image accuracy and visual quality. Finally, the image is formatted into RGB565 to reduce memory usage while maintaining acceptable color performance.

This entire process transforms sensor data into a compact, display-ready image suitable for embedded systems and real-time applications.