NuttX RTOS for PinePhone: Framebuffer

đź“ť 1 Jan 2023

Apache NuttX Framebuffer App on Pine64 PinePhone

Suppose we’re running Apache NuttX RTOS on Pine64 PinePhone…

How will we create Graphical Apps for NuttX? (Pic above)

Today we’ll learn about the…

NuttX Framebuffer App running on PinePhone

NuttX Framebuffer App running on PinePhone

§1 Framebuffer Demo

Our Demo Code for today comes (mostly) from this Example App…

How do we build the app?

To enable the app in our NuttX Project…

make menuconfig

And select…

Application Configuration > Examples > Framebuffer Driver Example

Save the configuration and exit menuconfig.

Look for this line: apps/examples/fb/fb_main.c

#ifdef CONFIG_FB_OVERLAY

And change it to…

#ifdef NOTUSED

Because our PinePhone Framebuffer Driver doesn’t support overlays yet.

Then build NuttX with…

make

Before we run the demo, let’s look at the code…

§2 Framebuffer Interface

What’s inside the app?

We begin with the Framebuffer Interface that NuttX provides to our apps for rendering graphics.

To call the Framebuffer Interface, our app opens the Framebuffer Driver at /dev/fb0: fb_main.c

#include <nuttx/video/fb.h>
#include <nuttx/video/rgbcolors.h>

// Open the Framebuffer Driver
int fd = open("/dev/fb0", O_RDWR);

// Quit if we failed to open
if (fd < 0) { return; }

Next we fetch the Framebuffer Characteristics, which will tell us the Screen Size (720 x 1440) and Pixel Format (ARGB 8888)…

// Get the Characteristics of the Framebuffer
struct fb_videoinfo_s vinfo;
int ret = ioctl(          // Do I/O Control...
  fd,                     // File Descriptor of Framebuffer Driver
  FBIOGET_VIDEOINFO,      // Get Characteristics
  (unsigned long) &vinfo  // Framebuffer Characteristics
);

// Quit if FBIOGET_VIDEOINFO failed
if (ret < 0) { return; }

(fb_videoinfo_s is defined here)

Then we fetch the Plane Info, which describes the RAM Framebuffer that we’ll use for drawing: fb_main.c

// Get the Plane Info
struct fb_planeinfo_s pinfo;
ret = ioctl(              // Do I/O Control...
  fd,                     // File Descriptor of Framebuffer Driver
  FBIOGET_PLANEINFO,      // Get Plane Info
  (unsigned long) &pinfo  // Returned Plane Info
);

// Quit if FBIOGET_PLANEINFO failed
if (ret < 0) { return; }

(fb_planeinfo_s is defined here)

To access the RAM Framebuffer, we map it to a valid address: fb_main.c

// Map the Framebuffer Address
void *fbmem = mmap(  // Map the address of...
  NULL,              // Hint (ignored)
  pinfo.fblen,       // Framebuffer Size
  PROT_READ | PROT_WRITE,  // Read and Write Access
  MAP_SHARED | MAP_FILE,   // Map as Shared Memory
  fd,  // File Descriptor of Framebuffer Driver               
  0    // Offset for Memory Mapping
);

// Quit if we failed to map the Framebuffer Address
if (fbmem == MAP_FAILED) { return; }

This returns fbmem, a pointer to the RAM Framebuffer.

Let’s blast some pixels to the RAM Framebuffer…

Render Grey Screen

§3 Render Grey Screen

What’s the simplest thing we can do with our Framebuffer?

Let’s fill the entire Framebuffer with Grey: fb_main.c

// Fill entire framebuffer with grey
memset(        // Fill the buffer...
  fbmem,       // Framebuffer Address
  0x80,        // Value
  pinfo.fblen  // Framebuffer Size
);

(We’ll explain in a while why this turns grey)

After filling the Framebuffer, we refresh the display: fb_main.c

// Area to be refreshed
struct fb_area_s area = {
  .x = 0,  // X Offset
  .y = 0,  // Y Offset
  .w = pinfo.xres_virtual,  // Width
  .h = pinfo.yres_virtual   // Height
};

// Refresh the display
ioctl(  // Do I/O Control...
  fd,   // File Descriptor of Framebuffer Driver
  FBIO_UPDATE,           // Refresh the Display
  (unsigned long) &area  // Area to be refreshed
);

(fb_area_s is defined here)

If we skip this step, we’ll see missing pixels in our display.

(More about this below)

Remember to close the Framebuffer when we’re done: fb_main.c

// Unmap the Framebuffer Address
munmap(        // Unmap the address of...
  fbmem,       // Framebuffer Address
  pinfo.fblen  // Framebuffer Size
);

// Close the Framebuffer Driver
close(fd);

When we run this, PinePhone turns grey! (Pic above)

To understand why, let’s look inside the Framebuffer…

PinePhone Framebuffer

Why did PinePhone turn grey when we filled it with 0x80?

Our Framebuffer has 720 x 1440 pixels. Each pixel has 32-bit ARGB 8888 format (pic above)…

(Alpha has no effect, since this is the Base Layer and there’s nothing underneath)

When we fill the Framebuffer with 0x80, we’re setting Alpha (unused), Red, Green and Blue to 0x80.

Which produces the grey screen.

Let’s do some colours…

(Alpha Channel looks redundant, but it will be used when we support Overlays)

Render Blocks

§4 Render Blocks

This is how we render the Blue, Green and Red Blocks in the pic above: fb_main.c

// Fill framebuffer with Blue, Green and Red Blocks
uint32_t *fb = fbmem;  // Access framebuffer as 32-bit pixels
const size_t fblen = pinfo.fblen / 4;  // 4 bytes per pixel

// For every pixel...
for (int i = 0; i < fblen; i++) {

  // Colors are in ARGB 8888 format
  if (i < fblen / 4) {
    // Blue for top quarter.
    // RGB24_BLUE is 0x0000 00FF
    fb[i] = RGB24_BLUE;

  } else if (i < fblen / 2) {
    // Green for next quarter.
    // RGB24_GREEN is 0x0000 FF00
    fb[i] = RGB24_GREEN;

  } else {
    // Red for lower half.
    // RGB24_RED is 0x00FF 0000
    fb[i] = RGB24_RED;
  }
}

// Omitted: Refresh the display with ioctl(FBIO_UPDATE)

Everything is hunky dory for chunks of pixels! Let’s set individual pixels by row and column…

Render Circle

§5 Render Circle

This is how we render the Green Circle in the pic above: fb_main.c

// Fill framebuffer with Green Circle
uint32_t *fb = fbmem;  // Access framebuffer as 32-bit pixels
const size_t fblen = pinfo.fblen / 4;  // 4 bytes per pixel

const int width  = pinfo.xres_virtual;  // Framebuffer Width
const int height = pinfo.yres_virtual;  // Framebuffer Height

// For every pixel row...
for (int y = 0; y < height; y++) {

  // For every pixel column...
  for (int x = 0; x < width; x++) {

    // Get pixel index
    const int p = (y * width) + x;

    // Shift coordinates so that centre of screen is (0,0)
    const int half_width  = width  / 2;
    const int half_height = height / 2;
    const int x_shift = x - half_width;
    const int y_shift = y - half_height;

    // If x^2 + y^2 < radius^2, set the pixel to Green.
    // Colors are in ARGB 8888 format.
    if (x_shift*x_shift + y_shift*y_shift <
        half_width*half_width) {
      // RGB24_GREEN is 0x0000 FF00
      fb[p] = RGB24_GREEN;

    } else {  // Otherwise set to Black
      // RGB24_BLACK is 0x0000 0000
      fb[p] = RGB24_BLACK;
    }
  }
}

// Omitted: Refresh the display with ioctl(FBIO_UPDATE)

Yep we have full control over every single pixel! Let’s wrap up our demo with some mesmerising rectangles…

Render Rectangles

§6 Render Rectangle

When we run the NuttX Framebuffer App, we’ll see a stack of Color Rectangles. (Pic above)

We render each Rectangle like so: fb_main.c

// Rectangle to be rendered
struct fb_area_s area = {
  .x = 0,  // X Offset
  .y = 0,  // Y Offset
  .w = pinfo.xres_virtual,  // Width
  .h = pinfo.yres_virtual   // Height
}

// Render the rectangle
draw_rect(&state, &area, color);

// Omitted: Refresh the display with ioctl(FBIO_UPDATE)

(draw_rect is defined here)

The pic below shows the output of the Framebuffer App fb when we run it on PinePhone…

NuttX Framebuffer App running on PinePhone

(See the Complete Log)

And we’re all done with Circles and Rectangles on PinePhone! Let’s talk about Graphical User Interfaces…

LVGL on NuttX on PinePhone

§7 LVGL Graphics Library

Rendering graphics pixel by pixel sounds tedious…

Is there a simpler way to render Graphical User Interfaces?

Yep just call the LVGL Graphics Library! (Pic above)

To build the LVGL Demo App on NuttX…

make menuconfig

Select these options…

Enable “Application Configuration > Graphics Support > Light and Versatile Graphics Library (LVGL)”

Enable “LVGL > Enable Framebuffer Port”

Browse into “LVGL > LVGL Configuration”

Enable “Application Configuration > Examples > LVGL Demo”

Save the configuration and exit menuconfig. Rebuild NuttX…

make

Boot NuttX on PinePhone. At the NSH Command Prompt, enter…

lvgldemo widgets

And we’ll see the LVGL Graphical User Interface on PinePhone! (Like this)

But it won’t respond to our touch right?

Yeah we haven’t started on the I2C Touch Input Driver for PinePhone.

Maybe someday LVGL Touchscreen Apps will run OK on PinePhone!

What’s inside the LVGL App?

Here’s how it works…

Now we talk about the internals of our Framebuffer Driver…

§8 PinePhone Framebuffer Driver

We’ve seen the Framebuffer Interface for NuttX Apps…

What’s inside the Framebuffer Driver for PinePhone?

Let’s talk about the internals of our Framebuffer Driver for PinePhone…

Complete Display Driver for PinePhone

Complete Display Driver for PinePhone

§8.1 RAM Framebuffer

Inside PinePhone’s Allwinner A64 SoC are the Display Engine and Timing Controller TCON0. (Pic above)

Display Engine and TCON0 will blast pixels from the RAM Framebuffer to the LCD Display, over Direct Memory Access (DMA).

(More about Display Engine and TCON0)

Here’s our RAM Framebuffer: pinephone_display.c

// Frame Buffer for Display Engine 
// Fullscreen 720 x 1440 (4 bytes per XRGB 8888 pixel)
// PANEL_WIDTH is 720
// PANEL_HEIGHT is 1440
static uint32_t g_pinephone_fb0[  // 32 bits per pixel
  PANEL_WIDTH * PANEL_HEIGHT      // 720 x 1440 pixels
];

(Memory Protection is not turned on yet, so mmap returns the actual address of g_pinephone_fb0 to NuttX Apps for rendering)

We describe PinePhone’s LCD Display like so (pic below)…

// Video Info for PinePhone
// (Framebuffer Characteristics)
// PANEL_WIDTH is 720
// PANEL_HEIGHT is 1440
static struct fb_videoinfo_s g_pinephone_video = {
  .fmt       = FB_FMT_RGBA32,  // Pixel format (XRGB 8888)
  .xres      = PANEL_WIDTH,    // Horizontal resolution in pixel columns
  .yres      = PANEL_HEIGHT,   // Vertical resolution in pixel rows
  .nplanes   = 1,  // Color planes: Base UI Channel
  .noverlays = 2   // Overlays: 2 Overlay UI Channels
};

(fb_videoinfo_s is defined here)

(We’re still working on the Overlays)

We tell NuttX about our RAM Framebuffer with this Plane Info…

// Color Plane for Base UI Channel:
// Fullscreen 720 x 1440 (4 bytes per XRGB 8888 pixel)
static struct fb_planeinfo_s g_pinephone_plane = {
  .fbmem        = &g_pinephone_fb0,         // Framebuffer Address
  .fblen        = sizeof(g_pinephone_fb0),  // Framebuffer Size
  .stride       = PANEL_WIDTH * 4,  // Length of a line (4-byte pixel)
  .display      = 0,   // Display number (Unused)
  .bpp          = 32,  // Bits per pixel (XRGB 8888)
  .xres_virtual = PANEL_WIDTH,   // Virtual Horizontal resolution
  .yres_virtual = PANEL_HEIGHT,  // Virtual Vertical resolution
  .xoffset      = 0,  // X Offset from virtual to visible
  .yoffset      = 0   // Y Offset from virtual to visible
};

(fb_planeinfo_s is defined here)

PinePhone Framebuffer

§8.2 Framebuffer Operations

Our Framebuffer Driver supports these Framebuffer Operations: pinephone_display.c

// Vtable for Frame Buffer Operations
static struct fb_vtable_s g_pinephone_vtable = {

  // Basic Framebuffer Operations
  .getvideoinfo    = pinephone_getvideoinfo,
  .getplaneinfo    = pinephone_getplaneinfo,
  .updatearea      = pinephone_updatearea,

  // TODO: Framebuffer Overlay Operations
  .getoverlayinfo  = pinephone_getoverlayinfo,
  .settransp       = pinephone_settransp,
  .setchromakey    = pinephone_setchromakey,
  .setcolor        = pinephone_setcolor,
  .setblank        = pinephone_setblank,
  .setarea         = pinephone_setarea
};

We haven’t implemented the Overlays, so let’s talk about the first 3 operations…

But before that we need to initialise the Framebuffer and return the Video Plane…

§8.3 Initialise Framebuffer

At Startup, NuttX Kernel calls up_fbinitialize to initialize the Framebuffer…

up_fbinitialize comes from our Framebuffer Driver (LCD Driver)…

Then NuttX Kernel interrogates our Framebuffer Driver…

§8.4 Get Video Plane

NuttX Kernel calls our Framebuffer Driver to discover the Framebuffer Operations supported by our driver.

This is how we return the Framebuffer Operations: pinephone_display.c

// Get the Framebuffer Object for the supported operations
struct fb_vtable_s *up_fbgetvplane(
  int display,  // Display Number should be 0
  int vplane    // Video Plane should be 0
) {
  // Return the supported Framebuffer Operations
  return &g_pinephone_vtable;
}

(We’ve seen g_pinephone_vtable earlier)

Now it gets interesting: NuttX Kernel and NuttX Apps will call the operations exposed by our Framebuffer Driver…

Test Pattern on NuttX for PinePhone

§8.5 Get Video Info

Remember FBIOGET_VIDEOINFO for fetching the Framebuffer Characteristics?

The first operation exposed by our Framebuffer Driver returns the Video Info that contains our Framebuffer Characteristics: pinephone_display.c

// Get the Video Info for our Framebuffer
// (ioctl Entrypoint: FBIOGET_VIDEOINFO)
static int pinephone_getvideoinfo(
  struct fb_vtable_s *vtable,   // Framebuffer Driver
  struct fb_videoinfo_s *vinfo  // Returned Video Info
) {
  // Copy and return the Video Info
  memcpy(vinfo, &g_pinephone_video, sizeof(struct fb_videoinfo_s));

  // Keep track of the stages during startup:
  // Stage 0: Initialize driver at startup
  // Stage 1: First call by apps
  // Stage 2: Subsequent calls by apps
  // We erase the framebuffers at stages 0 and 1. This allows the
  // Test Pattern to be displayed for as long as possible before erasure.
  static int stage = 0;
  if (stage < 2) {
    stage++;
    memset(g_pinephone_fb0, 0, sizeof(g_pinephone_fb0));
    memset(g_pinephone_fb1, 0, sizeof(g_pinephone_fb1));
    memset(g_pinephone_fb2, 0, sizeof(g_pinephone_fb2));
  }
  return OK;
}

(We’ve seen g_pinephone_video earlier)

This code looks interesting: We’re trying to show the Startup Test Pattern for as long as possible. (Pic above)

Normally NuttX Kernel will erase our Framebuffer at startup. But with the logic above, our Test Pattern will be visible until the first app call to our Framebuffer Driver.

(Test Pattern is rendered by pinephone_display_test_pattern)

(Which is called by pinephone_bringup at startup)

§8.6 Get Plane Info

Earlier we’ve seen FBIOGET_PLANEINFO that fetches the RAM Framebuffer…

This is how we return the Plane Info that describes the RAM Framebuffer: pinephone_display.c

// Get the Plane Info for our Framebuffer
// (ioctl Entrypoint: FBIOGET_PLANEINFO)
static int pinephone_getplaneinfo(
  struct fb_vtable_s *vtable,   // Framebuffer Driver
  int planeno,  // Plane Number should be 0
  struct fb_planeinfo_s *pinfo  // Returned Plane Info
) {
  // Copy and return the Plane Info
  memcpy(pinfo, &g_pinephone_plane, sizeof(struct fb_planeinfo_s));
  return OK;
}

(We’ve seen g_pinephone_plane earlier)

§8.7 Update Area

The final operation updates the display when there’s a change to the Framebuffer: pinephone_display.c

// Update the Display when there is a change to the Framebuffer
// (ioctl Entrypoint: FBIO_UPDATE)
static int pinephone_updatearea(
  struct fb_vtable_s *vtable,   // Framebuffer Driver
  const struct fb_area_s *area  // Updated area of Framebuffer
) {
  // Mystery Code...

This operation is invoked when NuttX Apps call FBIO_UPDATE, as we’ve seen earlier…

The code inside looks totally baffling, but first let’s talk about a mysterious rendering problem…

Missing Pixels in PinePhone Image

§9 Mystery of the Missing Pixels

When we tested our Framebuffer Driver for the very first time, we discovered missing pixels in the rendered image (pic above)…

Maybe we didn’t render the pixels correctly?

Or maybe the RAM Framebuffer got corrupted?

When we slowed down the rendering, we see the missing pixels magically appear later in a curious pattern…

According to the video, the pixels are actually written correctly to the RAM Framebuffer.

But the pixels at the lower half don’t get pushed to the display until the next screen update.

Maybe it’s a problem with Framebuffer DMA / Display Engine / Timing Controller TCON0?

Yeah there seems to be a lag between the writing of pixels to RAM Framebuffer, and the pushing of pixels to the display over DMA / Display Engine / Timing Controller TCON0.

We found an unsatisfactory workaround for the lag in rendering pixels…

Fixed Missing Pixels in PinePhone Image

§10 Fix Missing Pixels

In the previous section we saw that there was a lag pushing pixels from the RAM Framebuffer to the PinePhone Display.

(Over DMA / Display Engine / Timing Controller TCON0)

Can we overcome this lag by copying the RAM Framebuffer to itself, forcing the display to refresh?

This sounds very strange, but yes it works!

From pinephone_display.c:

// Update the Display when there is a change to the Framebuffer
// (ioctl Entrypoint: FBIO_UPDATE)
static int pinephone_updatearea(
  struct fb_vtable_s *vtable,   // Framebuffer Driver
  const struct fb_area_s *area  // Updated area of framebuffer
) {
  // Access Framebuffer as bytes
  uint8_t *fb = (uint8_t *)g_pinephone_fb0;
  const size_t fbsize = sizeof(g_pinephone_fb0);

  // Copy the Entire Framebuffer to itself,
  // to fix the missing pixels.
  // Not sure why this works.
  for (int i = 0; i < fbsize; i++) {

    // Declare as volatile to prevent compiler optimization
    volatile uint8_t v = fb[i];
    fb[i] = v;
  }
  return OK;
}

With the code above, the Red, Orange and Yellow Boxes are now rendered correctly in our NuttX Framebuffer Driver for PinePhone. (Pic above)

Instead of copying the entire RAM Framebuffer, can we copy only the updated screen area?

Yep probably, we need more rigourous testing.

But how do we really fix this?

We need to flush the CPU Cache, and verify that our Framebuffer has been mapped with the right attributes.

(Thanks to Barry Nolte, Victor Suarez Rovere and crzwdjk for the tips!)

(Commenters on Hacker News also think it’s a CPU Cache issue)

Who calls pinephone_updatearea?

After writing the pixels to the RAM Framebuffer, NuttX Apps will call ioctl(FBIO_UPDATE) to update the display.

This triggers pinephone_updatearea in our NuttX Framebuffer Driver: fb_main.c

// Omitted: NuttX App writes pixels to RAM Framebuffer

// Update the Framebuffer
#ifdef CONFIG_FB_UPDATE
  ret = ioctl(    // I/O Control
    state->fd,    // File Descriptor for Framebuffer Driver
    FBIO_UPDATE,  // Update the Framebuffer
    (unsigned long)((uintptr_t)area)  // Updated area
  );
#endif

How do other PinePhone operating systems handle this?

See this…

LVGL on NuttX on PinePhone

§11 What’s Next

Now that we can render Graphical User Interfaces with LVGL Graphics Library… It’s time to build the NuttX Touch Input Driver for PinePhone!

The NuttX Community is now adding support for I2C on Allwinner A64 SoC, which will be super helpful for our I2C Touch Input Driver. Stay Tuned!

Please check out the other articles on NuttX for PinePhone…

Many Thanks to my GitHub Sponsors for supporting my work! This article wouldn’t have been possible without your support.

Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…

lupyuen.github.io/src/fb.md