Wayland and LVGL on PinePhone with Ubuntu Touch

Work-in-progress LVGL GUI Framework ported to Wayland EGL on PinePhone with Ubuntu Touch

Work-in-progress LVGL GUI Framework ported to Wayland EGL on PinePhone with Ubuntu Touch

We ❤️   Old Underwear...

They feel comfy, they fit our contours. Nevermind the holes and the old stains 🤢

X11 is like Old Underwear. It's been around for 30 years... Yet we still use it in spite of its feature gaps and wonky quirks.

PinePhone on Ubuntu Touch feels like... New Underwear.

It runs Linux but it has none of the legacy X11 code. Because it's optimised for a great mobile experience with Wayland.

But New Underwear feels uncomfortable. So today we'll learn Wayland and understand how apps are built with Wayland.

Hopefully someday we'll move on to newer, simpler app frameworks (like LVGL and Flutter) as we discard our Old Underwear: X11, SDL, GTK, Qt, ...

The source code for this article may be found here...

1 How X11 works

X11 is the Graphical Display Service that runs on most Linux desktops and notebooks.

Let's hunt for the X11 Service on Pinebook Pro...

X11 Service on Pinebook Pro

That's the X11 Service... A 2.2 MB executable named Xorg.

The X11 Service controls the rendering of Linux apps (as well as the keyboard and mouse input) like this...

X11 Architecture

(Adapted from "Wayland Architecture")

  1. At the top we have the Linux programs running on our Linux machine: Terminal, Editor, Web Browser.

    Each program renders its graphical display and transmits the raw graphics to the X11 Service (via a local TCP socket).

  2. X11 Service forwards the rendered graphics to the Window Manager / Compositor.

    The Window Manager / Compositor is provided by the Desktop Environment: Xfce, KDE, Gnome, ...

  3. The Window Manager / Compositor wraps the rendered graphics into Display Windows and "decorates" them with scrollbars, title bar and minimise / maximise / close buttons.

    The Window Manager / Compositor then draws the Display Windows into a Screen Buffer according to their screen coordinates.

  4. The Screen Buffer is rendered to our screen by the X11 Service, talking to the Linux Display Driver.

  5. Any keyboard and mouse input is captured by the X11 Service, and forwarded to the programs.

Why is X11 so complex? So many hops?

Because X11 was designed for Distributed Computing Systems.

Here's how I used (abused?) X11R4 at UIUC Systems Research Group way back in 1990 (30 years ago!)...

Distributed X11 System

Thankfully things are a lot simpler now, lemme explain...

2 Wayland on Ubuntu Touch

Do we need overlapping or tiled windows on PinePhone?

Do we need to need to decorate PinePhone windows with a title bar and minimise / maximise / close buttons?

Do we even need any windows on PinePhone?

No! Because each PinePhone app takes control of the entire screen!

PinePhone uses a simpler Graphical Display Service: the Wayland Compositor.

Let's hunt for the Wayland Compositor on PinePhone...

Wayland Compositor on PinePhone

That's the Wayland Compositor... A 262 KB executable named unity-system-compositor.

Compare that with the 2.2 MB X11 Server on Pinebook Pro!

Here's how the Wayland Compositor controls apps and touchscreen input on PinePhone with Ubuntu Touch...

Wayland Architecture

(Adapted from "Wayland Architecture" and "EGL API")

  1. At the top we have the apps running on our phone: Terminal, Editor, Web Browser.

    Since each app runs fullscreen, only the active app will be rendered.

    When then app starts, it queries the Wayland Compositor for the graphical display interfaces available. (They talk via a Linux socket file: /run/user/32011/wayland-0)

  2. Wayland Compositor returns the EGL Interface to the app.

  3. App calls the EGL Interface to render OpenGL graphics directly to the Linux Display Driver.

  4. Linux Display Driver forwards the OpenGL rendering commands to the GPU to update the screen.

  5. Any touchscreen input is captured by the Wayland Compositor, and forwarded to the active app.

Wayland looks so much simpler and faster than X11!

Wayland is designed for OpenGL and GPUs?

Yes! And I lied about Wayland being New Underwear... Wayland is not really that New!

Wayland was first released in 2008 (11 years ago)... Yet it was designed around OpenGL and GPUs, the same tech that powers our beautiful games today. (And websites too)

Read on to learn how to render our own OpenGL graphics with Wayland and Ubuntu Touch on PinePhone...

Rendering a yellow rectangle with Wayland and OpenGL on PinePhone

Rendering a yellow rectangle with Wayland and OpenGL on PinePhone

3 Render OpenGL Graphics with Wayland

Here's the function that calls OpenGL to render the yellow box above: pinephone-mir/egl.c

/// Render the OpenGL ES2 display
static void render_display() {
    //  Fill the rectangular region with yellow
    glClearColor(
        1.0,  //  Red
        1.0,  //  Green
        0.0,  //  Blue
        1.0   //  Alpha
    );
    glClear(GL_COLOR_BUFFER_BIT);

    // Render now
    glFlush();
}

render_display() looks exactly like normal OpenGL, and it works on PinePhone with Wayland! (Thanks to Ubuntu Touch)

Two things to note...

  1. PinePhone supports a popular subset of OpenGL, known as OpenGL for Embedded Systems Version 2.0.

    OpenGL ES is optimised for Embedded Devices. It's used by many mobile and console games today.

  2. To render OpenGL ES graphics, we need to get the OpenGL ES Context and Window Surface from Wayland

Before calling render_display(), we fetch the OpenGL Window Surface from Wayland like so: pinephone-mir/egl.c

/// Dimensions of the OpenGL region to be rendered
static int WIDTH  = 480;
static int HEIGHT = 360;

static struct wl_egl_window *egl_window;  //  Wayland EGL Window
static EGLSurface egl_surface;            //  EGL Surface

//  Create the EGL Window and render OpenGL graphics
static void create_window(void) {
    //  Create an EGL Window from a Wayland Surface 
    egl_window = wl_egl_window_create(surface, WIDTH, HEIGHT);
    assert(egl_window != EGL_NO_SURFACE);  //  Failed to create OpenGL Window

    //  Create an OpenGL Window Surface for rendering
    egl_surface = eglCreateWindowSurface(egl_display, egl_conf,
        egl_window, NULL);
    assert(egl_surface != NULL);  //  Failed to create OpenGL Window Surface

    //  Set the current rendering surface
    EGLBoolean madeCurrent = eglMakeCurrent(egl_display, egl_surface,
        egl_surface, egl_context);
    assert(madeCurrent);  //  Failed to set rendering surface

    //  Render the display
    render_display();

    //  Swap the display buffers to make the display visible
    EGLBoolean swappedBuffers = eglSwapBuffers(egl_display, egl_surface);
    assert(swappedBuffers);  //  Failed to swap display buffers
}

Functions named wl_egl_... are provided by the Wayland EGL Interface. Functions named egl... come from the cross-platform Mesa 3D Graphics Library.

EGL vs OpenGL... What's the difference?

In Wayland, EGL is the Enabler for OpenGL.

Wayland only understands EGL and it will gladly hand us EGL objects... But it's up to us to transform EGL into OpenGL for rendering.

Thus in the code above, we take a Wayland Surface surface and transform it into an EGL Window egl_window...

//  Create an EGL Window from a Wayland Surface 
egl_window = wl_egl_window_create(surface, WIDTH, HEIGHT);

Then we create an OpenGL Window Surface egl_surface from that EGL Window...

//  Create an OpenGL Window Surface for rendering
egl_surface = eglCreateWindowSurface(egl_display, egl_conf,
    egl_window, NULL);

And we begin the OpenGL rendering...

//  Set the current rendering surface
eglMakeCurrent(egl_display, egl_surface,
    egl_surface, egl_context);

//  Render the display
render_display();

//  Swap the display buffers to make the display visible
eglSwapBuffers(egl_display, egl_surface);

Here's how we create a Wayland Region for OpenGL rendering: pinephone-mir/egl.c

static struct wl_region *region;  //  Wayland Region

//  Create the Wayland Region for rendering OpenGL graphics
static void create_opaque_region(void) {
    //  Create a Wayland Region
    region = wl_compositor_create_region(compositor);
    assert(region != NULL);  //  Failed to create EGL Region

    //  Set the dimensions of the Wayland Region
    wl_region_add(region, 0, 0, WIDTH, HEIGHT);

    //  Add the Region to the Wayland Surface
    wl_surface_set_opaque_region(surface, region);
}

To learn more about EGL, check out "Programming Wayland Clients"

The Wayland EGL code in this article was adapted from that document.

4 Get EGL Context from Wayland

Earlier in create_window() we called an EGL Context egl_context to render OpenGL graphics.

Here's how we get the EGL Context: pinephone-mir/egl.c

/// Wayland EGL Interfaces for OpenGL Rendering
static EGLDisplay egl_display;  //  EGL Display
static EGLConfig  egl_conf;     //  EGL Configuration
static EGLContext egl_context;  //  EGL Context

//  Create the EGL Context for rendering OpenGL graphics
static void init_egl(void) {
    //  Attributes for our EGL Display
    EGLint config_attribs[] = {
        EGL_SURFACE_TYPE,    EGL_WINDOW_BIT,
        EGL_RED_SIZE,        8,
        EGL_GREEN_SIZE,      8,
        EGL_BLUE_SIZE,       8,
        EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
        EGL_NONE
    };
    static const EGLint context_attribs[] = {
        EGL_CONTEXT_CLIENT_VERSION, 2,
        EGL_NONE
    };

    //  Get the EGL Display
    egl_display = eglGetDisplay((EGLNativeDisplayType) display);
    assert(egl_display != EGL_NO_DISPLAY);  //  Failed to get EGL Display

    //  Init the EGL Display
    EGLint major, minor;
    EGLBoolean egl_init = eglInitialize(egl_display, &major, &minor);
    assert(egl_init);  //  Failed to init EGL Display

    //  Get the EGL Configurations
    EGLint count, n;
    eglGetConfigs(egl_display, NULL, 0, &count);
    EGLConfig *configs = calloc(count, sizeof *configs);
    eglChooseConfig(egl_display, config_attribs,
        configs, count, &n);

    //  Choose the first EGL Configuration
    for (int i = 0; i < n; i++) {
        EGLint size;
        eglGetConfigAttrib(egl_display, configs[i], EGL_BUFFER_SIZE, &size);
        eglGetConfigAttrib(egl_display, configs[i], EGL_RED_SIZE, &size);
        egl_conf = configs[i];
        break;
    }
    assert(egl_conf != NULL);  //  Failed to get EGL Configuration

    //  Create the EGL Context based on the EGL Display and Configuration
    egl_context = eglCreateContext(egl_display, egl_conf,
        EGL_NO_CONTEXT, context_attribs);
    assert(egl_context != NULL);  //  Failed to create EGL Context
}

More about EGL

The above code in init_egl() creates the EGL Context.

We call init_egl() in our Main Function like so: pinephone-mir/egl.c

/// Wayland Interfaces
static struct wl_surface       *surface;       //  Wayland Surface
static struct wl_shell_surface *shell_surface; //  Wayland Shell Surface

/// Connect to Wayland Compositor and render OpenGL graphics
int main(int argc, char **argv) {
    //  Get interfaces for Wayland Compositor and Wayland Shell
    get_server_references();
    assert(display != NULL);     //  Failed to get Wayland Display
    assert(compositor != NULL);  //  Failed to get Wayland Compositor
    assert(shell != NULL);       //  Failed to get Wayland Shell

    //  Create a Wayland Surface for rendering our app
    surface = wl_compositor_create_surface(compositor);
    assert(surface != NULL);  //  Failed to create Wayland Surface

    //  Get the Wayland Shell Surface for rendering our app window
    shell_surface = wl_shell_get_shell_surface(shell, surface);
    assert(shell_surface != NULL);

    //  Set the Shell Surface as top level
    wl_shell_surface_set_toplevel(shell_surface);

    //  Create the Wayland Region for rendering OpenGL graphics
    create_opaque_region();

    //  Create the EGL Context for rendering OpenGL graphics
    init_egl();

    //  Create the EGL Window and render OpenGL graphics
    create_window();

    //  Handle all Wayland Events in the Event Loop
    while (wl_display_dispatch(display) != -1) {}

    //  Disconnect from the Wayland Display
    wl_display_disconnect(display);
    return 0;
}

In all Wayland apps, the main() function follows the same steps...

  1. Fetch the Wayland Compositor and Wayland Shell from the Wayland Registry...

    //  Get interfaces for Wayland Compositor and Wayland Shell
    get_server_references();
    

    We'll talk about get_server_references() and the Wayland Registry in a while.

  2. Every Wayland App needs a Wayland Surface (screen buffer) for displaying the app...

    //  Create a Wayland Surface for rendering our app
    surface = wl_compositor_create_surface(compositor);
    
  3. Create a Wayland Shell Surface (app window) for rendering our app...

    //  Get the Wayland Shell Surface for rendering our app window
    shell_surface = wl_shell_get_shell_surface(shell, surface);
    
  4. Set the Shell Surface as the Top Level window for our app...

    //  Set the Shell Surface as top level
    wl_shell_surface_set_toplevel(shell_surface);
    
  5. This part is specific to OpenGL apps...

    Earlier we have seen create_opaque_region(), init_egl() and create_window(). We call them to create the Wayland Region, EGL Context and EGL Window, and to render the OpenGL graphics.

    //  Create the Wayland Region for rendering OpenGL graphics
    create_opaque_region();
    //  Create the EGL Context for rendering OpenGL graphics
    init_egl();
    //  Create the EGL Window and render OpenGL graphics
    create_window();
    
  6. Every Wayland App needs an Event Loop for handling Wayland Events...

    //  Handle all Wayland Events in the Event Loop
    while (wl_display_dispatch(display) != -1) {}
    
  7. When our app terminates, we disconnect the Wayland Display...

    //  Disconnect from the Wayland Display
    wl_display_disconnect(display);
    

Now let's build and test the app on our Linux development machine. (We'll run it on PinePhone later)

5 Build and Test Wayland App on Linux

Now that we have created a simple Wayland app that renders OpenGL graphics... Let's build it!

Building a Wayland app is refreshingly simple (if you're used to GDK, Qt and SDL).

Here's how we build the Wayland app in egl.c on a Linux machine (that has Wayland, MESA EGL and OpenGL ES2 libraries installed)...

# Build the Wayland EGL app
gcc \
    -g \
    -o egl \
    egl.c \
    -Wl,-Map=egl.map \
    -L/usr/lib/aarch64-linux-gnu/mesa-egl \
    -lwayland-client \
    -lwayland-server \
    -lwayland-egl \
    -lEGL \
    -lGLESv2

This produces the executable app egl. Run the egl app on our Linux machine like so...

# Install Weston Wayland Compositor...
# For Arch Linux and Manjaro:
sudo pacman -S weston

# For Other Distros:
# Check https://github.com/wayland-project/weston

# Start the Weston Wayland Compositor on our computer with the PinePhone screen dimensions
weston --width=720 --height=1398 &

# Run the Wayland EGL app
./egl

This uses the Weston Compositor, the reference implementation of the Wayland Compositor that runs on X11.

We'll see this Inception-like window within a window...

EGL App running with Wayland Weston Compositor on Pinebook Pro

We learn in a while how to build and run the app on PinePhone.

6 Fetch Wayland Interfaces

Earlier we used the Wayland Compositor and the Wayland Shell in our app...

  1. Wayland Compositor (compositor): Manages the screen buffer used by apps

  2. Wayland Shell (shell): Manages the app windows

Here's how we fetch the two interfaces from Wayland: pinephone-mir/egl.c

/// Wayland Interfaces
static struct wl_display       *display;       //  Wayland Display
static struct wl_compositor    *compositor;    //  Wayland Compositor
static struct wl_shell         *shell;         //  Wayland Shell

/// Connect to Wayland Service and fetch the interfaces for Wayland Compositor and Wayland Shell
static void get_server_references(void) {
    //  Connect to the Wayland Service
    display = wl_display_connect(NULL);
    if (display == NULL) {
        fprintf(stderr, "Failed to connect to display\n");
        exit(1);
    }

    //  Get the Wayland Registry
    struct wl_registry *registry = wl_display_get_registry(display);
    assert(registry != NULL);  //  Failed to get Wayland Registry

    //  Add Registry Callbacks to handle interfaces returned by Wayland Service
    wl_registry_add_listener(registry, &registry_listener, NULL);

    //  Wait for Registry Callbacks to fetch Wayland Interfaces
    wl_display_dispatch(display);
    wl_display_roundtrip(display);

    //  We should have received interfaces for Wayland Compositor and Wayland Shell
    assert(compositor != NULL);  //  Failed to get Wayland Compositor
    assert(shell != NULL);       //  Failed to get Wayland Shell
}

What happens inside get_server_references()?

  1. The Wayland Compositor runs as a Linux Service that listens on a Linux Socket File: /run/user/32011/wayland-0 for PinePhone on Ubuntu Touch.

    We connect to the Wayland Service like so...

    //  Connect to the Wayland Service
    display = wl_display_connect(NULL);
    

    Remember that all functions named wl_... come from the Wayland Library.

  2. To work with the Wayland Service, we fetch the Interfaces for the Wayland Compositor and Wayland Shell.

    Wayland Interfaces are defined in the Wayland Registry...

    //  Get the Wayland Registry
    struct wl_registry *registry = wl_display_get_registry(display);
    
  3. To fetch the Compositor and Shell from the Wayland Registry, we add a Registry Listener (more about this later)...

    //  Add Registry Callbacks to handle interfaces returned by Wayland Service
    wl_registry_add_listener(registry, &registry_listener, NULL);
    
  4. Now we dispatch the Registry Listener request to the Wayland Service. (Remember that the Wayland Service operates on Linux Socket Messages)

    //  Wait for Registry Callbacks to fetch Wayland Interfaces
    wl_display_dispatch(display);
    wl_display_roundtrip(display);
    

And we'll get the compositor and shell objects populated from the Wayland Registry!

If you're curious, the Registry Listener works like this: pinephone-mir/egl.c

/// Callbacks for interfaces returned by Wayland Service
static const struct wl_registry_listener registry_listener = {
    global_registry_handler,
    global_registry_remover
};

/// Callback for interfaces returned by Wayland Service
static void global_registry_handler(void *data, struct wl_registry *registry, uint32_t id,
    const char *interface, uint32_t version) {
    printf("Got interface %s id %d\n", interface, id);

    if (strcmp(interface, "wl_compositor") == 0) {
        //  Bind to Wayland Compositor Interface
        compositor = wl_registry_bind(registry, id,
            &wl_compositor_interface,   //  Interface Type
            1);                         //  Interface Version
    } else if (strcmp(interface, "wl_shell") == 0){
        //  Bind to Wayland Shell Interface
        shell = wl_registry_bind(registry, id,
            &wl_shell_interface,        //  Interface Type
            1);                         //  Interface Version
    }
}

global_registry_handler() is the Callback Function that will be triggered for every interface in the Wayland Registry.

The Wayland Service for Ubuntu Touch unity-system-compositor returns a whole bunch of interesting Wayland Interfaces (like qt_windowmanager).

But today we'll bind to the Compositor Interface named wl_compositor and Shell Interface named wl_shell.

And that's how we render a yellow rectangle with Wayland and OpenGL!

Let's move on to something more interesting: Rendering a simple bitmap texture...

Rendering a simple bitmap texture with Wayland and OpenGL on PinePhone

Rendering a simple bitmap texture with Wayland and OpenGL on PinePhone

7 Render OpenGL Bitmap Texture with Wayland

The four boxes we see above are rendered from a magnified 2-pixel by 2-pixel bitmap: pinephone-mir/texture.c

// 2x2 Image, 3 bytes per pixel (R, G, B)
GLubyte pixels[4 * 3] = {
    255, 0, 0,  // Red
    0, 255, 0,  // Green
    0, 0, 255,  // Blue
    255, 255, 0 // Yellow
};

We render the bitmap by creating an OpenGL Texture: pinephone-mir/texture.c

// Create a simple 2x2 texture image with four different colors
GLuint CreateSimpleTexture2D() {
    // Texture object handle
    GLuint textureId;

    // 2x2 Image, 3 bytes per pixel (R, G, B)
    GLubyte pixels[4 * 3] = {
        255, 0, 0,  // Red
        0, 255, 0,  // Green
        0, 0, 255,  // Blue
        255, 255, 0 // Yellow
    };

    // Use tightly packed data
    glPixelStorei(GL_UNPACK_ALIGNMENT, 1);

    // Generate a texture object
    glGenTextures(1, &textureId);

    // Bind the texture object
    glBindTexture(GL_TEXTURE_2D, textureId);

    // Load the texture
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 2, 2, 0, GL_RGB, GL_UNSIGNED_BYTE, pixels);

    // Set the filtering mode
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
    return textureId;
}

(Not the most efficient way to render a bitmap... But let's try this and test drive PinePhone's GPU!)

This is the usual way we create an OpenGL Texture, as explained in "OpenGL® ES 3.0 Programming Guide".

Here comes the tricky part... Before rendering the OpenGL Texture, we need to program the GPU Shaders on PinePhone with a C-like language: pinephone-mir/texture.c

// Initialize the shader and program object
int Init(ESContext *esContext) {
    esContext->userData = malloc(sizeof(UserData));
    UserData *userData = esContext->userData;
    GLbyte vShaderStr[] =
        "attribute vec4 a_position;   \n"
        "attribute vec2 a_texCoord;   \n"
        "varying vec2 v_texCoord;     \n"
        "void main()                  \n"
        "{                            \n"
        "   gl_Position = a_position; \n"
        "   v_texCoord = a_texCoord;  \n"
        "}                            \n";

    GLbyte fShaderStr[] =
        "precision mediump float;                            \n"
        "varying vec2 v_texCoord;                            \n"
        "uniform sampler2D s_texture;                        \n"
        "void main()                                         \n"
        "{                                                   \n"
        "  gl_FragColor = texture2D( s_texture, v_texCoord );\n"
        "}                                                   \n";

    // Load the shaders and get a linked program object
    userData->programObject = esLoadProgram(vShaderStr, fShaderStr);
    ...

(Yep a C program within a C program... Inception!)

esLoadProgram() is defined in pinephone-mir/shader.c

We're now talking to PinePhone's GPU, which is so low-level that it understand only Triangles, not Rectangles.

Hence to render the OpenGL Texture, we map the Rectangular Texture onto two Triangles and render them: pinephone-mir/texture.c

// Draw a triangle using the shader pair created in Init()
void Draw(ESContext *esContext) {
    GLfloat vVertices[] = {
        -0.5f,   0.5f,  0.0f,  // Position 0
         0.0f,   0.0f,         // TexCoord 0
        -0.5f,  -0.5f,  0.0f,  // Position 1
         0.0f,   1.0f,         // TexCoord 1
         0.5f,  -0.5f,  0.0f,  // Position 2
         1.0f,   1.0f,         // TexCoord 2
         0.5f,   0.5f,  0.0f,  // Position 3
         1.0f,   0.0f          // TexCoord 3
    };
    GLushort indices[] = {
        0, 1, 2,  //  First Triangle
        0, 2, 3   //  Second Triangle
    };
    ...
    //  Draw the 6 vertices as 2 triangles
    glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_SHORT, indices);
}

(Yes the math is starting to hurt... But that's the end of it!)

Finally we connect the above code to render the four colour boxes on PinePhone, thanks to Wayland and OpenGL: pinephone-mir/egl2.c

/// Render the OpenGL ES2 display
static void render_display() {
    //  Create the texture context
    static ESContext esContext;
    esInitContext ( &esContext );
    esContext.width  = WIDTH;
    esContext.height = HEIGHT;

    //  Draw the texture
    Init(&esContext);
    Draw(&esContext);

    //  Render now
    glFlush();
}

And that's our Wayland App that renders a simple OpenGL Bitmap Texture!

The OpenGL Texture code in this article was adapted from "OpenGL® ES 2.0 Programming Guide"

Let's head on towards greatness and something really useful: Graphical User Interfaces...

Button rendered with LVGL and Wayland on PinePhone

Button rendered with LVGL and Wayland on PinePhone

8 LVGL Toolkit for Graphical User Interfaces

Now that we can render bitmaps on PinePhone, let's think...

How would we render a simple Graphical User Interface (GUI) on PinePhone, like the button above?

Why don't we use a simple GUI Toolkit like LVGL? (Formerly LittleVGL)

Here's how we call the LVGL library to render that button: lvgl-wayland/wayland/lvgl.c

#include "../lvgl.h"

/// Render a Button Widget and a Label Widget
static void render_widgets(void) {
    lv_obj_t * btn = lv_btn_create(lv_scr_act(), NULL);     //  Add a button the current screen
    lv_obj_set_pos(btn, 10, 10);                            //  Set its position
    lv_obj_set_size(btn, 120, 50);                          //  Set its size

    lv_obj_t * label = lv_label_create(btn, NULL);          //  Add a label to the button
    lv_label_set_text(label, "Button");                     //  Set the labels text
}

Easy peasy!

LVGL is a simple C toolkit designed for Embedded Devices, so it needs very little memory and processing power. LVGL is self-contained... Fonts and icons are bundled into the LVGL library.

It's used on PineTime Smart Watch to render watch faces.

LVGL doesn't run on Wayland yet... But we'll fix that!

Remember how we rendered a simple 2-pixel by 2-pixel bitmap by creating an OpenGL Texture with CreateSimpleTexture2D()?

Let's extend that bitmap to cover the entire PinePhone screen: 720 pixels by 1398 pixels.

And we create the OpenGL Texture for the entire PinePhone screen like so: lvgl-wayland/wayland/texture.c

///  PinePhone Screen Resolution, defined in lv_conf.h
#define LV_HOR_RES_MAX          (720)
#define LV_VER_RES_MAX          (1398)
#define LV_SCALE_RES            1

///  Screen buffer 
#define BYTES_PER_PIXEL 3
GLubyte pixels[LV_HOR_RES_MAX * LV_VER_RES_MAX * BYTES_PER_PIXEL];

/// Create an OpenGL Texture for the screen buffer
GLuint CreateTexture(void) {
    GLuint texId;
    glGenTextures ( 1, &texId );
    glBindTexture ( GL_TEXTURE_2D, texId );

    glTexImage2D (
        GL_TEXTURE_2D, 
        0,  //  Level
        GL_RGB, 
        LV_HOR_RES_MAX,  //  Width
        LV_VER_RES_MAX,  //  Height 
        0,  //  Format 
        GL_RGB, 
        GL_UNSIGNED_BYTE, 
        pixels 
    );
    glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
    glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
    glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
    glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
    return texId;
}

pixels is the screen buffer that will contain the pixels for our rendered UI controls, like our button.

We'll tell LVGL to render into pixels like so: lvgl-wayland/wayland/texture.c

/// Set the colour of a pixel in the screen buffer
void put_px(uint16_t x, uint16_t y, uint8_t r, uint8_t g, uint8_t b, uint8_t a) {
    assert(x >= 0); assert(x < LV_HOR_RES_MAX);
    assert(y >= 0); assert(y < LV_VER_RES_MAX);
    int i = (y * LV_HOR_RES_MAX * BYTES_PER_PIXEL) + (x * BYTES_PER_PIXEL);
    pixels[i++] = r;  //  Red
    pixels[i++] = g;  //  Green
    pixels[i++] = b;  //  Blue
}

(Simplistic, not efficient though)

We'll render the OpenGL Texture the same way as before: lvgl-wayland/wayland/lvgl.c

/// Render the OpenGL ES2 display
static void render_display() {
    //  This part is new...

    //  Init the LVGL display
    lv_init();
    lv_port_disp_init();

    //  Create the LVGL widgets: Button and label
    render_widgets();

    //  Render the LVGL widgets
    lv_task_handler();

    //  This part is the same as before...

    //  Create the texture context
    static ESContext esContext;
    esInitContext ( &esContext );
    esContext.width  = WIDTH;
    esContext.height = HEIGHT;

    //   Draw the texture
    Init(&esContext);
    Draw(&esContext);

    //  Render now
    glFlush();
}

But now we have injected the calls to the LVGL library...

  1. lv_init(): Initialise the LVGL library

  2. lv_port_disp_init(): Initialise our display

  3. render_widgets(): Calls the LVGL library to create two UI controls: a Button and a Label

  4. lv_task_handler(): Let LVGL render the UI controls into our screen buffer

Now let's tweak the LVGL library to render UI controls into our screen buffer pixels

9 Port LVGL to Wayland

Porting LVGL to Wayland and Ubuntu Touch is straightforward.

According to the LVGL Porting Doc, we need to code a Flush Callback Function disp_flush() that will be called by LVGL to render UI controls to the screen buffer.

Here's our implementation for Wayland: lvgl-wayland/wayland/lv_port_disp.c

//  Flush the content of the internal buffer to the specific area on the display
//  You can use DMA or any hardware acceleration to do this operation in the background but
//  'lv_disp_flush_ready()' has to be called when finished.
static void disp_flush(lv_disp_drv_t * disp_drv, const lv_area_t * area, lv_color_t * color_p) {
    //  The most simple case (but also the slowest) to put all pixels to the screen one-by-one
    for(int32_t y = area->y1; y <= area->y2; y++) {
        for(int32_t x = area->x1; x <= area->x2; x++) {
            //  Put a pixel to the screen buffer
            put_px(x, y, 
                color_p->ch.red, 
                color_p->ch.green, 
                color_p->ch.blue, 
                0xff);
            color_p++;
        }
    }
    //  Inform the graphics library that we are ready with the flushing
    lv_disp_flush_ready(disp_drv);
}

We've seen earlier that put_px() draws pixels in the simplest way possible. Eventually we should use PinePhone's GPU for rendering LVGL controls, by implementing the LVGL GPU Callbacks.

Light and Dark Themes are provided by LVGL. To select the default theme just edit lvgl-wayland/lv_conf.h

Here's Dark Theme...

//  For Dark Theme...
#define LV_THEME_DEFAULT_FLAG LV_THEME_MATERIAL_FLAG_DARK

LVGL Dark Theme with Wayland on PinePhone

And Light Theme...

//  For Light Theme...
#define LV_THEME_DEFAULT_FLAG LV_THEME_MATERIAL_FLAG_LIGHT

LVGL Light Theme with Wayland on PinePhone

The screens above were rendered by updating one line in lvgl-wayland/wayland/lvgl.c...

/// Render the OpenGL ES2 display
static void render_display() {
    //  Init the LVGL display
    lv_init();
    lv_port_disp_init();

    //  Create the LVGL widgets
    lv_demo_widgets();  //  Previously render_widgets()

lv_demo_widgets() comes from lvgl-wayland/demo/lv_demo_widgets.c

What about Touch Input in LVGL for Ubuntu Touch?

We haven't handled Touch Input yet... Lemme know if you're keen to help!

Do we really have to code LVGL Apps for PinePhone in C?

Rust is supported too!

We may write LVGL Apps for PinePhone in Rust by calling the lvgl-rs Rust Wrapper for LVGL by Rafael Carício.

(Fun Fact: lvgl-rs was originally created for PineTime Smart Watch... Today it's used by Rust on PlayStation Portable too!)

Size of LVGL Demo App on PinePhone with Ubuntu Touch

How small is LVGL on PinePhone with Ubuntu Touch?

1.5 MB for the Light / Dark Theme LVGL Demo App above.

Not that big, considering that the font, icons and debugging symbols are bundled inside.

How does LVGL compare with Qt, GTK and SDL on PinePhone with Ubuntu Touch?

Qt is the only officially supported App Toolkit on Ubuntu Touch.

GTK and SDL are supposed to work on Wayland... But I couldn't get them to work on Ubuntu Touch.

(Probably because legacy X11 compatibility is missing from Ubuntu Touch, i.e. XWayland)

I applaud the maintainers of X11, Qt, GTK and SDL because every new release needs to support so many legacy features. Kudos!

But what if we could start from scratch, drop the legacy stuff, and build a simpler UI toolkit for Wayland?

LVGL is the experiment that we're undertaking today!

10 Build LVGL on PinePhone with Ubuntu Touch

Follow these steps to build LVGL on PinePhone over SSH.

(If we haven't enabled SSH on PinePhone, check the "Configure SSH on PinePhone" instructions below)

Connect to PinePhone over SSH and enter these commands...

# Make system folders writeable before installing any packages
sudo mount -o remount,rw /

# Install dev tools and GLES2 library
sudo apt install gcc gdb git make libgles2-mesa-dev

# Download the source code
cd ~
git clone https://github.com/lupyuen/lvgl-wayland
cd lvgl-wayland

# Build the app
make

This creates the executable ~/lvgl-wayland/wayland/lvgl

Can we just run lvgl from the Terminal Command Line?

Nope! Because Wayland and Ubuntu Touch are super-secure, thanks to AppArmor.

But there's a way to trick AppArmor into allowing lvgl to be launched (since we are Superuser).

Read on to learn how...

Fighting AppArmor Security... Permission Denied!

Fighting AppArmor Security... Permission Denied!

11 Inject LVGL into File Manager App

For rapid testing (and to work around AppArmor), we shall replace the File Manager app by our lvgl app because File Manager has no AppArmor restrictions (Unconfined).

(More about AppArmor in a while)

Connect to PinePhone over SSH and enter these commands...

# Make system folders writeable and go to File Manager Click Package folder
sudo mount -o remount,rw /
cd /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager

# Back up the desktop file. Restore this desktop file to go back to the normal File Manager
sudo cp com.ubuntu.filemanager.desktop com.ubuntu.filemanager.desktop.old

# Edit the desktop file
sudo nano com.ubuntu.filemanager.desktop 

We're now altering the behaviour of File Manager, by tampering with the Click Package settings for File Manager.

(Why are we tampering with a Click Package? We'll learn in a while)

Change the Exec line from...

Exec=filemanager

To...

Exec=./run.sh

Save and exit nano

We have modded the File Manager icon so that it now launches run.sh instead of the usual filemanager executable.

(It's like switching the executable for a Windows Explorer Shortcut)

We'll be installing run.sh later with a script: lvgl.sh

In the meantime, check that run.sh (located at ~/lvgl-wayland/wayland) contains the following...

# Log Wayland messages
export WAYLAND_DEBUG=1

# Run lvgl app
./lvgl

If we see this...

# Debug lvgl app
gdb \
    -ex="r" \
    -ex="bt" \
    -ex="frame" \
    --args ./lvgl

It means that the lvgl app will be started with the gdb debugger.

If it crashes with a bad C pointer, the gdb debugger will show a helpful stack trace.

And this...

# Run lvgl app with strace
./strace \
   -s 1024 \
   ./lvgl

Is for tracing the lvgl app with strace. It shows everything done by the app.

Check out this strace log for the File Manager on Ubuntu Touch

12 Run LVGL on PinePhone with Ubuntu Touch

Finally we're ready to run our lvgl app!

Connect to PinePhone over SSH and enter these commands...

cd ~/lvgl-wayland
./wayland/lvgl.sh

The script lvgl.sh copies run.sh from ~/lvgl-wayland/wayland to the Click Package Folder for File Manager...

/usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager

In a few seconds we'll see the message...

*** Tap on File Manager icon on PinePhone

Go ahead and tap the File Manager icon on PinePhone.

Our LVGL App shall run instead of the File Manager.

In the SSH console, press Ctrl-C to stop the log display.

The log file for the app is located at...

/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log

The log for the Wayland Compositor unity-system-compositor may be useful for troubleshooting...

/home/phablet/.cache/upstart/unity8.log

Copy the log files to our machine like this...

scp -i ~/.ssh/pinephone_rsa phablet@192.168.1.160:/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log .
scp -i ~/.ssh/pinephone_rsa phablet@192.168.1.160:/home/phablet/.cache/upstart/unity8.log .

Check out the sample logs

13 Overcome AppArmor Security on Ubuntu Touch

To understand Wayland, AppArmor and Ubuntu Touch Security, let's look inside the script lvgl.sh and discover how it launches our lvgl app...

  1. Our lvgl app doesn't have a close button, so let's terminate the app if it's already running...

    # Kill the app if it's already running
    pkill lvgl
    
  2. In Ubuntu Touch, User Directories (like our Home Directory) are writeable by default.

    System Directories (like /usr/share) are mounted with read-only access, to prevent tampering of system files. (Think malware)

    Since we're Superuser, we may remount System Directories with read-write access...

    # Make system folders writeable
    sudo mount -o remount,rw /
    
  3. Why do we need read-write access?

    Because we'll be copying our app lvgl and the script run.sh to a System Directory...

    # Copy app to File Manager folder
    cd wayland
    sudo cp lvgl /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
    
    # Copy run script to File Manager folder
    sudo cp run.sh /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
    
  4. What's this folder /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager?

    This is the Click Package folder for File Manager.

    Ubuntu Touch Apps (like File Manager) are packaged as Click Packages for installation on our phones.

    When the app is installed, Ubuntu Touch extracts the Click Package into a folder under /usr/share/click.

    Inside the Click Package folder we'll find the executables, libraries and data files that are needed for running the app.

    The folder also contains a .desktop file. (Earlier we've seen com.ubuntu.filemanager.desktop for File Manager) This file tells Ubuntu Touch how to launch the app.

  5. Does the app run as our user account phablet?

    Nope. For security, Ubuntu Touch Apps run under an account with restricted privileges: clickpkg

    This account has no access to our phablet files. That's why we copy the lvgl app and run.sh script to the Click Package folder, which is accessible by clickpkg

    We set the ownership of lvgl and run.sh to clickpkg so that it can execute the files...

    # Set ownership on the app and the run script
    sudo chown clickpkg:clickpkg /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager/lvgl
    sudo chown clickpkg:clickpkg /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager/run.sh
    
  6. Our lvgl app and run.sh script have been staged in the Click Package folder.

    We ask the human to tap the File Manager icon...

    # Start the File Manager
    echo "*** Tap on File Manager icon on PinePhone"
    
  7. Ubuntu Touch launches our lvgl app. As our app runs, it logs debugging messages to Standard Output and Standard Error.

    The messages are captured in this log file...

    # Monitor the log file
    echo >/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log
    tail -f /home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log
    

Why can't we run our lvgl app from the Terminal Command Line?

Because Ubuntu Touch's Wayland Service stops unauthorized processes from grabbing the Compositor...

Stopped by Wayland Security

We see this in the Wayland Compositor log: /home/phablet/.cache/upstart/unity8.log

ApplicationManager REJECTED connection from app with pid 6710 
as it was not launched by upstart, and no desktop_file_hint is specified

That's why we need to inject lvgl into File Manager... So that Wayland thinks that the File Manager is grabbing the Compositor.

Why did we choose the File Manager app instead of another app like Camera?

Because File Manager has Unconfined AppArmor Permissions... It can do anything!

(But still restricted by the clickpkg user permissions)

Look at the AppArmor Policy for the File Manager App: filemanager.apparmor

{
    "policy_version": 16.04,
    "template": "unconfined",
    "policy_groups": []
}

Compare this with the AppArmor Policy for the Camera App: camera.apparmor

{
    "policy_groups": [
        "picture_files",
        "video_files",
        "camera",
        "audio",
        "video",
        "usermetrics",
        "content_exchange",
        "content_exchange_source",
        "location"
    ],
    "policy_version": 16.04,
    "read_path": [
        "@{PROC}/*/mounts",
        "/dev/disk/by-label/"
    ]
}

The AppArmor Policy says that the Camera App may only access selected features (like recording audio and video). And it's only allowed to read specific paths (like /dev/disk/by-label).

strace won't work with the AppArmor Policy for Camera App.

So for tracing our app with strace, we "borrow" the Unconfined AppArmor Policy for File Manager.

To troubleshoot problems with AppArmor, check the system log in /var/log/syslog

Check out my syslog

14 What I like about Ubuntu Touch on PinePhone

While attempting to port the PineTime Companion App to PinePhone with GTK (and failing miserably), I had these thoughts...

  1. AppArmor is good, because iOS and Android have similar apps security

  2. Read-only file system is good (system files are read-only by default, user files are read-write). Helps to prevent security holes. (Even PineTime has a read-only Flash ROM)

  3. Why is Qt supported on Ubuntu Touch and not GTK? Because building a Linux mobile app requires mobile-friendly widgets.

    I think Qt has more mobile-friendly widgets, even through the internal plumbing is way too complicated.

    When I get GTK running on Ubuntu Touch, I will face the same problem with widgets. And I have to make GTK widgets look and feel consistent with Qt / Ubuntu Touch widgets.

    That's why I decided to move away from GTK and focus on a simpler widget framework with LVGL.

  4. Older kernel base in Ubuntu Touch... I don't do kernel hacking much so it doesn't matter to me.

    I think for mobiles we only need to support a few common chipsets, so an older kernel is probably fine.

    That explains why Raspberry Pi 4 isn't supported by Ubuntu Touch... The hardware is just too new.

  5. The issues I'm struggling with now... Wayland, GTK3, ... are actually really old stuff. Updating the kernel won't help.

  6. Ubuntu Touch is pure Wayland, none of the legacy X11 stuff. Xwayland is not even there (unless you use the Libertine containers ugh).

    The pure Wayland environment causes GTK to break, because GTK assumes some minimal X11 support (i.e. Xwayland).

    It's better to start from scratch with a toolkit that's not based on X11, like LVGL.

  7. So Ubuntu Touch is not really that bad for PinePhone... It's just painful for building non-Qt apps. 🙂

After posting my thoughts, the UBports, GNOME and Xfce Community responded with encouraging and insightful notes...

14.1 UBports on Ubuntu Touch, Wayland and Mir

Read about Unity8 / Mir / Lomiri's complicated history

Another article

14.2 GNOME and GTK on Wayland

14.3 Wayland on Xfce

15 What's Next?

Wayland feels like New Underwear... And it needs a New App Toolkit like LVGL to make us comfortable.

If you're keen to help make LVGL the Newer, Simpler App Toolkit on Wayland, Ubuntu Touch and PinePhone, please lemme know! 😀

Check out my RSS Feed

16 Configure SSH on PinePhone

First Thing to do when we get our new PinePhone: Open the PinePhone Back Cover and Remove the Battery Insulation Sticker!

(Can't believe I missed that!)

Second Thing: Protect the SSH Service on PinePhone with SSH Keys. And start the SSH Service only when necessary.

Here's how...

16.1 Generate SSH Keys

  1. On our Computer (not PinePhone), open a Command Prompt. Enter this (and fill in our email)...

    ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
    
  2. When prompted...

    Enter a file in which to save the key

    Press Enter. This stores the new SSH Key in the .ssh folder in our Home Directory.

  3. When prompted...

    Enter a file in which to save the key

    Enter...

    pinephone_rsa

    We'll create an SSH Key Pair named pinephone_rsa (Private Key) and pinephone_rsa.pub (Public Key)

  4. When prompted...

    Enter passphrase

    Press Enter. We won't need a passphrase unless our PinePhone needs to be super-secure.

This creates an SSH Key Pair in the .ssh folder in our Home Directory...

(Adapted from this doc)

16.2 Install SSH Keys

  1. Copy pinephone_rsa.pub from the .ssh folder in our Home Directory to a MicroSD Card.

  2. Insert the MicroSD Card into PinePhone. Copy pinephone_rsa.pub to our Home Directory on PinePhone.

    (Check the section "Copy Files from MicroSD Card on PinePhone" below)

  3. Tap the Terminal icon on PinePhone. Enter...

    # Go to home directory
    cd
    
    # If .ssh folder doesn't exist, create it
    mkdir .ssh
    chmod 700 .ssh
    
    # Set public key as the authorized key
    cp pinephone_rsa.pub .ssh/authorized_keys
    chmod 600 .ssh/authorized_keys
    
    # Show the SSH files
    ls -la ~/.ssh
    

    We should see this...

    drwx------  2 phablet phablet 4096 Jul  7 20:06 .
    drwxr-xr-x 28 phablet phablet 4096 Jul 24 11:38 ..
    -rw-------  1 phablet phablet  743 Jul  7 20:08 authorized_keys

    Check that the permissions (rw) and owner (phablet) are correct.

16.3 Start SSH Service

To start the SSH Service on PinePhone, open the Terminal app.

Create a file named a...

nano a

Type this script into the a file...

#!/bin/sh
# Script to start SSH service and show IP address

# Start SSH service
sudo service ssh start

# Show IP address
ifconfig | \
    grep -v "127.0.0.1" | \
    grep "inet addr:"

# Ping repeatedly to keep WiFi alive
ping google.com

Save the file and exit nano.

(Or download the file from lvgl-wayland/a and copy via a MicroSD Card. Check the next section for instructions.)

When we're ready do coding on PinePhone, enter this at the Terminal command line...

. a 

(There's a space between "." and "a")

The script starts the SSH Service and displays the IP address for PinePhone...

Starting SSH Service on PinePhone

From our Computer, we'll connect to PinePhone at the IP adddress indicated by inet addr, say 192.168.1.160...

ssh -i ~/.ssh/pinephone_rsa phablet@192.168.1.160

And that's how we access PinePhone via SSH!

When we press PinePhone's power button to switch off PinePhone, we'll see ths amusing message from olden times...

Powering off PinePhone

If typing on a touch keyboard is not your thing, try copying the files from a MicroSD card...

How we insert a MicroSD Card into PinePhone at night

How we insert a MicroSD Card into PinePhone at night

17 Copy Files from MicroSD Card on PinePhone

It's useful to transfer files to PinePhone via MicroSD Card, like SSH Keys and the SSH Script a above.

(Sadly PinePhone on Ubuntu Touch doesn't allow receiving files over Bluetooth)

The MicroSD card on PinePhone doesn't appear in the File Manager unless we mount it.

Tap the Terminal icon on PinePhone and enter the following...

ls -l /dev/disk/by-label

We should see something like this...

lrwxrwxrwx 1 root root 15 Jul 23 22:24 BOOT_MNJRO -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 15 Jul 23 22:24 cache -> ../../mmcblk2p8
lrwxrwxrwx 1 root root 15 Jul 23 22:24 ROOT_MNJRO -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 16 Jul 23 22:24 userdata -> ../../mmcblk2p10

These are the Partition Labels on our MicroSD Card.

Let's say we wish to mount the MicroSD Card partition ROOT_MNJRO, which links to /dev/mmcblk0p2...

mkdir /tmp/sdcard
sudo mount /dev/mmcblk0p2 /tmp/sdcard
ls -l /tmp/sdcard

(If we don't see our Patition Label, try mounting the numbered partitions anyway: /dev/mmcblk0p1, p2, p3, ...)

We should see the contents of our MicroSD Card.

The MicroSD Card will now appear in File Manager as /tmp/sdcard, ready for us to copy the files.

Or just copy files from the Command Line like so...

cp /tmp/sdcard/a ~

When we're done, unmount our MicroSD Card...

sudo umount /tmp/sdcard

LVGL App on Pinebook Pro

18 Build and Test LVGL App on Linux

Our LVGL App works on Linux machines like Pinebook Pro...

# Download the source code
git clone https://github.com/lupyuen/lvgl-wayland
cd lvgl-wayland

# Build the lvgl executable
make

# Install Weston Wayland Compositor...
# For Arch Linux and Manjaro:
sudo pacman -S weston

# For Other Distros:
# Check https://github.com/wayland-project/weston

# Start the Weston Wayland Compositor with the PinePhone screen dimensions
weston --width=720 --height=1398 &

# Run the lvgl executable
./wayland/lvgl