Work-in-progress LVGL GUI Framework ported to Wayland EGL on PinePhone with Ubuntu Touch
📝 25 Jul 2020
We ❤️ Old Underwear...
They feel comfy, they fit our contours. Nevermind the holes and the old stains 🤢
X11 is like Old Underwear. It's been around for 30 years... Yet we still use it in spite of its feature gaps and wonky quirks.
PinePhone on Ubuntu Touch feels like... New Underwear.
It runs Linux but it has none of the legacy X11 code. Because it's optimised for a great mobile experience with Wayland.
But New Underwear feels uncomfortable. So today we'll learn Wayland and understand how apps are built with Wayland.
Hopefully someday we'll move on to newer, simpler app frameworks (like LVGL and Flutter) as we discard our Old Underwear: X11, SDL, GTK, Qt, ...
The source code for this article may be found here...
X11 is the Graphical Display Service that runs on most Linux desktops and notebooks.
Let's hunt for the X11 Service on Pinebook Pro...
That's the X11 Service... A 2.2 MB executable named Xorg.
The X11 Service controls the rendering of Linux apps (as well as the keyboard and mouse input) like this...
(Adapted from "Wayland Architecture")
At the top we have the Linux programs running on our Linux machine: Terminal, Editor, Web Browser.
Each program renders its graphical display and transmits the raw graphics to the X11 Service (via a local TCP socket).
X11 Service forwards the rendered graphics to the Window Manager / Compositor.
The Window Manager / Compositor is provided by the Desktop Environment: Xfce, KDE, Gnome, ...
The Window Manager / Compositor wraps the rendered graphics into Display Windows and "decorates" them with scrollbars, title bar and minimise / maximise / close buttons.
The Window Manager / Compositor then draws the Display Windows into a Screen Buffer according to their screen coordinates.
The Screen Buffer is rendered to our screen by the X11 Service, talking to the Linux Display Driver.
Any keyboard and mouse input is captured by the X11 Service, and forwarded to the programs.
Why is X11 so complex? So many hops?
Because X11 was designed for Distributed Computing Systems.
Here's how I used (abused?) X11R4 at UIUC Systems Research Group way back in 1990 (30 years ago!)...
Thankfully things are a lot simpler now, lemme explain...
Do we need overlapping or tiled windows on PinePhone?
Do we need to need to decorate PinePhone windows with a title bar and minimise / maximise / close buttons?
Do we even need any windows on PinePhone?
No! Because each PinePhone app takes control of the entire screen!
PinePhone uses a simpler Graphical Display Service: the Wayland Compositor.
Let's hunt for the Wayland Compositor on PinePhone...
That's the Wayland Compositor... A 262 KB executable named unity-system-compositor.
Compare that with the 2.2 MB X11 Server on Pinebook Pro!
Here's how the Wayland Compositor controls apps and touchscreen input on PinePhone with Ubuntu Touch...
(Adapted from "Wayland Architecture" and "EGL API")
At the top we have the apps running on our phone: Terminal, Editor, Web Browser.
Since each app runs fullscreen, only the active app will be rendered.
When then app starts, it queries the Wayland Compositor for the graphical display interfaces available. (They talk via a Linux socket file: /run/user/32011/wayland-0)
Wayland Compositor returns the EGL Interface to the app.
App calls the EGL Interface to render OpenGL graphics directly to the Linux Display Driver.
Linux Display Driver forwards the OpenGL rendering commands to the GPU to update the screen.
Any touchscreen input is captured by the Wayland Compositor, and forwarded to the active app.
Wayland looks so much simpler and faster than X11!
Wayland is designed for OpenGL and GPUs?
Yes! And I lied about Wayland being New Underwear... Wayland is not really that New!
Wayland was first released in 2008 (11 years ago)... Yet it was designed around OpenGL and GPUs, the same tech that powers our beautiful games today. (And websites too)
Read on to learn how to render our own OpenGL graphics with Wayland and Ubuntu Touch on PinePhone...
Rendering a yellow rectangle with Wayland and OpenGL on PinePhone
Here's the function that calls OpenGL to render the yellow box above: pinephone-mir/egl.c
/// Render the OpenGL ES2 display
static void render_display() {
// Fill the rectangular region with yellow
glClearColor(
1.0, // Red
1.0, // Green
0.0, // Blue
1.0 // Alpha
);
glClear(GL_COLOR_BUFFER_BIT);
// Render now
glFlush();
}
render_display() looks exactly like normal OpenGL, and it works on PinePhone with Wayland! (Thanks to Ubuntu Touch)
Two things to note...
PinePhone supports a popular subset of OpenGL, known as OpenGL for Embedded Systems Version 2.0.
OpenGL ES is optimised for Embedded Devices. It's used by many mobile and console games today.
To render OpenGL ES graphics, we need to get the OpenGL ES Context and Window Surface from Wayland
Before calling render_display(), we fetch the OpenGL Window Surface from Wayland like so: pinephone-mir/egl.c
/// Dimensions of the OpenGL region to be rendered
static int WIDTH = 480;
static int HEIGHT = 360;
static struct wl_egl_window *egl_window; // Wayland EGL Window
static EGLSurface egl_surface; // EGL Surface
// Create the EGL Window and render OpenGL graphics
static void create_window(void) {
// Create an EGL Window from a Wayland Surface
egl_window = wl_egl_window_create(surface, WIDTH, HEIGHT);
assert(egl_window != EGL_NO_SURFACE); // Failed to create OpenGL Window
// Create an OpenGL Window Surface for rendering
egl_surface = eglCreateWindowSurface(egl_display, egl_conf,
egl_window, NULL);
assert(egl_surface != NULL); // Failed to create OpenGL Window Surface
// Set the current rendering surface
EGLBoolean madeCurrent = eglMakeCurrent(egl_display, egl_surface,
egl_surface, egl_context);
assert(madeCurrent); // Failed to set rendering surface
// Render the display
render_display();
// Swap the display buffers to make the display visible
EGLBoolean swappedBuffers = eglSwapBuffers(egl_display, egl_surface);
assert(swappedBuffers); // Failed to swap display buffers
}
Functions named wl_egl_... are provided by the Wayland EGL Interface. Functions named egl... come from the cross-platform Mesa 3D Graphics Library.
EGL vs OpenGL... What's the difference?
In Wayland, EGL is the Enabler for OpenGL.
Wayland only understands EGL and it will gladly hand us EGL objects... But it's up to us to transform EGL into OpenGL for rendering.
Thus in the code above, we take a Wayland Surface surface and transform it into an EGL Window egl_window...
// Create an EGL Window from a Wayland Surface
egl_window = wl_egl_window_create(surface, WIDTH, HEIGHT);
Then we create an OpenGL Window Surface egl_surface from that EGL Window...
// Create an OpenGL Window Surface for rendering
egl_surface = eglCreateWindowSurface(egl_display, egl_conf,
egl_window, NULL);
And we begin the OpenGL rendering...
// Set the current rendering surface
eglMakeCurrent(egl_display, egl_surface,
egl_surface, egl_context);
// Render the display
render_display();
// Swap the display buffers to make the display visible
eglSwapBuffers(egl_display, egl_surface);
Here's how we create a Wayland Region for OpenGL rendering: pinephone-mir/egl.c
static struct wl_region *region; // Wayland Region
// Create the Wayland Region for rendering OpenGL graphics
static void create_opaque_region(void) {
// Create a Wayland Region
region = wl_compositor_create_region(compositor);
assert(region != NULL); // Failed to create EGL Region
// Set the dimensions of the Wayland Region
wl_region_add(region, 0, 0, WIDTH, HEIGHT);
// Add the Region to the Wayland Surface
wl_surface_set_opaque_region(surface, region);
}
To learn more about EGL, check out "Programming Wayland Clients"
The Wayland EGL code in this article was adapted from that document.
Earlier in create_window() we called an EGL Context egl_context to render OpenGL graphics.
Here's how we get the EGL Context: pinephone-mir/egl.c
/// Wayland EGL Interfaces for OpenGL Rendering
static EGLDisplay egl_display; // EGL Display
static EGLConfig egl_conf; // EGL Configuration
static EGLContext egl_context; // EGL Context
// Create the EGL Context for rendering OpenGL graphics
static void init_egl(void) {
// Attributes for our EGL Display
EGLint config_attribs[] = {
EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
EGL_RED_SIZE, 8,
EGL_GREEN_SIZE, 8,
EGL_BLUE_SIZE, 8,
EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
EGL_NONE
};
static const EGLint context_attribs[] = {
EGL_CONTEXT_CLIENT_VERSION, 2,
EGL_NONE
};
// Get the EGL Display
egl_display = eglGetDisplay((EGLNativeDisplayType) display);
assert(egl_display != EGL_NO_DISPLAY); // Failed to get EGL Display
// Init the EGL Display
EGLint major, minor;
EGLBoolean egl_init = eglInitialize(egl_display, &major, &minor);
assert(egl_init); // Failed to init EGL Display
// Get the EGL Configurations
EGLint count, n;
eglGetConfigs(egl_display, NULL, 0, &count);
EGLConfig *configs = calloc(count, sizeof *configs);
eglChooseConfig(egl_display, config_attribs,
configs, count, &n);
// Choose the first EGL Configuration
for (int i = 0; i < n; i++) {
EGLint size;
eglGetConfigAttrib(egl_display, configs[i], EGL_BUFFER_SIZE, &size);
eglGetConfigAttrib(egl_display, configs[i], EGL_RED_SIZE, &size);
egl_conf = configs[i];
break;
}
assert(egl_conf != NULL); // Failed to get EGL Configuration
// Create the EGL Context based on the EGL Display and Configuration
egl_context = eglCreateContext(egl_display, egl_conf,
EGL_NO_CONTEXT, context_attribs);
assert(egl_context != NULL); // Failed to create EGL Context
}
The above code in init_egl() creates the EGL Context.
We call init_egl() in our Main Function like so: pinephone-mir/egl.c
/// Wayland Interfaces
static struct wl_surface *surface; // Wayland Surface
static struct wl_shell_surface *shell_surface; // Wayland Shell Surface
/// Connect to Wayland Compositor and render OpenGL graphics
int main(int argc, char **argv) {
// Get interfaces for Wayland Compositor and Wayland Shell
get_server_references();
assert(display != NULL); // Failed to get Wayland Display
assert(compositor != NULL); // Failed to get Wayland Compositor
assert(shell != NULL); // Failed to get Wayland Shell
// Create a Wayland Surface for rendering our app
surface = wl_compositor_create_surface(compositor);
assert(surface != NULL); // Failed to create Wayland Surface
// Get the Wayland Shell Surface for rendering our app window
shell_surface = wl_shell_get_shell_surface(shell, surface);
assert(shell_surface != NULL);
// Set the Shell Surface as top level
wl_shell_surface_set_toplevel(shell_surface);
// Create the Wayland Region for rendering OpenGL graphics
create_opaque_region();
// Create the EGL Context for rendering OpenGL graphics
init_egl();
// Create the EGL Window and render OpenGL graphics
create_window();
// Handle all Wayland Events in the Event Loop
while (wl_display_dispatch(display) != -1) {}
// Disconnect from the Wayland Display
wl_display_disconnect(display);
return 0;
}
In all Wayland apps, the main() function follows the same steps...
Fetch the Wayland Compositor and Wayland Shell from the Wayland Registry...
// Get interfaces for Wayland Compositor and Wayland Shell
get_server_references();
We'll talk about get_server_references() and the Wayland Registry in a while.
Every Wayland App needs a Wayland Surface (screen buffer) for displaying the app...
// Create a Wayland Surface for rendering our app
surface = wl_compositor_create_surface(compositor);
Create a Wayland Shell Surface (app window) for rendering our app...
// Get the Wayland Shell Surface for rendering our app window
shell_surface = wl_shell_get_shell_surface(shell, surface);
Set the Shell Surface as the Top Level window for our app...
// Set the Shell Surface as top level
wl_shell_surface_set_toplevel(shell_surface);
This part is specific to OpenGL apps...
Earlier we have seen create_opaque_region(), init_egl() and create_window(). We call them to create the Wayland Region, EGL Context and EGL Window, and to render the OpenGL graphics.
// Create the Wayland Region for rendering OpenGL graphics
create_opaque_region();
// Create the EGL Context for rendering OpenGL graphics
init_egl();
// Create the EGL Window and render OpenGL graphics
create_window();
Every Wayland App needs an Event Loop for handling Wayland Events...
// Handle all Wayland Events in the Event Loop
while (wl_display_dispatch(display) != -1) {}
When our app terminates, we disconnect the Wayland Display...
// Disconnect from the Wayland Display
wl_display_disconnect(display);
Now let's build and test the app on our Linux development machine. (We'll run it on PinePhone later)
Now that we have created a simple Wayland app that renders OpenGL graphics... Let's build it!
Building a Wayland app is refreshingly simple (if you're used to GDK, Qt and SDL).
Here's how we build the Wayland app in egl.c on a Linux machine (that has Wayland, MESA EGL and OpenGL ES2 libraries installed)...
# Build the Wayland EGL app
gcc \
-g \
-o egl \
egl.c \
-Wl,-Map=egl.map \
-L/usr/lib/aarch64-linux-gnu/mesa-egl \
-lwayland-client \
-lwayland-server \
-lwayland-egl \
-lEGL \
-lGLESv2
This produces the executable app egl. Run the egl app on our Linux machine like so...
# Install Weston Wayland Compositor...
# For Arch Linux and Manjaro:
sudo pacman -S weston
# For Other Distros:
# Check https://github.com/wayland-project/weston
# Start the Weston Wayland Compositor on our computer with the PinePhone screen dimensions
weston --width=720 --height=1398 &
# Run the Wayland EGL app
./egl
This uses the Weston Compositor, the reference implementation of the Wayland Compositor that runs on X11.
We'll see this Inception-like window within a window...
We learn in a while how to build and run the app on PinePhone.
Earlier we used the Wayland Compositor and the Wayland Shell in our app...
Wayland Compositor (compositor): Manages the screen buffer used by apps
Wayland Shell (shell): Manages the app windows
Here's how we fetch the two interfaces from Wayland: pinephone-mir/egl.c
/// Wayland Interfaces
static struct wl_display *display; // Wayland Display
static struct wl_compositor *compositor; // Wayland Compositor
static struct wl_shell *shell; // Wayland Shell
/// Connect to Wayland Service and fetch the interfaces for Wayland Compositor and Wayland Shell
static void get_server_references(void) {
// Connect to the Wayland Service
display = wl_display_connect(NULL);
if (display == NULL) {
fprintf(stderr, "Failed to connect to display\n");
exit(1);
}
// Get the Wayland Registry
struct wl_registry *registry = wl_display_get_registry(display);
assert(registry != NULL); // Failed to get Wayland Registry
// Add Registry Callbacks to handle interfaces returned by Wayland Service
wl_registry_add_listener(registry, ®istry_listener, NULL);
// Wait for Registry Callbacks to fetch Wayland Interfaces
wl_display_dispatch(display);
wl_display_roundtrip(display);
// We should have received interfaces for Wayland Compositor and Wayland Shell
assert(compositor != NULL); // Failed to get Wayland Compositor
assert(shell != NULL); // Failed to get Wayland Shell
}
What happens inside get_server_references()?
The Wayland Compositor runs as a Linux Service that listens on a Linux Socket File: /run/user/32011/wayland-0 for PinePhone on Ubuntu Touch.
We connect to the Wayland Service like so...
// Connect to the Wayland Service
display = wl_display_connect(NULL);
Remember that all functions named wl_... come from the Wayland Library.
To work with the Wayland Service, we fetch the Interfaces for the Wayland Compositor and Wayland Shell.
Wayland Interfaces are defined in the Wayland Registry...
// Get the Wayland Registry
struct wl_registry *registry = wl_display_get_registry(display);
To fetch the Compositor and Shell from the Wayland Registry, we add a Registry Listener (more about this later)...
// Add Registry Callbacks to handle interfaces returned by Wayland Service
wl_registry_add_listener(registry, ®istry_listener, NULL);
Now we dispatch the Registry Listener request to the Wayland Service. (Remember that the Wayland Service operates on Linux Socket Messages)
// Wait for Registry Callbacks to fetch Wayland Interfaces
wl_display_dispatch(display);
wl_display_roundtrip(display);
And we'll get the compositor and shell objects populated from the Wayland Registry!
If you're curious, the Registry Listener works like this: pinephone-mir/egl.c
/// Callbacks for interfaces returned by Wayland Service
static const struct wl_registry_listener registry_listener = {
global_registry_handler,
global_registry_remover
};
/// Callback for interfaces returned by Wayland Service
static void global_registry_handler(void *data, struct wl_registry *registry, uint32_t id,
const char *interface, uint32_t version) {
printf("Got interface %s id %d\n", interface, id);
if (strcmp(interface, "wl_compositor") == 0) {
// Bind to Wayland Compositor Interface
compositor = wl_registry_bind(registry, id,
&wl_compositor_interface, // Interface Type
1); // Interface Version
} else if (strcmp(interface, "wl_shell") == 0){
// Bind to Wayland Shell Interface
shell = wl_registry_bind(registry, id,
&wl_shell_interface, // Interface Type
1); // Interface Version
}
}
global_registry_handler() is the Callback Function that will be triggered for every interface in the Wayland Registry.
The Wayland Service for Ubuntu Touch unity-system-compositor returns a whole bunch of interesting Wayland Interfaces (like qt_windowmanager).
But today we'll bind to the Compositor Interface named wl_compositor and Shell Interface named wl_shell.
And that's how we render a yellow rectangle with Wayland and OpenGL!
Let's move on to something more interesting: Rendering a simple bitmap texture...
Rendering a simple bitmap texture with Wayland and OpenGL on PinePhone
The four boxes we see above are rendered from a magnified 2-pixel by 2-pixel bitmap: pinephone-mir/texture.c
// 2x2 Image, 3 bytes per pixel (R, G, B)
GLubyte pixels[4 * 3] = {
255, 0, 0, // Red
0, 255, 0, // Green
0, 0, 255, // Blue
255, 255, 0 // Yellow
};
We render the bitmap by creating an OpenGL Texture: pinephone-mir/texture.c
// Create a simple 2x2 texture image with four different colors
GLuint CreateSimpleTexture2D() {
// Texture object handle
GLuint textureId;
// 2x2 Image, 3 bytes per pixel (R, G, B)
GLubyte pixels[4 * 3] = {
255, 0, 0, // Red
0, 255, 0, // Green
0, 0, 255, // Blue
255, 255, 0 // Yellow
};
// Use tightly packed data
glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
// Generate a texture object
glGenTextures(1, &textureId);
// Bind the texture object
glBindTexture(GL_TEXTURE_2D, textureId);
// Load the texture
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 2, 2, 0, GL_RGB, GL_UNSIGNED_BYTE, pixels);
// Set the filtering mode
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
return textureId;
}
(Not the most efficient way to render a bitmap... But let's try this and test drive PinePhone's GPU!)
This is the usual way we create an OpenGL Texture, as explained in "OpenGL® ES 3.0 Programming Guide".
Here comes the tricky part... Before rendering the OpenGL Texture, we need to program the GPU Shaders on PinePhone with a C-like language: pinephone-mir/texture.c
// Initialize the shader and program object
int Init(ESContext *esContext) {
esContext->userData = malloc(sizeof(UserData));
UserData *userData = esContext->userData;
GLbyte vShaderStr[] =
"attribute vec4 a_position; \n"
"attribute vec2 a_texCoord; \n"
"varying vec2 v_texCoord; \n"
"void main() \n"
"{ \n"
" gl_Position = a_position; \n"
" v_texCoord = a_texCoord; \n"
"} \n";
GLbyte fShaderStr[] =
"precision mediump float; \n"
"varying vec2 v_texCoord; \n"
"uniform sampler2D s_texture; \n"
"void main() \n"
"{ \n"
" gl_FragColor = texture2D( s_texture, v_texCoord );\n"
"} \n";
// Load the shaders and get a linked program object
userData->programObject = esLoadProgram(vShaderStr, fShaderStr);
...
(Yep a C program within a C program... Inception!)
esLoadProgram() is defined in pinephone-mir/shader.c
We're now talking to PinePhone's GPU, which is so low-level that it understand only Triangles, not Rectangles.
Hence to render the OpenGL Texture, we map the Rectangular Texture onto two Triangles and render them: pinephone-mir/texture.c
// Draw a triangle using the shader pair created in Init()
void Draw(ESContext *esContext) {
GLfloat vVertices[] = {
-0.5f, 0.5f, 0.0f, // Position 0
0.0f, 0.0f, // TexCoord 0
-0.5f, -0.5f, 0.0f, // Position 1
0.0f, 1.0f, // TexCoord 1
0.5f, -0.5f, 0.0f, // Position 2
1.0f, 1.0f, // TexCoord 2
0.5f, 0.5f, 0.0f, // Position 3
1.0f, 0.0f // TexCoord 3
};
GLushort indices[] = {
0, 1, 2, // First Triangle
0, 2, 3 // Second Triangle
};
...
// Draw the 6 vertices as 2 triangles
glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_SHORT, indices);
}
(Yes the math is starting to hurt... But that's the end of it!)
Finally we connect the above code to render the four colour boxes on PinePhone, thanks to Wayland and OpenGL: pinephone-mir/egl2.c
/// Render the OpenGL ES2 display
static void render_display() {
// Create the texture context
static ESContext esContext;
esInitContext ( &esContext );
esContext.width = WIDTH;
esContext.height = HEIGHT;
// Draw the texture
Init(&esContext);
Draw(&esContext);
// Render now
glFlush();
}
And that's our Wayland App that renders a simple OpenGL Bitmap Texture!
The OpenGL Texture code in this article was adapted from "OpenGL® ES 2.0 Programming Guide"
Let's head on towards greatness and something really useful: Graphical User Interfaces...
Button rendered with LVGL and Wayland on PinePhone
Now that we can render bitmaps on PinePhone, let's think...
How would we render a simple Graphical User Interface (GUI) on PinePhone, like the button above?
Why don't we use a simple GUI Toolkit like LVGL? (Formerly LittleVGL)
Here's how we call the LVGL library to render that button: lvgl-wayland/wayland/lvgl.c
#include "../lvgl.h"
/// Render a Button Widget and a Label Widget
static void render_widgets(void) {
lv_obj_t * btn = lv_btn_create(lv_scr_act(), NULL); // Add a button the current screen
lv_obj_set_pos(btn, 10, 10); // Set its position
lv_obj_set_size(btn, 120, 50); // Set its size
lv_obj_t * label = lv_label_create(btn, NULL); // Add a label to the button
lv_label_set_text(label, "Button"); // Set the labels text
}
Easy peasy!
LVGL is a simple C toolkit designed for Embedded Devices, so it needs very little memory and processing power. LVGL is self-contained... Fonts and icons are bundled into the LVGL library.
It's used on PineTime Smart Watch to render watch faces.
LVGL doesn't run on Wayland yet... But we'll fix that!
Remember how we rendered a simple 2-pixel by 2-pixel bitmap by creating an OpenGL Texture with CreateSimpleTexture2D()?
Let's extend that bitmap to cover the entire PinePhone screen: 720 pixels by 1398 pixels.
And we create the OpenGL Texture for the entire PinePhone screen like so: lvgl-wayland/wayland/texture.c
/// PinePhone Screen Resolution, defined in lv_conf.h
#define LV_HOR_RES_MAX (720)
#define LV_VER_RES_MAX (1398)
#define LV_SCALE_RES 1
/// Screen buffer
#define BYTES_PER_PIXEL 3
GLubyte pixels[LV_HOR_RES_MAX * LV_VER_RES_MAX * BYTES_PER_PIXEL];
/// Create an OpenGL Texture for the screen buffer
GLuint CreateTexture(void) {
GLuint texId;
glGenTextures ( 1, &texId );
glBindTexture ( GL_TEXTURE_2D, texId );
glTexImage2D (
GL_TEXTURE_2D,
0, // Level
GL_RGB,
LV_HOR_RES_MAX, // Width
LV_VER_RES_MAX, // Height
0, // Format
GL_RGB,
GL_UNSIGNED_BYTE,
pixels
);
glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
glTexParameteri ( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
return texId;
}
pixels is the screen buffer that will contain the pixels for our rendered UI controls, like our button.
We'll tell LVGL to render into pixels like so: lvgl-wayland/wayland/texture.c
/// Set the colour of a pixel in the screen buffer
void put_px(uint16_t x, uint16_t y, uint8_t r, uint8_t g, uint8_t b, uint8_t a) {
assert(x >= 0); assert(x < LV_HOR_RES_MAX);
assert(y >= 0); assert(y < LV_VER_RES_MAX);
int i = (y * LV_HOR_RES_MAX * BYTES_PER_PIXEL) + (x * BYTES_PER_PIXEL);
pixels[i++] = r; // Red
pixels[i++] = g; // Green
pixels[i++] = b; // Blue
}
(Simplistic, not efficient though)
We'll render the OpenGL Texture the same way as before: lvgl-wayland/wayland/lvgl.c
/// Render the OpenGL ES2 display
static void render_display() {
// This part is new...
// Init the LVGL display
lv_init();
lv_port_disp_init();
// Create the LVGL widgets: Button and label
render_widgets();
// Render the LVGL widgets
lv_task_handler();
// This part is the same as before...
// Create the texture context
static ESContext esContext;
esInitContext ( &esContext );
esContext.width = WIDTH;
esContext.height = HEIGHT;
// Draw the texture
Init(&esContext);
Draw(&esContext);
// Render now
glFlush();
}
But now we have injected the calls to the LVGL library...
lv_init(): Initialise the LVGL library
lv_port_disp_init(): Initialise our display
render_widgets(): Calls the LVGL library to create two UI controls: a Button and a Label
lv_task_handler(): Let LVGL render the UI controls into our screen buffer
Now let's tweak the LVGL library to render UI controls into our screen buffer pixels
Porting LVGL to Wayland and Ubuntu Touch is straightforward.
According to the LVGL Porting Doc, we need to code a Flush Callback Function disp_flush() that will be called by LVGL to render UI controls to the screen buffer.
Here's our implementation for Wayland: lvgl-wayland/wayland/lv_port_disp.c
// Flush the content of the internal buffer to the specific area on the display
// You can use DMA or any hardware acceleration to do this operation in the background but
// 'lv_disp_flush_ready()' has to be called when finished.
static void disp_flush(lv_disp_drv_t * disp_drv, const lv_area_t * area, lv_color_t * color_p) {
// The most simple case (but also the slowest) to put all pixels to the screen one-by-one
for(int32_t y = area->y1; y <= area->y2; y++) {
for(int32_t x = area->x1; x <= area->x2; x++) {
// Put a pixel to the screen buffer
put_px(x, y,
color_p->ch.red,
color_p->ch.green,
color_p->ch.blue,
0xff);
color_p++;
}
}
// Inform the graphics library that we are ready with the flushing
lv_disp_flush_ready(disp_drv);
}
We've seen earlier that put_px() draws pixels in the simplest way possible. Eventually we should use PinePhone's GPU for rendering LVGL controls, by implementing the LVGL GPU Callbacks.
Light and Dark Themes are provided by LVGL. To select the default theme just edit lvgl-wayland/lv_conf.h
Here's Dark Theme...
// For Dark Theme...
#define LV_THEME_DEFAULT_FLAG LV_THEME_MATERIAL_FLAG_DARK
And Light Theme...
// For Light Theme...
#define LV_THEME_DEFAULT_FLAG LV_THEME_MATERIAL_FLAG_LIGHT
The screens above were rendered by updating one line in lvgl-wayland/wayland/lvgl.c...
/// Render the OpenGL ES2 display
static void render_display() {
// Init the LVGL display
lv_init();
lv_port_disp_init();
// Create the LVGL widgets
lv_demo_widgets(); // Previously render_widgets()
lv_demo_widgets() comes from lvgl-wayland/demo/lv_demo_widgets.c
What about Touch Input in LVGL for Ubuntu Touch?
We haven't handled Touch Input yet... Lemme know if you're keen to help!
Do we really have to code LVGL Apps for PinePhone in C?
Rust is supported too!
We may write LVGL Apps for PinePhone in Rust by calling the lvgl-rs Rust Wrapper for LVGL by Rafael Carício.
(Fun Fact: lvgl-rs was originally created for PineTime Smart Watch... Today it's used by Rust on PlayStation Portable too!)
How small is LVGL on PinePhone with Ubuntu Touch?
1.5 MB for the Light / Dark Theme LVGL Demo App above.
Not that big, considering that the font, icons and debugging symbols are bundled inside.
How does LVGL compare with Qt, GTK and SDL on PinePhone with Ubuntu Touch?
Qt is the only officially supported App Toolkit on Ubuntu Touch.
GTK and SDL are supposed to work on Wayland... But I couldn't get them to work on Ubuntu Touch.
(Probably because legacy X11 compatibility is missing from Ubuntu Touch, i.e. XWayland)
I applaud the maintainers of X11, Qt, GTK and SDL because every new release needs to support so many legacy features. Kudos!
But what if we could start from scratch, drop the legacy stuff, and build a simpler UI toolkit for Wayland?
LVGL is the experiment that we're undertaking today!
Follow these steps to build LVGL on PinePhone over SSH.
(If we haven't enabled SSH on PinePhone, check the "Configure SSH on PinePhone" instructions below)
Connect to PinePhone over SSH and enter these commands...
# Make system folders writeable before installing any packages
sudo mount -o remount,rw /
# Install dev tools and GLES2 library
sudo apt install gcc gdb git make libgles2-mesa-dev
# Download the source code
cd ~
git clone https://github.com/lupyuen/lvgl-wayland
cd lvgl-wayland
# Build the app
make
This creates the executable ~/lvgl-wayland/wayland/lvgl
Can we just run lvgl from the Terminal Command Line?
Nope! Because Wayland and Ubuntu Touch are super-secure, thanks to AppArmor.
But there's a way to trick AppArmor into allowing lvgl to be launched (since we are Superuser).
Read on to learn how...
Fighting AppArmor Security... Permission Denied!
For rapid testing (and to work around AppArmor), we shall replace the File Manager app by our lvgl app because File Manager has no AppArmor restrictions (Unconfined).
(More about AppArmor in a while)
Connect to PinePhone over SSH and enter these commands...
# Make system folders writeable and go to File Manager Click Package folder
sudo mount -o remount,rw /
cd /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
# Back up the desktop file. Restore this desktop file to go back to the normal File Manager
sudo cp com.ubuntu.filemanager.desktop com.ubuntu.filemanager.desktop.old
# Edit the desktop file
sudo nano com.ubuntu.filemanager.desktop
We're now altering the behaviour of File Manager, by tampering with the Click Package settings for File Manager.
(Why are we tampering with a Click Package? We'll learn in a while)
Change the Exec line from...
Exec=filemanager
To...
Exec=./run.sh
Save and exit nano
We have modded the File Manager icon so that it now launches run.sh instead of the usual filemanager executable.
(It's like switching the executable for a Windows Explorer Shortcut)
We'll be installing run.sh later with a script: lvgl.sh
In the meantime, check that run.sh (located at ~/lvgl-wayland/wayland) contains the following...
# Log Wayland messages
export WAYLAND_DEBUG=1
# Run lvgl app
./lvgl
If we see this...
# Debug lvgl app
gdb \
-ex="r" \
-ex="bt" \
-ex="frame" \
--args ./lvgl
It means that the lvgl app will be started with the gdb debugger.
If it crashes with a bad C pointer, the gdb debugger will show a helpful stack trace.
And this...
# Run lvgl app with strace
./strace \
-s 1024 \
./lvgl
Is for tracing the lvgl app with strace. It shows everything done by the app.
Check out this strace log for the File Manager on Ubuntu Touch
Finally we're ready to run our lvgl app!
Connect to PinePhone over SSH and enter these commands...
cd ~/lvgl-wayland
./wayland/lvgl.sh
The script lvgl.sh copies run.sh from ~/lvgl-wayland/wayland to the Click Package Folder for File Manager...
/usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
In a few seconds we'll see the message...
*** Tap on File Manager icon on PinePhone
Go ahead and tap the File Manager icon on PinePhone.
Our LVGL App shall run instead of the File Manager.
In the SSH console, press Ctrl-C to stop the log display.
The log file for the app is located at...
/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log
The log for the Wayland Compositor unity-system-compositor may be useful for troubleshooting...
/home/phablet/.cache/upstart/unity8.log
Copy the log files to our machine like this...
scp -i ~/.ssh/pinephone_rsa phablet@192.168.1.160:/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log .
scp -i ~/.ssh/pinephone_rsa phablet@192.168.1.160:/home/phablet/.cache/upstart/unity8.log .
To understand Wayland, AppArmor and Ubuntu Touch Security, let's look inside the script lvgl.sh and discover how it launches our lvgl app...
Our lvgl app doesn't have a close button, so let's terminate the app if it's already running...
# Kill the app if it's already running
pkill lvgl
In Ubuntu Touch, User Directories (like our Home Directory) are writeable by default.
System Directories (like /usr/share) are mounted with read-only access, to prevent tampering of system files. (Think malware)
Since we're Superuser, we may remount System Directories with read-write access...
# Make system folders writeable
sudo mount -o remount,rw /
Why do we need read-write access?
Because we'll be copying our app lvgl and the script run.sh to a System Directory...
# Copy app to File Manager folder
cd wayland
sudo cp lvgl /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
# Copy run script to File Manager folder
sudo cp run.sh /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager
What's this folder /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager?
This is the Click Package folder for File Manager.
Ubuntu Touch Apps (like File Manager) are packaged as Click Packages for installation on our phones.
When the app is installed, Ubuntu Touch extracts the Click Package into a folder under /usr/share/click.
Inside the Click Package folder we'll find the executables, libraries and data files that are needed for running the app.
The folder also contains a .desktop file. (Earlier we've seen com.ubuntu.filemanager.desktop for File Manager) This file tells Ubuntu Touch how to launch the app.
Does the app run as our user account phablet?
Nope. For security, Ubuntu Touch Apps run under an account with restricted privileges: clickpkg
This account has no access to our phablet files. That's why we copy the lvgl app and run.sh script to the Click Package folder, which is accessible by clickpkg
We set the ownership of lvgl and run.sh to clickpkg so that it can execute the files...
# Set ownership on the app and the run script
sudo chown clickpkg:clickpkg /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager/lvgl
sudo chown clickpkg:clickpkg /usr/share/click/preinstalled/.click/users/@all/com.ubuntu.filemanager/run.sh
Our lvgl app and run.sh script have been staged in the Click Package folder.
We ask the human to tap the File Manager icon...
# Start the File Manager
echo "*** Tap on File Manager icon on PinePhone"
Ubuntu Touch launches our lvgl app. As our app runs, it logs debugging messages to Standard Output and Standard Error.
The messages are captured in this log file...
# Monitor the log file
echo >/home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log
tail -f /home/phablet/.cache/upstart/application-click-com.ubuntu.filemanager_filemanager_0.7.5.log
Why can't we run our lvgl app from the Terminal Command Line?
Because Ubuntu Touch's Wayland Service stops unauthorized processes from grabbing the Compositor...
We see this in the Wayland Compositor log: /home/phablet/.cache/upstart/unity8.log
ApplicationManager REJECTED connection from app with pid 6710
as it was not launched by upstart, and no desktop_file_hint is specified
That's why we need to inject lvgl into File Manager... So that Wayland thinks that the File Manager is grabbing the Compositor.
Why did we choose the File Manager app instead of another app like Camera?
Because File Manager has Unconfined AppArmor Permissions... It can do anything!
(But still restricted by the clickpkg user permissions)
Look at the AppArmor Policy for the File Manager App: filemanager.apparmor
{
"policy_version": 16.04,
"template": "unconfined",
"policy_groups": []
}
Compare this with the AppArmor Policy for the Camera App: camera.apparmor
{
"policy_groups": [
"picture_files",
"video_files",
"camera",
"audio",
"video",
"usermetrics",
"content_exchange",
"content_exchange_source",
"location"
],
"policy_version": 16.04,
"read_path": [
"@{PROC}/*/mounts",
"/dev/disk/by-label/"
]
}
The AppArmor Policy says that the Camera App may only access selected features (like recording audio and video). And it's only allowed to read specific paths (like /dev/disk/by-label).
strace won't work with the AppArmor Policy for Camera App.
So for tracing our app with strace, we "borrow" the Unconfined AppArmor Policy for File Manager.
To troubleshoot problems with AppArmor, check the system log in /var/log/syslog
While attempting to port the PineTime Companion App to PinePhone with GTK (and failing miserably), I had these thoughts...
AppArmor is good, because iOS and Android have similar apps security
Read-only file system is good (system files are read-only by default, user files are read-write). Helps to prevent security holes. (Even PineTime has a read-only Flash ROM)
Why is Qt supported on Ubuntu Touch and not GTK? Because building a Linux mobile app requires mobile-friendly widgets.
I think Qt has more mobile-friendly widgets, even through the internal plumbing is way too complicated.
When I get GTK running on Ubuntu Touch, I will face the same problem with widgets. And I have to make GTK widgets look and feel consistent with Qt / Ubuntu Touch widgets.
That's why I decided to move away from GTK and focus on a simpler widget framework with LVGL.
Older kernel base in Ubuntu Touch... I don't do kernel hacking much so it doesn't matter to me.
I think for mobiles we only need to support a few common chipsets, so an older kernel is probably fine.
That explains why Raspberry Pi 4 isn't supported by Ubuntu Touch... The hardware is just too new.
The issues I'm struggling with now... Wayland, GTK3, ... are actually really old stuff. Updating the kernel won't help.
Ubuntu Touch is pure Wayland, none of the legacy X11 stuff. Xwayland is not even there (unless you use the Libertine containers ugh).
The pure Wayland environment causes GTK to break, because GTK assumes some minimal X11 support (i.e. Xwayland).
It's better to start from scratch with a toolkit that's not based on X11, like LVGL.
So Ubuntu Touch is not really that bad for PinePhone... It's just painful for building non-Qt apps. 🙂
After posting my thoughts, the UBports, GNOME and Xfce Community responded with encouraging and insightful notes...
Ubuntu Touch supports Wayland only on non-Android devices like PinePhone
Ubuntu Touch uses older kernels on Android devices, only because that's the only way to run on those devices. PinePhone uses a newer kernel.
UBports is fixing that with Halium9, a Hardware Abstraction Layer based on parts of AOSP 9 and other hardware enablement components
Read about Unity8 / Mir / Lomiri's complicated history
GNOME Shell developers are working into making X11 completely optional
GTK can already be built without X11 backend
Proprietary Nvidia graphics drivers didn't have Wayland support out of the box as they wanted to use EGLStreams rather than GBM. So Nvidia did the work to get GNOME and KDE to support EGLStreams.
Only GNOME and KDE have Wayland support and the support isn't feature complete compared to X11, which is why most distros with GNOME and KDE editions still do not default to Wayland
I'm sorry the port of LVGL on Wayland to PinePhone is incomplete, and I am now working on other PineTime Smart Watch projects.
The Touch Input needs to be ported for LVGL like this...
The LVGL Input Driver needs to be implemented with Wayland (without X) like this...
Also please take care of the LVGL Tick Interface. We need to call lv_tick_inc() periodically or our LVGL Display Driver won't flush when calling lv_task_handler()...
I noticed this behaviour so I called lv_tick_inc(100) before lv_task_handler()...
That's the code I used for my latest project, a WebAssembly LVGL Simulator for PineTime Smart Watch...
lvgl-wasm: PineTime Watch Face Simulator with LVGL ported to WebAssembly
Preview PineTime Watch Faces in your Web Browser with WebAssembly
Wayland feels like New Underwear... And it needs a New App Toolkit like LVGL to make us comfortable.
If you're keen to help make LVGL the Newer, Simpler App Toolkit on Wayland, Ubuntu Touch and PinePhone, please lemme know! 😀
Got a question, comment or suggestion? Create an Issue or submit a Pull Request here...
pinetime-rust-mynewt/rust/ app/src/wayland.md
First Thing to do when we get our new PinePhone: Open the PinePhone Back Cover and Remove the Battery Insulation Sticker!
(Can't believe I missed that!)
Second Thing: Protect the SSH Service on PinePhone with SSH Keys. And start the SSH Service only when necessary.
Here's how...
On our Computer (not PinePhone), open a Command Prompt. Enter this (and fill in our email)...
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
When prompted...
Enter a file in which to save the key
Press Enter. This stores the new SSH Key in the .ssh folder in our Home Directory.
When prompted...
Enter a file in which to save the key
Enter...
pinephone_rsa
We'll create an SSH Key Pair named pinephone_rsa (Private Key) and pinephone_rsa.pub (Public Key)
When prompted...
Enter passphrase
Press Enter. We won't need a passphrase unless our PinePhone needs to be super-secure.
This creates an SSH Key Pair in the .ssh folder in our Home Directory...
pinephone_rsa contains the Private Key. Never give the Private Key to others!
pinephone_rsa.pub contains the Public Key. We'll copy this to PinePhone now.
Copy pinephone_rsa.pub from the .ssh folder in our Home Directory to a MicroSD Card.
Insert the MicroSD Card into PinePhone. Copy pinephone_rsa.pub to our Home Directory on PinePhone.
(Check the section "Copy Files from MicroSD Card on PinePhone" below)
Tap the Terminal icon on PinePhone. Enter...
# Go to home directory
cd
# If .ssh folder doesn't exist, create it
mkdir .ssh
chmod 700 .ssh
# Set public key as the authorized key
cp pinephone_rsa.pub .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
# Show the SSH files
ls -la ~/.ssh
We should see this...
drwx------ 2 phablet phablet 4096 Jul 7 20:06 .
drwxr-xr-x 28 phablet phablet 4096 Jul 24 11:38 ..
-rw------- 1 phablet phablet 743 Jul 7 20:08 authorized_keys
Check that the permissions (rw) and owner (phablet) are correct.
To start the SSH Service on PinePhone, open the Terminal app.
Create a file named a...
nano a
Type this script into the a file...
#!/bin/sh
# Script to start SSH service and show IP address
# Start SSH service
sudo service ssh start
# Show IP address
ifconfig | \
grep -v "127.0.0.1" | \
grep "inet addr:"
# Ping repeatedly to keep WiFi alive
ping google.com
Save the file and exit nano.
(Or download the file from lvgl-wayland/a and copy via a MicroSD Card. Check the next section for instructions.)
When we're ready do coding on PinePhone, enter this at the Terminal command line...
. a
(There's a space between "." and "a")
The script starts the SSH Service and displays the IP address for PinePhone...
From our Computer, we'll connect to PinePhone at the IP adddress indicated by inet addr, say 192.168.1.160...
ssh -i ~/.ssh/pinephone_rsa phablet@192.168.1.160
And that's how we access PinePhone via SSH!
When we press PinePhone's power button to switch off PinePhone, we'll see ths amusing message from olden times...
If typing on a touch keyboard is not your thing, try copying the files from a MicroSD card...
How we insert a MicroSD Card into PinePhone at night
It's useful to transfer files to PinePhone via MicroSD Card, like SSH Keys and the SSH Script a above.
(Sadly PinePhone on Ubuntu Touch doesn't allow receiving files over Bluetooth)
The MicroSD card on PinePhone doesn't appear in the File Manager unless we mount it.
Tap the Terminal icon on PinePhone and enter the following...
ls -l /dev/disk/by-label
We should see something like this...
lrwxrwxrwx 1 root root 15 Jul 23 22:24 BOOT_MNJRO -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 15 Jul 23 22:24 cache -> ../../mmcblk2p8
lrwxrwxrwx 1 root root 15 Jul 23 22:24 ROOT_MNJRO -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 16 Jul 23 22:24 userdata -> ../../mmcblk2p10
These are the Partition Labels on our MicroSD Card.
Let's say we wish to mount the MicroSD Card partition ROOT_MNJRO, which links to /dev/mmcblk0p2...
mkdir /tmp/sdcard
sudo mount /dev/mmcblk0p2 /tmp/sdcard
ls -l /tmp/sdcard
(If we don't see our Patition Label, try mounting the numbered partitions anyway: /dev/mmcblk0p1, p2, p3, ...)
We should see the contents of our MicroSD Card.
The MicroSD Card will now appear in File Manager as /tmp/sdcard, ready for us to copy the files.
Or just copy files from the Command Line like so...
cp /tmp/sdcard/a ~
When we're done, unmount our MicroSD Card...
sudo umount /tmp/sdcard
Our LVGL App works on Linux machines like Pinebook Pro...
# Download the source code
git clone https://github.com/lupyuen/lvgl-wayland
cd lvgl-wayland
# Build the lvgl executable
make
# Install Weston Wayland Compositor...
# For Arch Linux and Manjaro:
sudo pacman -S weston
# For Other Distros:
# Check https://github.com/wayland-project/weston
# Start the Weston Wayland Compositor with the PinePhone screen dimensions
weston --width=720 --height=1398 &
# Run the lvgl executable
./wayland/lvgl