I've been investigating how keyboards, mice, etc work in Linux. In this blog post we'll see how input events work, using libinput-ocaml, and then use that to write a little game.
Using libinput-ocaml to make a game
Table of Contents
Device files
My Wayland compositor (Sway) makes use of the keyboard and mouse, so let's see what devices it has open (note: all shell commands are using fish syntax):
$ lsof -p (pgrep -x sway) -a +D /dev
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sway 2031 tal mem CHR 226,128 892 /dev/dri/renderD128
sway 2031 tal 0u CHR 4,1 0t0 20 /dev/tty1
...
sway 2031 tal 46u CHR 13,69 0t0 648 /dev/input/event5
sway 2031 tal 47u CHR 13,68 0t0 647 /dev/input/event4
sway 2031 tal 48u CHR 13,67 0t0 646 /dev/input/event3
sway 2031 tal 49u CHR 13,64 0t0 641 /dev/input/event0
sway 2031 tal 50u CHR 13,65 0t0 644 /dev/input/event1
sway 2031 tal 52r CHR 1,9 0t0 9 /dev/urandom
sway 2031 tal 54u CHR 13,66 0t0 645 /dev/input/event2
/dev/dri is for graphics cards, which I looked at earlier -
see Investigating Linux graphics for details.
/dev/tty1 is the Linux console where Sway is running.
/dev/input/event* are the input devices.
The names aren't very helpful, but I see I also have some symlinks e.g.
$ ls -l /dev/input/by-id/usb-Logitech_USB_Optical_Mouse-event-mouse
lrwxrwxrwx 1 root root 9 2026-02-28 07:46
/dev/input/by-id/usb-Logitech_USB_Optical_Mouse-event-mouse -> ../event2
So event2 is the mouse. Not all of them have symlinks though, and if I plug in a second identical mouse then I get a new event18 device but still only one symlink, so this doesn't seem very useful.
What creates these device files?
$ sudo opensnoop | grep /dev/input
[ I plug in a second USB mouse here ]
45754 (udev-worker) 21 0 /dev/input/mouse1
45757 (udev-worker) 25 0 /dev/input/event18
45757 (udev-worker) 25 0 /dev/input/event18
1236 systemd-logind 36 0 /dev/input/event18
The device files are created by systemd-udevd.
The mouse* devices seems to be from some older event system and we can ignore them.
When the Linux kernel detects a new device, it adds some information about it under
/sys/devices/ and broadcasts an event (in the NETLINK_KOBJECT_UEVENT netlink group).
Any process can subscribe to these events, and you can see them like this:
$ udevadm monitor --kernel
monitor will print the received events for:
KERNEL - the kernel uevent
[ I plug in a USB mouse here ]
KERNEL[56090.928942] add /devices/pci0000:00/0000:00:14.0/usb1/1-10 (usb)
KERNEL[56090.929735] change /devices/pci0000:00/0000:00:14.0/usb1/1-10 (usb)
KERNEL[56090.930056] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0 (usb)
KERNEL[56090.933228] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009 (hid)
KERNEL[56090.933351] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009/input/input22 (input)
KERNEL[56090.933401] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009/input/input22/mouse1 (input)
KERNEL[56090.933445] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009/input/input22/event18 (input)
KERNEL[56090.933500] add /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009/hidraw/hidraw5 (hidraw)
KERNEL[56090.933571] bind /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009 (hid)
KERNEL[56090.933642] bind /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0 (usb)
KERNEL[56090.933714] bind /devices/pci0000:00/0000:00:14.0/usb1/1-10 (usb)
There's lots of information here. For example, to see the new device's name:
$ cat /sys/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C077.0009/input/input22/event18/device/name
Logitech USB Optical Mouse
systemd-udevd gets these events and creates the device files in /dev/input/ according to its configuration. You can see what it will do like this (I've trimmed the output a lot):
$ udevadm test /dev/input/event18
event18: /nix/store/...systemd-258.3/lib/udev/rules.d/50-udev-default.rules:48
GROUP="input": Set group ID: 174
event18: /nix/store/...systemd-258.3/lib/udev/rules.d/60-persistent-input.rules:28
SYMLINK+="input/by-id/$env{ID_BUS}-$env{ID_SERIAL}-event-$env{.INPUT_CLASS}":
Added device node symlink "input/by-id/usb-Logitech_USB_Optical_Mouse-event-mouse".
event18: /nix/store/...systemd-258.3/lib/udev/rules.d/60-persistent-input.rules:38
SYMLINK+="input/by-path/$env{ID_PATH}-event-$env{.INPUT_CLASS}":
Added device node symlink "input/by-path/pci-0000:00:14.0-usb-0:10:1.0-event-mouse".
event18: /nix/store/...systemd-258.3/lib/udev/rules.d/60-persistent-input.rules:39
SYMLINK+="input/by-path/$env{ID_PATH_WITH_USB_REVISION}-event-$env{.INPUT_CLASS}":
Added device node symlink "input/by-path/pci-0000:00:14.0-usbv2-0:10:1.0-event-mouse".
...
Device name:
event18
Device node:
/dev/input/event18 (char 13:82)
Device node owner group:
input (gid=174)
Device node symlinks: (priority=0)
/dev/input/by-path/pci-0000:00:14.0-usb-0:10:1.0-event-mouse
/dev/input/by-path/pci-0000:00:14.0-usbv2-0:10:1.0-event-mouse
/dev/input/by-id/usb-Logitech_USB_Optical_Mouse-event-mouse
Properties:
.INPUT_CLASS=mouse
ACTION=add
...
Using devices directly
Let's see the event data coming from the new mouse:
$ hexdump /dev/input/event18
hexdump: /dev/input/event18: Permission denied
$ sudo hexdump /dev/input/event18
0000000 b9c5 69a6 0000 0000 d5ca 0008 0000 0000
...
$ sudo hexdump /dev/input/event18 \
-e '/8 "%d." /8 "%06d" /2 " ty=%x" /2 " code=%x" 1/4 " val=%2d\n"'
1772534246.362989 ty=2 code=0 val= 3
1772534246.362989 ty=2 code=1 val=-1
1772534246.362989 ty=0 code=0 val= 0
The evtest utility formats things better:
$ sudo evtest /dev/input/event18
Event: time 1772534337.699150, type 2 (EV_REL), code 0 (REL_X), value 6
Event: time 1772534337.699150, type 2 (EV_REL), code 1 (REL_Y), value -2
Event: time 1772534337.699150, -------------- SYN_REPORT ------------
Event: time 1772534337.707193, type 2 (EV_REL), code 0 (REL_X), value 4
Event: time 1772534337.707193, type 2 (EV_REL), code 1 (REL_Y), value -2
Event: time 1772534337.707193, -------------- SYN_REPORT ------------
So, when I move the mouse the kernel generates an event giving the relative motion for the X axis, then another for the Y axis, then a sync marker to finish the group. Seems simple enough.
Permissions
I had to use sudo above to be able to open the device.
But Sway manages to use it without help. How did it do that?
$ strace -y -p (pgrep -x sway) &| grep /dev/input
[ I plug in a USB mouse here ]
newfstatat(AT_FDCWD</home/tal>, "/dev/input/event18",
{st_mode=S_IFCHR|0660, st_rdev=makedev(0xd, 0x52), ...}, 0) = 0
recvmsg(6<socket:[14582]>, {
msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="...", iov_len=24}], msg_iovlen=1,
msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS,
cmsg_data=[101</dev/input/event18>]}],
msg_controllen=24, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 24
Sway didn't open event18 itself. Instead, it received the handle over a Unix domain socket (FD 6).
What is FD 6 connected to?
$ sudo lsof -a -U +E -p (pgrep -x sway) -d6
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dbus-daem 1217 messagebus 16u unix 0xffff8d960b22b800 0t0 9814 /run/dbus/system_bus_socket type=STREAM ->INO=14582 2031,sway,8u 2031,sway,6u (CONNECTED)
sway 2031 tal 6u unix 0xffff8d960b228800 0t0 14582 @ea2d5fdc608641b3/bus/sway/system type=STREAM ->INO=9814 1217,dbus-daem,16u (CONNECTED)
...
Sway is using FD 6 to communicate via DBus. Let's capture the messages and see what it's saying:
$ sudo busctl capture --system > plug-mouse.pcap
Monitoring bus message stream.
[ I plug in a USB mouse here ]
^C
Examining the capture with wireshark, I see that Sway sent a TakeDevice message
to the org.freedesktop.login1 service (provided by systemd-logind)
with the major and minor number of the device it wanted.
The privileged systemd-logind service returned a handle to the device
(you might have noticed this service opened the device in the opensnoop output earlier).
The documentation notes that:
systemd-logind automatically mutes the file descriptor if the session is inactive and resumes it once the session is activated again. This guarantees that a session can only access session devices if the session is active.
That makes sense, but it's weird how Linux seems to have three different permission systems here:
- For graphics output, all users (anyone in the
videogroup) can open the device, but only the first becomes the "master" and can control it. - For mice and keyboards, the devices are owned by a privileged service and you must request them over DBus. The service revokes access when you switch user.
- For joysticks, the permissions on the device allow the currently-logged-in user to open them directly, and access is not revoked when switching users.
libinput
Dealing with input from actual mice looks fairly easy, but things get very complicated when you add in touchpads, gesture recognition, etc. libinput knows how to handle a wide variety of devices and provides a uniform API for using them (it also has surprisingly good user documentation). I made some OCaml bindings to make it easier to play with it.
Getting a REPL
If you want to follow along, you'll need to install libinput-ocaml and an interactive REPL like utop. With Nix, you can set everything up like this:
git clone https://github.com/talex5/libinput-ocaml.git
cd libinput-ocaml
nix develop
dune utop
You should see a utop # prompt, where you can enter OCaml expressions.
Use ;; to tell the REPL you've finished typing and it's time to evaluate, e.g.
1 2 | |
Alternatively, you can install things using opam (OCaml's package manager). Note: you'll need a Linux distribution with libinput 1.20 or later.
opam install libinput utop
utop
Then, at the utop prompt enter #require "libinput";; (including the leading #).
Opening devices with libinput
To use libinput, you start by creating a context:
1 2 | |
Here, I'm using a "path" context, where we add devices manually. You can also use a "udev" one and it will find all the devices automatically.
The "interface" says how libinput should open the device files.
unix_direct means we just try to open them directly using OCaml's Unix module
(without using DBus).
It's a good idea to turn on logging at this point:
1 2 | |
As noted above, ordinary users can't normally open devices directly:
1 2 3 4 | |
We could use DBus to get access, but for testing purposes this isn't very helpful.
systemd-logind requires us to call TakeControl before we can use TakeDevice,
and this fails because Sway is already using the devices.
We could run it from a text console,
but taking control also takes control of the screen and keyboard away from the console,
and then we can't see what we're doing!
It's actually pretty difficult to recover from this situation
(I managed it once using SysRq somehow, but the second time I had to power-off).
Instead, I suggest just making yourself the owner of the devices:
$ sudo chown "$USER" /dev/input/event*
Then we can read events from them without interfering with Sway.
The owner will get set back to root next time systemd-udevd creates them
(e.g. after reboot or hotplug).
1 2 3 4 | |
Calling dispatch causes libinput to read low-level events from Linux
and queue up its own high-level ones for us.
We can then pop them off the queue with Event.get:
1 2 3 4 5 6 7 8 9 10 | |
Once there are no more events in the queue, it returns None:
1 2 | |
After wiggling the mouse we can use dispatch again:
1 2 3 4 5 6 | |
We have to call dispatch quickly after an event otherwise it gets dropped.
Also, libinput has various timers that need to be called promptly
(for example, it can discard very short clicks that are just the contacts bouncing after a real click).
So when calling dispatch manually, you'll probably see things like this:
1 2 3 | |
Still, you should be able to get some events by being reasonably quick at
pressing Return to run dispatch after using the mouse:
1 2 3 4 5 6 7 8 9 | |
Events from Event.get are typed as Unclassified.
Although the REPL displays them as OCaml records, they're actually C structures
and need to be accessed via getter functions.
Calling Event.get_type returns the event as an OCaml variant:
1 2 3 4 5 6 7 8 | |
You can then match on it and get a typed C event:
1 2 | |
Now we know this is a button event, we can use button-event-specific functions, e.g.
1 2 | |
simple.ml is an example program that prints out events in a loop:
$ dune exec -- ./examples/simple.exe
Created libinput context
{type = `Device_added _;
device = {sysname = "event2"; name = "Logitech USB Optical Mouse";
bustype = 0; vendor = 0x46d; product = 0xc077; output = null;
seat = {physical_name = "seat0"; logical_name = "default"}}]}
Waiting for events...
{type = `Pointer_scroll_wheel {time = 47360.187142; value120 = (0, -120)};
device = {sysname = "event2"; name = "Logitech USB Optical Mouse"; ...}]}
...
There's lots more to the libinput API (I probably should have looked at the length of the header file before deciding to make the OCaml bindings!). For example:
- Support for keyboards, switches, graphics tablets and touch screens (but not for joysticks, oddly).
- Checking what features devices provide.
- Configuration settings (e.g. left-handled mode, middle-button emulation, scroll direction).
- Controlling LEDs.
Thoughts on the C bindings
As with libdrm-ocaml, I used ctypes to make the bindings and it worked well again.
Ref-counting and garbage collection
The main problem I had was with the lifecycle of resources,
due to libinput's unusual reference counting system.
In this system, e.g. libinput_device_ref increments the count for a device,
libinput_device_unref decrements it,
and libinput will free a resource when its count reaches zero.
This is all perfectly normal.
What is unusual about libinput is that it can also free resources when the ref-count is not zero.
Specifically, freeing a context immediately destroys everything owned by that context
(see Bug 91872, resolved as WONTFIX to avoid breaking existing users).
I had a look at how python-libinput handles this. The docs say it "provides high-level object oriented api, taking care of reference counting, memory management and the like automatically". However, I didn't see any code to deal with this, and when I tested it Python just segfaulted.
I ended up with a somewhat complicated system where every reference can be invalidated or GC'd.
As the GC finaliser can run in any thread and libinput isn't thread-safe,
the finaliser pushes the reference onto a (lock-free) queue and
lets the main thread collect it on the next call to dispatch.
See context0.ml for more details.
Richer types
The C code has lots of comments about event types, e.g.
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
OCaml doesn't need any of this because the type prevents it from being used incorrectly:
1 2 | |
Callbacks
One annoying feature of libinput's design is that it wants to invoke a user-provided function to open devices. When using DBus, this means we have to run a nested event loop inside the callback handler, and the application can't do anything else until it gets a reply.
libinput already has a queue of events for the application to process; it would have been easier if it just added a "new device detected" event to the queue and let the application open it in its own time and notify the library at the end.
Lander game
To test my new libinput bindings, I made a Lander-style game. You can run it like this (if not using Nix, consult the README for alternative instructions):
git clone https://github.com/talex5/vulkan-test.git -b lander
cd vulkan-test
nix develop
dune exec -- ./src/main.exe
If run with $WAYLAND_DISPLAY set it will play in a Wayland window (and the compositor will use libinput),
while running it from a text console (e.g. with Ctrl-Alt-F2) will use libinput-ocaml.
It uses DBus to get access to the devices, if the login service is available.
The game starts with the ship flying across a randomly generated landscape, and you must take control quickly or it will crash (this is partly for dramatic purposes, but also so that if my libinput code doesn't work for some reason then the game will end after a few seconds and you'll get back control of your computer).
Move the mouse to angle the craft and use button-1 for thrust. The aim of the game is to find the landing pad and land on it. You can also steer the craft using a graphics tablet, which works surprisingly well and also gives you pressure-sensitive thrust (this only works in libinput-ocaml mode, although in Wayland mode your compositor might have the tablet emulate a mouse, without pressure sensitivity).
You'll probably come across the pad just from flying around randomly for a minute or so, but you could also read the map generation code for hints on how to find it more quickly.
To make the game, I started with the Viking room code from the Vulkan tutorial
(from my earlier Vulkan graphics in OCaml vs C post).
I replaced the room model with a simple spaceship model
and added a scrolling landscape below it,
then used libinput to add controls.
The changes to vt.ml use libinput-ocaml,
while the window.ml changes are similar but for running under Wayland.
Some suggestions for possible extensions you might like to try:
- Allow using button-3 for half-thrust (very easy).
- Allow using Tab for thrust to make playing on a laptop's touchpad easier (easy for vt.ml, but you'll need to add Wayland keyboard support if you want Wayland to work too).
- Allow pausing the game by pressing
P(working out which key corresponds to which letter looks tricky in general and requires using a keymap, but for getting it working on your own computer you can just see what numberPproduces). - Add touch-screen support (I wanted to try this on my old PinePhone, but sadly its GPU doesn't support Vulkan).
- Hold down
Mto view a map (hint: the landscape module already has an overhead view for debugging).
Non-libinput aspects of the game
Making the game made me realise that I'd misunderstood something important about Vulkan. I'd got the impression from the tutorial that a Vulkan application has a single graphics pipeline that renders everything, but in fact a "pipeline" is more like a drawing tool and typically you need several of them. The lander game uses three: one to draw the landscape, one to draw the ship, and one for particles.
My previous code had e.g. Pipeline and Input modules,
but this structure doesn't make sense when you have multiple pipelines.
To minimise the diff, I first refactored the Viking room code,
splitting the Pipeline module into Scene (for the whole thing) and Room (the room-rendering pipeline).
After that, the Vulkan API started making a lot more sense to me!
I also discovered that my Viking room code didn't work on the Raspberry Pi 4 (for multiple reasons),
and spent quite a while getting it working there too.
Sadly, the lander game still doesn't run there as the Pi's graphics drivers don't fully support even Vulkan 1.1
(needed for SV_VertexID).
If you'd like to experiment with the game, here are some more ideas:
- Changing
gravityandengine_powerat the top of ship.ml will change how hard the game is. - Edit map.ml to make the terrain more interesting. For example, you could manually add pillars or pyramids, or simulate coastal erosion to get cliffs, etc.
- I used Perlin noise to generate the landscape. My code had various bugs while I was writing it, and most of them led to interesting effects!
- Add a radar display showing the map in the corner of the screen. This should be pretty easy because the tile colours are already available as a texture, so it should be just a case of displaying a textured rectangle.
Thanks to the OCaml Software Foundation for sponsoring this work.