Category: ocaml | Thomas Leonard's blog2025-11-16T09:00:00+00:00https://roscidus.com/blog/Thomas LeonardLinux mode setting, from the comfort of OCaml2025-11-16T09:00:00+00:00https://roscidus.com/blog/blog/2025/11/16/libdrm-ocamlLinux provides the KMS (Kernel Mode Setting) API to let applications query and configure display settings.
It's used by Wayland compositors and other programs that need to configure the hardware directly.
I found the C API a little verbose and hard to follow so I made libdrm-ocaml,
which lets us run commands interactively in a REPL.
We'll start by discovering what hardware is available and how it's currently configured,
then configure a monitor to display a simple bitmap, and then finally render a 3D animation.
The post should be a useful introduction to KMS even if you don't know OCaml.
libdrm scans the /dev/dri/ directory looking for devices.
It uses stat to find the device major and minor numbers and uses the virtual /sys filesystem to get information about each one.
This is a PCI device, and the information corresponds to the values from lspci, e.g.
Each graphics device can have a primary and a render node.
The primary node gives full access to the device, including configuring monitors,
while the render node just allows applications to render scenes to memory.
In the last post I was using the render to node to create a 3D image,
and then sending it to the Wayland compositor for display.
This time we'll be doing the display ourselves, so we need to open the primary node:
If you're familiar with the C API, this corresponds to the drmGetVersion function,
and Drm.Device.list corresponds to drmGetDevices2;
I reorganised things a bit to make better use of OCaml's modules.
Note: The Kernel Mode Setting functions are in the Drm.Kms module.
The C API calls these functions drmMode*, but I found that confusing as
e.g. drmModeGetResources sounds like you're asking for the resources of a mode.
A CRTC is a CRT Controller, and typically controls a single monitor
(known as a Cathode Ray Tube for historical reasons).
Framebuffers provide image data to a CRTC (we create framebuffers as needed).
Connectors correspond to physical connectors (e.g. where you plug in a monitor cable).
An Encoder encodes data from the CRTC for a particular connector.
This is DisplayPort connector 1 (usually called DP-1) and it's currently Connected.
The connector also says which modes are available on the connected monitor.
I was lucky in that the first connector was the one I'm using,
but really we should get all the connectors and filter them to find the connected ones.
List.map can be used to run get on each of them:
In the libdrm C API, IDs are just integers.
To avoid mix-ups, I made them distinct types in the OCaml API.
For example, if you try to use an encoder ID as a connector ID:
You can get any kind of ID with Drm.Id.of_int (e.g. K.Connector.get dev (Drm.Id.of_int 71)),
but that's still a bit verbose, so you might prefer to (re)define a prefix operator for it, e.g.
Note: I annotated various pretty-printer functions with [@@ocaml.toplevel_printer],
which causes utop to use them by default to display values of the corresponding type.
For example, showing a list of modes uses this short summary form.
Displaying an individual mode shows all the information.
Here's the first mode:
(Module.(expr) is a short-hand that brings all of Module's symbols into scope for expr,
so we don't have to repeat the module name for both set and atomic)
And getting the properties again, we now have an extra CRTC_ID,
telling us which controller this connector is currently using:
Those are really just internal artifacts of the helper libraries used to
implement KMS drivers. Besides that they make it unnecessarily more
complicated for userspace to figure out which connections between a CRTC and
a connector are possible, and what kind of cloning is supported, they serve
no purpose in the userspace API. Unfortunately encoders have been exposed to
userspace, hence can’t remove them at this point. Furthermore the exposed
restrictions are often wrongly set by drivers, and in many cases not powerful
enough to express the real restrictions.
Note: We need Option.get here because a connector might not have an encoder set yet.
Where the C API uses 0 to indicate no resource,
the OCaml API uses None to force us to think about that case.
As the documentation says, the encoder is mainly useful to get the CRTC ID:
An active CRTC has a mode set (presumably from the connector's list of supported modes),
and a framebuffer with the image to be displayed.
If I keep calling Crtc.get, I see that it is sometimes showing framebuffer 93 and sometimes 94.
My Wayland compositor (Sway) updates one framebuffer while the other is being shown, then switches which one is displayed.
Framebuffers
My CRTC is currently displaying the contents of framebuffer 93:
A framebuffer has up to 4 framebuffer planes (not to be confused with CRTC planes; see later),
each of which references a buffer object (also known as a BO and referenced with a GEM handle).
This framebuffer is using the XR24 format, where there is a single BO with 32 bits for each pixel
(8 for red, 8 green, 8 blue and 8 unused).
Some formats use e.g. a separate buffer for each component
(or a different part of the same buffer, using offset).
Modern graphics cards also support format modifiers, but my card is too old so I just get None.
Linux's fourcc.h header file describes the various formats and modifiers.
Modifiers seem to be mainly used to specify the tiling.
I don't have permission to see the buffer object, so it appears as (handle = None).
The pitch is the number of bytes from one row to the next (also known as the stride).
Here, the 15360 is simply the width (3840) multiplied by the 4 bytes per pixel.
CRTC planes
In fact, Crtc.get is an old API that only covers the basic case of a single framebuffer.
In reality, a CRTC can combine multiple CRTC planes, which for some reason aren't returned with the other resources
and must be requested separately:
Plane 52 is a Primary plane and is using framebuffer 93 (as we saw before).
Plane 55 is a Cursor plane, using framebuffer 98 (and the AR24 format, with alpha/transparency).
A plane chooses which part of the frame buffer to show (SRC_X, SRC_Y, SRC_W and SRC_H)
and where it should appear on the screen (CRTC_X, CRTC_Y, CRTC_W and CRTC_H).
The source values are in 16.16 format (i.e. shifted left 16 bits).
Oddly, Plane.get returned crtc_x,crtc_y = 0,0 for both planes, but
the properties show the correct cursor location (CRTC_X = 3105; CRTC_Y = 1518;).
Having the cursor on a separate plane avoids having to modify the main screen image
whenever the mouse pointer moves, which is good for low latency
(especially if the GPU is busy rendering something else at the time),
power consumption (the GPU can stay powered down),
and allows showing an application's buffer full screen without the compositor
needing to modify the application's buffer.
You might also have some Overlay planes,
which can be useful for displaying video.
My graphics card seems to be too old for that.
Expanded resources diagram
Here's an expanded diagram showing some more possibilities:
The reason is that I'm currently running a graphical desktop and Sway owns the device
(so my dev is not the DRM "master"):
12
utop#Drm.Device.is_masterdev;;-:bool=false
That can be fixed by switching to a different VT (e.g. with Ctrl-Alt-F2) and running it there.
However, this will result in a second problem: I won't be able to see what I'm doing!
If you have a second computer then you can SSH in and test things out from there, but
for simplicity we'll leave the utop REPL at this point and write some programs instead.
For example, query.ml shows the information we discovered above:
Linux provides two ways to configure modes: the old non-atomic API and the newer atomic one.
examples/nonatomic.ml contains a simple example of the older (but simpler) API.
It starts by finding a device (the first one with a primary node supporting KMS), then
finds all connected connectors (as we did above), and calls show_test_page on each one:
restoring_afterwards stores the current configuration, runs the callback,
and then puts things back to normal when that finishes (or you press Ctrl-C).
The program waits for 2 seconds after showing the test page before exiting.
show_test_page finds the CRTC (as we did above),
takes the first supported mode, creates a test framebuffer of that size,
and configures the CRTC to display it:
1234567891011121314
letshow_test_page(t:Resources.t)(c:K.Connector.t)=matchc.encoder_idwith|None->println"%a has no encoder (skipping)"K.Connector.pp_namec|Someencoder_id->matchK.Encoder.gett.devencoder_idwith|{crtc_id=None;_}->println"%a's encoder has no CRTC (skipping)"K.Connector.pp_namec|{crtc_id=Somecrtc_id;_}->println"Showing test page on %a"K.Connector.pp_namec;letmode=List.hdc.modesinletsize=(mode.hdisplay,mode.vdisplay)inletfb=Test_image.createt.devsizeinK.Crtc.sett.devcrtc_id(Somemode)~fb~pos:(0,0)~connectors:[c.connector_id]
If the connector doesn't have a CRTC, we could find a suitable one and use that,
but for simplicity the example just skips such connectors.
To run the example (switch away from any graphical desktop first or it won't work):
dune exec -- ./examples/nonatomic.exe
Dumb buffers
Typically the pixel data to be displayed comes from some complex rendering pipeline,
but Linux also provides dumb buffers for simple cases such as testing.
The Test_image.create function used above creates a dumb buffer with a test pattern:
Dumb.create allocates memory for the image data.
Dumb.map makes it appear in host-memory as an OCaml bigarray.
The loop sets each 32-bit int in the image to some colour c.
Then we wrap this data up as an XR24-format framebuffer with a single plane:
let()=Utils.with_device@@funt->Drm.Client_cap.(set_exnatomic)t.devtrue;letconnected=List.filterUtils.is_connectedt.connectorsinprintln"Found %d connected connectors"(List.lengthconnected);letfree_planes=ref(K.Plane.listt.dev)inletrq=K.Atomic_req.create()inList.iter(show_test_page~free_planestrq)connected;println"Checking that commit will work...";matchK.Atomic_req.commit~test_only:truet.devrqwith|exceptionUnix.Unix_error(code,_,_)->println"Mode-setting would fail with error: %s"(Unix.error_messagecode)|()->println"Pre-commit test passed.";Utils.restoring_afterwardst@@fun()->K.Atomic_req.committ.devrq;Unix.sleep2
The steps are:
Use set_exn atomic to enable atomic mode.
Create an atomic request (rq).
Use show_test_page to populate it with the desired property changes.
(optional) Check that it will work (~test_only:true).
Commit the changes (Atomic_req.commit).
The advantage here is that either all changes are successfully applied at once or nothing changes.
This avoids various problems with flickering or trying to roll back partial changes.
show_test_page needs a couple of modifications.
First, we have to find a plane (rather than using the old Crtc.set which assumes a single plane),
and then we set the plane's FB_ID property to the new framebuffer in the request:
For the example, I actually set more properties and defined an operator to make the code a bit neater:
1234567891011121314
let(.%{}<-)objpropvalue=K.Atomic_req.add_propertyrqobjpropvalueinplane.%{K.Plane.fb_id}<-Somefb;(* Source region on frame-buffer: *)plane.%{K.Plane.src_x}<-Drm.Ufixed.of_int0;plane.%{K.Plane.src_y}<-Drm.Ufixed.of_int0;plane.%{K.Plane.src_w}<-Drm.Ufixed.of_int(fstsize);plane.%{K.Plane.src_h}<-Drm.Ufixed.of_int(sndsize);(* Destination region on CRTC: *)plane.%{K.Plane.crtc_x}<-0;plane.%{K.Plane.crtc_y}<-0;plane.%{K.Plane.crtc_w}<-fstsize;plane.%{K.Plane.crtc_h}<-sndsize;
In libdrm-ocaml, properties are typed, so you can't forget to convert the source values to fixed point format.
3D rendering
The examples above use a dumb-buffer, but it's fairly simple to replace that with a Vulkan buffer.
The code in the last post exported the image memory from Vulkan as a dmabuf FD and sent it to the Wayland compositor.
Now, instead of sending it we just need to import it into our device (with Drm.Dmabuf.to_handle)
and use that handle instead of the dumb-buffer one.
I added a simple surface abstraction to the test code, wrapping the Window module's API
so that the rendering code doesn't need to care whether it's rendering to a Wayland window or directly to the screen.
Then I made a Vt module implementing the new Surface.t type for rendering directly to a Linux VT.
To get the animation working, I used K.Crtc.page_flip to update the framebuffer (I could also have used the atomic API).
The kernel waits until the encoder has finishing sending the current frame before switching to the new one,
which avoids tearing.
We also need to ask the kernel to tell us when this happens, which is done by setting the optional ~event argument to some number.
You can read events from the device file and parse them with Drm.Event.parse.
If you want to try it, this should produce an animated room:
git clone https://github.com/talex5/vulkan-test -b kms-3d
cd vulkan-test
nix develop
make download-example
dune exec -- ./src/main.exe 10000 viking_room.obj viking_room.png
If run with $WAYLAND_DISPLAY set, it will open a Wayland window (as before),
but if run from a text console then it should render the animation directly using KMS.
Linux VTs
When the user switches to another virtual terminal (e.g. with Ctrl-Alt-F3),
we should call Drm.Device.drop_master to give up being the master,
allowing the application running on the new terminal to take over.
We should also switch the VT to KD_GRAPHICS mode while using it,
to stop the kernel trying to manage it.
I didn't implement either of these features, but see How VT-switching works for details.
Debugging
If you get an unhelpful error code from the kernel (e.g. EINVAL), enabling debug messages is often helpful.
Writing 4 to /sys/module/drm/parameters/debug enables KMS debug messages, which can be seen in the dmesg output.
Write 0 to the file afterwards to turn the messages off again.
modinfo -p drm lists the various options.
Conclusions
I hope you found being able to explore the libdrm API interactively from the OCaml top-level
made it easier to learn about how Linux manages displays.
As when doing Vulkan in OCaml,
a lot of the noise from C is removed and I think that the essentials of what is going on are easier to see.
I used ocaml-ctypes for the C bindings, and this was my first time using it in "stubs" mode
(where it pre-generates C bindings from OCaml definitions).
This has the advantage that the C type checker checks that the definitions are correct,
and it worked well.
Dune's Stub Generation feature generates the build rules for this semi-automatically.
Deciding what OCaml types to use for the C types was quite difficult.
For example, C has many different integer types (int, long, uint32_t, etc),
but using lots of types is more painful in OCaml where e.g. + only works on int.
I used OCaml's int type when possible, and other types only when the value might not fit
(e.g. an image size on a 32-bit platform might not fit into an OCaml int, which is one bit shorter).
The C API is somewhat inconsistent about types.
e.g. drmModePageFlipTarget takes a uint32_t target_vblank argument for the sequence number,
while page_flip_handler confirms the event by giving it as unsigned int sequence.
Meanwhile, the sequence_handler event gives it as uint64_t sequence.
I'm not sure what happens if the sequence number gets too large to fit in a 32-bit integer.
Anyway, I think I understand mode setting a lot better now,
and I'm getting faster at debugging graphics problems on Linux
(e.g. when element-desktop failed to start recently after I updated it).
Thanks to the OCaml Software Foundation for sponsoring this work.
]]>Vulkan graphics in OCaml vs C2025-09-20T09:00:00+00:00https://roscidus.com/blog/blog/2025/09/20/ocaml-vulkanI convert my Vulkan test program from C to OCaml and compare the results,
then continue the Vulkan tutorial in OCaml, adding 3D, textures and depth buffering.
In Investigating Linux graphics,
I wrote a little C program to help me learn about GPUs by drawing a triangle.
But I wondered if using OCaml instead would make my life easier.
It didn't, because there were no released OCaml Vulkan bindings,
but I found some unfinished ones by Florian Angeletti.
The bindings are generated mostly automatically from the Vulkan XML specification,
and with a bit of effort I got them working well enough to
continue with the Vulkan tutorial,
which resulted in this nice Viking room:
In this post, I'll be looking at how the C code compares to the OCaml.
First, I did a direct line-by-line port of the C, then I refactored it to take better advantage of OCaml.
(Note: the Vulkan tutorial is actually using C++, but I'm comparing my C version to OCaml)
Running it yourself
If you want to try it yourself (note: it requires Wayland):
As the OCaml Vulkan bindings (Olivine) are unreleased,
I included a copy of my patched version in vendor/olivine.
The dune exec command will build them automatically.
The ocaml branch above just draws one triangle.
If you want to see the 3D room pictured above, use ocaml-3d instead:
git clone https://github.com/talex5/vulkan-test -b ocaml-3d
cd vulkan-test
nix develop
make download-example
dune exec -- ./src/main.exe 10000 viking_room.obj viking_room.png
The direct port
Porting the code directly, line by line, was pretty straight-forward:
This is only approximate; sometimes I added or removed blank lines, etc.
Some things were a bit easier and others a bit harder. It mostly balanced out.
As an example, one thing that makes the OCaml shorter is that arrays are passed as a single item,
whereas C takes the length separately.
On the other hand, single-item arrays can be passed in C by just giving the address of the pointer,
whereas OCaml requires an array to be constructed separately.
Also, I had to include some bindings for the libdrm C library.
Labelled arguments
The OCaml bindings use labelled arguments
(e.g. the VK_TRUE argument in the screenshot above became ~wait_all:true in the OCaml),
which is longer but clearer.
The OCaml code uses functions to create C structures, which looks pretty similar due to labels.
For example:
An advantage is that the sType field gets filled in automatically.
Enums and bit-fields
Enumerations and bit-fields are namespaced, which is a lot clearer
as you can see which part is the name of the enum and which part is the particular value.
For example, VK_ATTACHMENT_STORE_OP_STORE becomes Vkt.Attachment_store_op.Store.
Also, OCaml usually knows the expected type and you can omit the module, so:
12345678910
VkAttachmentDescriptioncolorAttachment={.format=format,.samples=VK_SAMPLE_COUNT_1_BIT,.loadOp=VK_ATTACHMENT_LOAD_OP_CLEAR,// Clear framebuffer before rendering.storeOp=VK_ATTACHMENT_STORE_OP_STORE,.stencilLoadOp=VK_ATTACHMENT_LOAD_OP_DONT_CARE,.stencilStoreOp=VK_ATTACHMENT_STORE_OP_DONT_CARE,.initialLayout=VK_IMAGE_LAYOUT_UNDEFINED,.finalLayout=VK_IMAGE_LAYOUT_GENERAL,};
becomes
12345678910
letcolor_attachment=Vkt.Attachment_description.make()~format:format~samples:Vkt.Sample_count_flags.n1~load_op:Clear(* Clear framebuffer before rendering *)~store_op:Store~stencil_load_op:Dont_care~stencil_store_op:Dont_care~initial_layout:Undefined~final_layout:Generalin
Bit-fields and enums get their own types (they're not just integers), so you can't use them in the wrong place
or try to combine things that aren't bit-fields (and so the _BIT suffix isn't needed).
One particularly striking example of the difference is that
because the spec says these are all non-optional, rather than that they are only needed when blending is enabled.
There's a similar situation with the Wayland code:
the OCaml compiler requires you to provide a handler for all possible events.
For example, OCaml forced me to write a handler for the window close event
(and so closing the window works in the OCaml version, but not in the C one).
Likewise, if the compositor returns an error from create_immed the OCaml version logs it,
while the C version ignored the error message, because the C compiler didn't remind me about that.
Loading shaders
Loading the shaders was easier.
The C version has code to load the shader bytecode from disk, but in the OCaml I used ppx_blob
to include it at compile time, producing a self-contained executable file:
1
load_shader_moduledevice[%blob"./vert.spv"]
Logging
OCaml has a somewhat standard logging library, so I was able to get the log messages shown as I wanted
without having to pipe the output through awk.
And, as a bonus, the log messages get written in the correct order now.
e.g. the C libwayland logs:
The OCaml bindings return a result type for functions that can return errors,
using polymorphic variants to say exactly which errors can be returned by each function.
That's clever, but I found it pretty useless in practice and I followed the Olivine example code
in immediately turning every Error result into an exception.
You can then handle errors at a higher level (unlike the C, which just calls exit).
Maybe Olivine should be changed to do that itself.
I thought I'd been rigorous about checking for errors in the C, but I missed some places (e.g. vkMapMemory).
The OCaml compiler forced me to handle those too, of course.
Refactored version
One reason to switch to OCaml was because I was finding it hard to see how all the C code fit together.
I felt that the overall structure was getting lost in the noise.
While the initial OCaml version was similar to the C,
I think the refactored version is quite a bit easier to read.
Moving code to separate files is much easier than in C.
There, you typically need to write a header file too, and then include it from the other files.
But in the OCaml I could just move e.g. export_semaphore to export in a new file called semaphore.ml and
refer to it as Semaphore.export.
Because each file gets its own namespace, you don't have to guess where functions are defined,
and you don't get naming conflicts between symbols in different files.
The build system (dune) automatically builds all modules in the correct order.
Olivine wrappers
I added a vulkan directory with wrappers around the auto-generated Vulkan functions
with the aim of removing some noise.
For example, the wrappers take OCaml lists and convert them to C arrays as needed,
and raise exceptions on error instead of returning a result type.
Sometimes they do more, as in the case of queue_submit.
That took separate wait_semaphores and wait_dst_stage_mask arrays,
requiring them to be the same length.
By taking a list of tuples, the wrapper avoids the possibility of this error.
The old submit code:
Sometimes the new API drops features I don't use (or don't currently understand).
For example, my new submit only lets you submit one command buffer at a time
(though each buffer can have many commands).
I moved various generic helper functions like find_memory_type to the wrapper library,
getting them out of the main application code.
Separating out these libraries made the code longer, but I think it makes it easier to read:
The C code has a single thread with a single stack,
using callbacks to redraw when the compositor is ready.
OCaml has fibers (light-weight cooperative threads), so we can use a plain loop:
The Promise.await suspends this fiber, allowing e.g. the Wayland code to handle incoming events.
I find that makes the logic easier to follow.
Using the CPU and GPU in parallel
Next I split off the input handling from the huge render.ml file into input.ml.
The Vulkan tutorial creates one uniform buffer for the input data for each frame-buffer, but this seems wasteful.
I think we only need at most two: one for the GPU to read, and one for the CPU to write for the next frame,
if we want to do that in parallel.
To allow this parallel operation I also had to create a pair of command buffers.
The duo.ml module holds the two (input, command-buffer) jobs and swaps them on submit.
Resizing and resource lifetimes
When the window size changes we need to destroy the old swap-chain and recreate all the images, views
and framebuffers.
My C code didn't bother, and just kept things at 640x480.
The main problem here is how to clean up the old resources.
We could use the garbage collector, but the framebuffers are rather large and I'd like to get them freed promptly.
Also, Vulkan requires things to be freed in the correct order, which the GC wouldn't ensure.
I added code to free resources by having each constructor take a sw switch argument.
When the switch is turned off, all resources attached to it are freed.
That makes it easy to scope things to the stack: when the Switch.run block ends, all resources it created are freed.
But the life-cycle of the swap-chain is a little complicated.
I don't want to clutter the main application loop with the logic of adapting to size changes.
Again, OCaml's fibers system makes it easy to have multiple stacks so I have another fiber run:
The C code created a fixed set of 4 framebuffers on each resize, but the OCaml only creates them as needed.
When dragging the window to resize that means we may only need to create one at each size,
and when keeping a steady size, it seems I only need 3 framebuffers with Sway.
The main loop changes slightly so that it just triggers the render_loop fiber:
I'm not sure if freeing the framebuffers immediately is safe,
since in theory the GPU might still be using them if the display server requests a new frame
at a new size before the previous one has finished rendering on the GPU.
Possibly freed OCaml resources should instead get added to
a list of things to free on the C side the next time the GPU is idle.
The 3D version
Although it looks a lot more impressive, the 3D version isn't that much more work than the 2D triangle.
I used the Cairo library to load the PNG file with the textures
and then added a Vulkan sampler for it.
The shader code has to be modified to read the colour from the texture.
The most complex bit is that the texture needs to be copied
from Cairo's memory to host memory that's visible to the GPU,
and from there to fast local memory on the GPU (see texture.ml).
Other changes needed:
There's a bit of matrix stuff to position the model and project it in 3D.
The pipeline adds a depth buffer so near things obscure things behind them, regardless of the drawing order.
I didn't get my C version to do the 3D bits, but for comparison here's the Vulkan tutorial's official C++ version.
Garbage collection
To render smoothly at 60Hz, we have about 16ms for each frame.
You might wonder if using a garbage collector would introduce pauses and cause us to miss frames,
but this doesn't seem to be a problem.
In C, you can improve performance for frame-based applications by using a bump allocator:
Create a fixed buffer with enough space for every allocation needed for one frame.
Allocate memory just by allocating sequentially in the region (bumping the next-free-address pointer).
At the end of each frame, reset the pointer.
This makes allocation really fast and freeing things at the end costs nothing.
Implementing this in C requires special code,
but OCaml works this way by default, allocating new values sequentially onto the minor heap.
At the end of each frame, we can call Gc.minor to reset the heap.
Gc.minor scans the stack looking for pointers to values that are still in use
and copies any it finds to the major heap.
However, since we're at the end of the frame, the stack is pretty much empty and there's almost nothing to scan.
I captured a trace of running the 3D room version with a forced minor GC at the end of every frame:
make && eio-trace run ./_build/default/src/main.exe
The four long grey horizontal bars are the main fibers.
From top to bottom they are:
The main application loop (incrementing the frame counter and triggering the render loop fiber).
An ocaml-wayland fiber, receiving messages from the display server
(and spawning some short-lived sending fibers).
The render_loop fiber (sending graphics commands to the GPU).
A fiber used internally by the IO system.
The green sections show when each fiber is running and the yellow background indicates when the process is sleeping.
The thin red columns indicate time spent in GC (which we're here triggering after every frame).
If I remove the forced Gc.minor after each frame then the GC happens less often,
but can take a bit longer when it does.
Still not nearly long enough to miss the deadline for rendering the frame though.
Collection of the major heap is done incrementally in small slices and doesn't cause any trouble.
So, we're only using a tiny fraction of the available time.
Also, I suspect the CPU is running in a slow power-saving mode due to all the sleeping;
if we had more work to do then it would probably speed up.
Conclusions
Doing Vulkan programming in OCaml has advantages (clearer code, easier refactoring),
but also disadvantages (unfinished and unreleased Vulkan bindings, some friction using a C API from OCaml,
and I had to write more support code, such as some bindings for libdrm).
As a C API, Vulkan is not safe and will happily segfault if passed incorrect arguments.
The OCaml bindings do not fix this, and so care is still needed.
I didn't bother about that because it wasn't a problem in practice,
and properly protecting against use-after-free will probably require some changes to OCaml
(e.g. unmapping memory isn't safe without something like the "modes" being prototyped in
OxCaml).
I'm slowly upstreaming my changes to Olivine; hopefully this will all be easier to use one day!
]]>OCaml 5 performance part 22024-07-22T11:00:00+00:00https://roscidus.com/blog/blog/2024/07/22/performance-2The last post looked at using various tools to understand why an OCaml 5 program was waiting a long time for IO.
In this post, I'll be trying out some tools to investigate a compute-intensive program that uses multiple CPUs.
Further discussion about this post can be found on discuss.ocaml.org.
The problem
OCaml 4 allowed running multiple "system threads", but only one can have the OCaml runtime lock,
so only one can be running OCaml code at a time.
OCaml 5 allows running multiple "domains", all of which can be running OCaml code at the same time
(each domain can also have multiple system threads; only one system thread can be running OCaml code per domain).
The ocaml-ci service provides CI for many OCaml programs,
and its first step when testing a commit is to run a solver to select compatible versions for its dependencies.
Running a solve typically only takes about a second, but it has to do it for each possible test platform,
which includes versions of the OCaml compiler from 4.02 to 4.14 and 5.0 to 5.2,
multiple architectures (32-bit and 64-bit x86, 32-bit and 64-bit ARM, PPC64 and s390x),
operating systems (Alpine, Debian, Fedora, FreeBSD, macos, OpenSUSE and Ubuntu, in multiple versions), etc.
In total, this currently does 132 solver runs per commit being tested
(which seems too high to me, but let's ignore that for now).
The solves are done by the solver-service,
which runs on a couple of ARM machines with 160 cores each.
The old OCaml 4 version used to work by spawning lots of sub-processes,
but when OCaml 5 came out, I ported it to use a single process with multiple domains.
That removed the need for lots of communication logic,
and allowed sharing common data such as the package definitions.
The code got a lot shorter and simpler, and I'm told it's been much more reliable too.
But the performance was surprisingly bad.
Here's a graph showing how the number of solves per second scales with the number of CPUs (workers) being used:
The "Processes" line shows performance when forking multiple processes to do the work, which looks pretty good.
The "Domains" line shows what happens if you instead spawn domains inside a single process.
Note: The original service used many libraries (a mix of Eio and Lwt ones),
but to make investigation easier I simplified it by removing most of them.
The simplified version doesn't use Eio or Lwt;
it just spawns some domains/processes and has each of them do the same solve in a loop a fixed number of times.
ThreadSanitizer
When converting a single-domain OCaml 4 program to use multiple cores it's easy to introduce races.
OCaml has ThreadSanitizer (TSan) support which can detect these.
To use it, install an OCaml compiler with the tsan option:
Things run a lot slower and require more memory with this compiler, but it's good to check:
$ ./_build/default/stress/stress.exe --internal-workers=2
[...]
WARNING: ThreadSanitizer: data race (pid=133127)
Write of size 8 at 0x7ff2b7814d38 by thread T4 (mutexes: write M88):
#0 camlOpam_0install__Model.group_ors_1288 lib/model.ml:70 (stress.exe+0x1d2bba)
#1 camlOpam_0install__Model.group_ors_1288 lib/model.ml:120 (stress.exe+0x1d2b47)
...
Previous write of size 8 at 0x7ff2b7814d38 by thread T1 (mutexes: write M83):
#0 camlOpam_0install__Model.group_ors_1288 lib/model.ml:70 (stress.exe+0x1d2bba)
#1 camlOpam_0install__Model.group_ors_1288 lib/model.ml:120 (stress.exe+0x1d2b47)
...
Mutex M88 (0x558368b95358) created at:
#0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1295 (libtsan.so.2+0x50468)
#1 caml_plat_mutex_init runtime/platform.c:57 (stress.exe+0x4763b2)
#2 caml_init_domains runtime/domain.c:943 (stress.exe+0x44ebfe)
...
Mutex M83 (0x558368b95240) created at:
#0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1295 (libtsan.so.2+0x50468)
#1 caml_plat_mutex_init runtime/platform.c:57 (stress.exe+0x4763b2)
#2 caml_init_domains runtime/domain.c:943 (stress.exe+0x44ebfe)
...
Thread T4 (tid=133132, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x5e686)
#1 caml_domain_spawn runtime/domain.c:1265 (stress.exe+0x4504c4)
...
Thread T1 (tid=133129, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x5e686)
#1 caml_domain_spawn runtime/domain.c:1265 (stress.exe+0x4504c4)
...
SUMMARY: ThreadSanitizer: data race lib/model.ml:70 in camlOpam_0install__Model.group_ors_1288
The two mutexes mentioned in the output, M83 and M88, are the domain_lock,
used to ensure only one sys-thread runs at a time in each domain.
In this program we only have one sys-thread per domain and so can ignore them.
The output reveals that the solver used a global variable to generate unique IDs:
With that fixed, TSan finds no further problems (in this simplified version).
This gives us good confidence that there isn't any shared state:
TSan would report use of shared state not protected by a mutex,
and since the program was written for OCaml 4 it won't be using any mutexes.
That's good, because if one thread writes to a location that another reads then that requires coordination between CPUs,
which is relatively slow
(though we could still experience slow-downs due to false sharing,
where two separate mutable items end up in the same cache line).
However, while important for correctness, it didn't make any noticeable difference to the benchmark results.
perf
perf is the obvious tool to use when facing CPU performance problems.
perf record -g PROG takes samples of the program's stack regularly,
so that functions that run a lot or for a long time will appear often.
perf report provides a UI to explore the results:
Looks like we're spending most of our time solving, as expected.
But this can be misleading.
Because perf only records stack traces when the code is running, it doesn't report any time the process spent sleeping.
Note: I removed some columns with all zero values to save space.
We might expect to see 7 CPUs running at 100% and one idle CPU,
but in fact they're all moderately busy.
On the other hand, none of them spent more than 91% of its time running the solver code.
offcputime
offcputime will show why a process wasn't using a CPU
(it's like offwaketime, which we saw earlier, but doesn't record the waker).
Here I'm using pidstat to see all running threads and then examining one of the workers,
to avoid the problem we saw last time where the diagram included multiple threads:
Note: The ARM machine's kernel was too old to run offcputime, so I ran this on my machine instead,
with one main domain and six workers.
As I needed good stacks for C functions too, I ran stress.exe in an Ubuntu 24.04 docker container,
as recent versions of Ubuntu compile with frame pointers by default.
The raw output was very noisy, showing it waiting in many different places.
Looking at a few, it was clear it was mostly the GC (which can run from almost anywhere).
The output is just a text-file with one line per stack-trace, and bit of sed cleaned it up:
That removes the part of the stack-trace before any of various interrupt-type functions that can be called from anywhere.
The graph is blue to indicate that it shows time when the process wasn't running.
There are rather a lot of traces where we missed the user stack.
However, the results seem clear enough: when our worker is waiting, it's in the garbage collector,
calling caml_plat_spin_wait.
This is used to sleep when a spin-lock has been spinning for too long (after 1000 iterations).
The OCaml garbage collector
OCaml has a major heap for long-lived values, plus one fixed-size minor heap for each domain.
New allocations are made sequentially on the allocating domain's minor heap
(which is very fast, just adjusting a pointer by the size required).
When the minor heap is full the program performs a minor GC,
moving any values that are still reachable to the major heap
and leaving the minor heap empty.
Garbage collection of the major heap is done in small slices so that the application doesn't pause for long,
and domains can do marking and sweeping work without needing to coordinate
(except at the very end of a major cycle, when they briefly synchronise to agree a new cycle is starting).
However, as minor GCs move values that other domains may be using, they do require all domains to stop.
Although the simplified test program doesn't use Eio, we can still use eio-trace to record GC events
(we just don't see any fibers).
Here's a screenshot of the solver running with 24 domains on the ARM machine,
showing it performing GC work (not all domains are visible in the picture):
The orange/red parts show when the GC is running and the yellow regions show when the domain is waiting for other domains.
The thick columns with yellow edges are minor GCs,
while the thin (almost invisible) red columns without any yellow between them are major slices.
The second minor GC from the left took longer than usual because the third domain from the top took a while to respond.
It also didn't do a major slice before that; perhaps it was busy doing something, or maybe Linux scheduled a different process to run then.
Traces recorded by eio-trace can also be viewed in Perfetto, which shows the nesting better:
Here's a close-up of a single minor GC, corresponding to the bottom two domains from the second column from the left:
The domain triggering the GC (the bottom one here) enters a "stw_leader" (stop-the-world) phase
and waits for the other domains to stop.
One by one, the other domains stop and enter "stw_api_barrier" until all domains have stopped.
All domains perform a minor GC, clearing their minor heaps.
They then enter a "minor_leave_barrier" phase, waiting until all domains have finished.
Each domain returns to running application code.
We can now see why the solver spends so much time sleeping;
when a domain performs a minor GC, it spends most of the time waiting for other domains.
(the above is a slight simplification; domains may do some work on the major GC while waiting)
statmemprof
One obvious solution to GC slowness is to produce less garbage in the first place.
To do that, we need to find out where the most costly allocations are coming from.
Tracing every memory allocation tends to make programs unusably slow,
so OCaml instead provides a statistical memory profiler.
It was temporarily removed in OCaml 5 because it needed updating for the new multicore GC,
but has recently been brought back and will be in OCaml 5.3.
There's a backport to 5.2, but I couldn't get it to work,
so I just removed the domains stuff from the test and did a single-domain run on OCaml 4.14.
You need the memtrace library to collect samples and memtrace_viewer to view them:
$ opam install memtrace memtrace_viewer
Put this at the start of the program to enable it:
The flame graph in the middle shows functions scaled by the amount of memory they allocated.
Initially it showed two groups, one for the warm-up request and one for the 10 runs.
To simplify the display, I used the filter panel (on the left) to show only allocations after the 2 second warm-up.
We can immediately see that OpamVersionCompare.compare is the source of most memory use.
Focusing on that function shows that it performed 54.1% of all allocations.
The display now shows allocations performed within it above it (in green),
and all the places it's called from in blue below:
The bulk of the allocations are coming from this loop:
12345678910
(* [skip_while_from i f w m] yields the index of the leftmost character * in the string [s], starting from [i], and ending at [m], that does * not satisfy the predicate [f], or [length w] if no such index exists. *)letskip_while_fromifwm=letrecloopi=ifi=mthenielseiffw.[i]thenloop(i+1)elseiinloopiletskip_zerosxxixl=skip_while_fromxi(func->c='0')xxl
It's used when processing a version like 1.2.3 to skip any leading "0" characters
(so that would compare equal to 1.02.3).
The loop function refers to other variables (such as f) from its context,
and so OCaml allocates a closure on the heap to hold these variables.
Even though these allocations are small, we have to do it for every component of every version.
And we compare versions a lot:
for every version of a package that says it requires e.g. libfoo { >= "1.2" },
we have to check the formula against every version of libfoo.
The solution is rather simple (and shorter than the original!):
Removing the other allocations from compare too reduces total memory allocations
from 21.8G to 9.6G!
The processes benchmark got about 14% faster, while the domains one was 23% faster:
A nice optimisation,
but using domains is still nowhere close to even the original version with separate processes.
magic-trace
The traces above show the solver taking a long time for all domains to enter the stw_api_barrier phase.
What was the slow domain doing to cause that?
magic-trace lets us tell it when to save the ring buffer and we can use this to get detailed information.
Tracing multiple threads with magic-trace doesn't seem to work well
(each thread gets a very small buffer, they don't stop at quite the same time, and triggers don't work)
so I find it's better to trace just one thread.
I modified the OCaml runtime so that the leader (the domain requesting the GC) records the time.
As each domain enters stw_api_barrier it checks how late it is and calls a function to print a warning if it's above a threshold.
Then I attached magic-trace to one of the worker threads and told it to save a sample when that function got called:
In the example above,
magic-trace saved about 7ms of the history of a domain up to the point where it entered stw_api_barrier.
The first few ms show the solver working normally.
Then it needs to do a minor GC and tries to become the leader.
But another domain has the lock and so it spins, calling handle_incoming 293,711 times in a loop for 2.5ms.
I had a look at the code in the OCaml runtime.
When a domain wants to perform a minor GC, the steps are:
Acquire all_domains_lock.
Populate the stw_request global.
Interrupt all domains.
Release all_domains_lock.
Wait for all domains to get the interrupt.
Mark self as ready, allowing GC work to start.
Do minor GC.
The last domain to finish its minor GC signals all_domains_cond and everyone resumes.
I added some extra event reporting to the GC, showing when a domain is trying to perform a GC (try),
when the leader is signalling other domains (signal), and when a domain is sleeping waiting for something (sleep).
Here's what that looks like (in some places):
The top domain finished its minor collection quickly (as it's mostly idle and had nothing to do),
and started waiting for the other domains to finish. For some reason, this sleep call took 3ms to run.
The other domains resume work. One by one, they fill their minor heaps and try to start a GC.
They can't start a new GC, as the old one hasn't completely finished yet, so they spin.
Eventually the top domain wakes up and finishes the previous STW section.
One of the other domains immediately starts a new minor GC and the pattern repeats.
These try events seem useful;
the program is spending much more time stuck in GC than the original traces indicated!
One obvious improvement here would be for idle domains to opt out of GC.
Another would be to tell the kernel when to wake instead of using sleeps —
and I see there's a PR already:
OS-based Synchronisation for Stop-the-World Sections.
Another possibility would be to let domains perform minor GCs independently.
The OCaml developers did make a version that worked that way,
but it requires changes to all C code that uses the OCaml APIs,
since a value in another domain's minor heap might move while it's running.
Finally, I wonder if the code could be simplified a bit using a compare-and-set instead of taking a lock to become leader.
That would eliminate the try state, where a domain knows another domain is the leader, but doesn't know what it wants to do.
It's also strange that there's a state where
the top domain has finished its critical section and allowed the other domains to resume,
but is not quite finished enough to let a new GC start.
We can work around this problem by having the main domain do work too.
That could be a problem for interactive applications (where the main domain is running the UI and needs to respond fast),
but it should be OK for the solver service.
This was about 15% faster on my machine, but appeared to have no effect on the ARM server.
Lesson: get traces on the target machine!
Tuning GC parameters
Another way to reduce the synchronisation overhead of minor GCs is to make them less frequent.
We can do that by increasing the size of the minor heap,
doing a few long GCs rather than many short ones.
The size is controlled by the setting e.g. OCAMLRUNPARAM=s=8192k.
On my machine, this actually makes things slower, but it's about 18% faster on the ARM server with 80 domains.
Here are the first few domains (from a total of 24) on the ARM server with different minor heap sizes
(both are showing 1s of execution):
Also, there's still a lot of variation between the time each domain spends doing GC
(despite the fact that they're all running exactly the same task), so they still end up waiting a lot.
Simplifying further
This is all still pretty odd, though.
We're getting small performance increases, but still nothing like when forking.
Can the test-case be simplified further?
Yes, it turns out!
This simple function takes much longer to run when using domains, compared to forking!
ref () allocates a small block (2 words, including the header) on the minor heap.
opaque_identity is to make sure the compiler doesn't optimise this pointless allocation away.
The domains all start to fill their minor heaps. One fills it and triggers a minor GC.
The triggering domain sets an indicator in each domain saying a GC is due.
None of the domains is sleeping, so the OS isn't involved in any wake-ups here.
The other domains check the indicator on their next allocation,
which happens immediately since that's all they're doing.
The GCs all proceed quickly, since there's nothing to scan and nothing to promote
(except possibly the current single allocation).
They all resume quickly and continue.
So ideally the lines would be flat.
In practice, we may hit physical limits due to memory bandwidth, CPU temperature or kernel limitations;
I assume this is why the "Processes" time starts to rise eventually.
But it looks like this minor slow-down causes knock-on effects in the "Domains" case.
If I remove the allocation, then the domains and processes versions take the same amount of time.
perf sched
perf sched record records kernel scheduling events, allowing it to show what is running on each CPU at all times.
perf sched timehist displays a report:
$ sudo perf sched record -k CLOCK_MONOTONIC
^C
$ sudo perf sched timehist
time cpu task name wait time sch delay run time
[tid/pid] (msec) (msec) (msec)
--------------- ------ ------------------------------ --------- --------- ---------
185296.715345 [0000] sway[175042] 1.694 0.025 0.775
185296.716024 [0002] crosvm_vcpu2[178276/178217] 0.012 0.000 2.957
185296.717031 [0003] main.exe[196519] 0.006 0.000 4.004
185296.717044 [0003] rcu_preempt[18] 4.004 0.015 0.012
185296.717260 [0001] main.exe[196526] 1.760 0.000 2.633
185296.717455 [0001] crosvm_vcpu1[193502/193445] 63.809 0.015 0.194
...
The first line here shows that sway needed to wait for 1.694 ms for some reason (possibly a sleep),
and then once it was due to resume, had to wait a further 0.025 ms for CPU 0 to be free. It then ran for 0.775 ms.
I decided to use perf sched to find out what the system was doing when a domain failed to respond quickly.
To make the output easier to read, I hacked eio-trace to display it on the traces.
perf script -g python will generate a skeleton Python script that can format all the events found in the perf.data file,
and I used that to convert the output to CSV.
To correlate OCaml domains with Linux threads, I also modified OCaml to report the thread ID (TID) for each new domain
(it was previously reporting the PID instead for some reason).
Here's a trace of the simple allocator from the previous section:
Note: the colour of stw_api_barrier has changed: previously eio-trace coloured it yellow to indicate sleeping,
but now we have the individual sleep events we can see exactly which part of it was sleeping.
The horizontal green bars show when each domain was running on the CPU.
Here, we see that most of the domains ran until they called sleep.
When the sleep timeout expires, the thread is ready to run again and goes on the run-queue.
Time spent waiting on the queue is shown with a black bar.
When switching to or from another process, the process name is shown.
Here we can see that crosvm_vcpu6 interrupted one of our domains, making it late to respond to the GC request.
Here we see another odd feature of the protocol: even though the late domain was the last to be ready,
it wasn't able to start its GC even then, because only the leader is allowed to say when everyone is ready.
Several domains wake after the late one is ready and have to go back to sleep again.
The diagram also shows when Linux migrated our OCaml domains between CPUs.
For example:
The bottom domain was initially running on CPU 0.
After sleeping briefly, it spent a while waiting to resume and Linux moved it to CPU 6 (the leader domain, which was idle then).
Once there, the bottom domain slept briefly again, and again was slow to wake, getting moved to CPU 7.
The bottom domain's sleep finished a while ago, and it's been stuck on the queue because it's on the same CPU as another domain.
All the other domains are spinning, trying to become the leader for the next minor GC.
Eventually, Linux preempts the 5th domain from the top to run the bottom domain
(the vertical green line indicates a switch between domains in the same process).
The bottom domain finishes the previous minor GC, allowing the 3rd from top to start a new one.
The new GC is delayed because the 5th domain is now waiting while the bottom domain spins.
Eventually the bottom domain sleeps, allowing 5 to join and the GC starts.
I tried using the processor package to pin each domain to a different CPU.
That cleaned up the traces a fair bit, but didn't make much difference to the runtime on my machine.
I also tried using chrt to run the program as a high-priority "real-time" task,
which also didn't seem to help.
I wrote a bpftrace script to report if one of our domains was ready to resume and the scheduler instead ran something else.
That showed various things.
Often Linux was migrating something else out of the way and we had to wait for that,
but there were also some kernel tasks that seemed to be even higher priority, such as GPU drivers or uring workers.
I suspect to make this work you'd need to set the affinity of all the other processes to keep them away from the cores being used
(but that wouldn't work in this example because I'm using all of them!).
Come to think of it, running a CPU intensive task on every CPU at realtime priority was a dumb idea;
had it worked I wouldn't have been able to do anything else with the computer!
olly
Exploring the scheduler behaviour was interesting, and might be needed for latency-sensitive tasks,
but how often do migrations and delays really cause trouble?
The slow GCs are interesting, but there are also sections like this where everything is going smoothly,
and minor GCs take less than 4 microseconds:
$ olly gc-stats './_build/default/stress/stress.exe --count=6 --internal-workers=24'
...
Solved 144 requests in 25.44s (0.18s/iter) (5.66 solves/s)
Execution times:
Wall time (s): 28.17
CPU time (s): 1.66
GC time (s): 169.88
GC overhead (% of CPU time): 10223.84%
GC time per domain (s):
Domain0: 0.47
Domain1: 9.34
Domain2: 6.90
Domain3: 6.97
Domain4: 6.68
Domain5: 6.85
Domain6: 6.59
...
10223.84% GC overhead sounds like a lot but I think this is a misleading, for a few reasons:
The CPU time looks wrong. time reports about 6 minutes, which sounds more likely.
GC time (as we've seen) includes time spent sleeping, while CPU time doesn't.
It doesn't include time spent trying to become a GC leader.
To double-check, I modified eio-trace to report GC statistics for a saved trace:
Solved 144 requests in 26.84s (0.19s/iter) (5.36 solves/s)
...
$ eio-trace gc-stats trace.fxt
./trace.fxt:
Ring GC/s App/s Total/s %GC
0 10.255 19.376 29.631 34.61
1 7.986 10.201 18.186 43.91
2 8.195 10.648 18.843 43.49
3 9.521 14.398 23.919 39.81
4 9.775 16.537 26.311 37.15
5 8.084 10.635 18.719 43.19
6 7.977 10.356 18.333 43.51
...
24 7.920 10.802 18.722 42.30
All 213.332 308.578 521.910 40.88
Note: all times are wall-clock and so include time spent blocking.
It ran slightly slower under eio-trace, perhaps because recording a trace file is more work than maintaining some counters,
but it's similar.
So this indicates that with 24 domains GC is taking about 40% of the total time (including time spent sleeping).
But something doesn't add up, on my machine at least:
With processes, the simple allocator test's main process spends 2% of its time in GC and takes 2.4s to run.
With domains, the main domain spends 20% of its time in GC and takes 8.2s.
Even if that 20% were removed completely, it should only save 20% of the 8.2s.
So with domains, the code must be running more slowly even when it's not in the GC.
magic-trace on the simple allocator
I tried running magic-trace to see what it was doing outside of the GC.
Since it wasn't calling any functions, it didn't show anything, but we can fix that:
Here we do blocks of 100 allocations in a function called foo.
The annotations are to ensure the compiler doesn't inline it.
The trace was surprisingly variable!
The code starts by (pointlessly) checking if 1 > 100 in case it can skip the whole loop.
After being disappointed, it:
Decreases %r15 (young_ptr) by 0x10 (two words).
Checks if that's now below young_limit, calling caml_call_gc if so to clear the minor heap.
Writes 0x400 to the first newly-allocated word (the block header, indicating 1 word of data).
Writes 1 to the second word, which represents ().
Increments the loop counter and loops, unless we're at the end.
Returns ().
Looks like we spent most of the time (77%) writing the block, which makes sense.
Reading young_limit took 13% of the time, which seems reasonable too.
If there was contention between domains, we'd expect to see it here.
The output looked similar whether using domains or processes.
perf c2c
To double-check, I also tried perf c2c.
This reports on cache-to-cache transfers, where two CPUs are accessing the same memory,
which requires the processors to communicate and is therefore relatively slow.
This shows a list of cache lines (memory addresses) and how often we loaded from a modified address.
There's a lot of information here and I don't understand most of it.
But I think the above is saying that address 0x7f90b4002b80 (young_limit, at offset 0) was accessed by these places across domains:
main.ml:7 (ref ()) checks against young_limit to see if we need to call into the GC.
domain.c:318 sets the limit to UINTNAT_MAX to signal that another domain wants a GC.
domain.c:1658 sets it back to young_trigger after being signalled.
The same cacheline was also accessed at offset 8, which contains young_ptr (address of last allocation):
minor_gc.c:622 sets young_ptr to young_end after a GC.
minor_gc.c:874 adjusts young_ptr to re-do the allocation that triggered the GC.
This indicates false sharing: young_ptr only gets accessed from one domain but it's in the same cache line as young_limit.
The main thing is that the counts are all very low, indicating that this doesn't happen often.
I tried adding an incr x on a global variable in the loop, and got some more operations reported.
But using Atomic.incr massively increased the number of records:
perf stat shows statistics about a process.
I ran it with -I 1000 to collect one-second samples.
Here are two samples from the test case on my machine,
one when it was running processes and one while it was using domains:
We're doing a lot more context switches with domains, as expected due to the sleeps,
and we're executing many fewer instructions, which isn't surprising.
Reporting the counts for individual CPUs gets more interesting though:
In the domains case all CPUs are doing roughly the same amount of work.
But when running separate processes the CPUs differ wildly!
Over the last 1-second interval, for example, CPU5 executed 5.3 times as many instructions as CPU4.
And indeed, some of the test processes are finishing much sooner than the others,
even though they all do the same work.
Setting /sys/devices/system/cpu/cpufreq/policy*/energy_performance_preference to performance didn't make it faster,
but setting it to power (power-saving mode) did make the processes benchmark much slower,
while having little effect on the domains case!
So I think what's happening here with separate processes is that
the CPU is boosting the performance of one or two cores at a time,
allowing them to make lots of progress.
But with domains this doesn't happen, either because no domain runs long enough before sleeping to trigger the boost,
or because as soon as it does it needs to stop and wait for the other domains for a GC and loses it.
Conclusions
The main profiling and tracing tools used were:
perf to take samples of CPU use, find hot functions and hot instructions within them,
record process scheduling, look at hardware counters, and find sources of cache contention.
statmemprof to find the source of allocations.
eio-trace to visualise GC events and as a generic canvas for custom visualisations.
magic-trace to see very detailed traces of recent activity when something goes wrong.
olly to report on GC statistics.
bpftrace for quick experiments about kernel behaviour.
offcputime to see why a process is sleeping.
I think OCaml 5's runtime events tracing was the star of the show here, making it much easier to see what was going on with GC,
especially in combination with perf sched.
statmemprof is also an essential tool for OCaml, and I'll be very glad to get it back with OCaml 5.3.
I think I need to investigate perf more; I'd never used many of these features before.
Though it is important to use it with offcputime etc to check you're not missing samples due to sleeping.
Unlike the previous post's example, where the cause was pretty obvious and led to a massive easy speed-up,
this one took a lot of investigation and revealed several problems, none of which seem very easy to fix.
I'm also a lot less confident that I really understand what's happening here, but here is a summary of my current guess:
OCaml applications typically allocate lots of short-lived values.
With a single domain this isn't much of a problem; minor GCs are fast.
With multiple domains however we have to wait for every domain to enter the
GC, and then wait again for them all to exit.
This can be very fast (4 microseconds or so per GC),
but if one domain is late due to OS scheduling then it can be much longer
(several ms in some cases).
When a domain needs to wait for another it spins for a bit and then sleeps.
If the other domain runs on the same CPU then spinning delays it from running.
On the other hand, sleeping introduces longer delays and can cause the CPU to slow down.
Idle domains are currently expensive.
An idle domain requires a syscall to wake it, and often causes all the other domains to sleep waiting for it.
When the idle domain does wake, it still can't start the GC and has to wait again for the leader.
If the leader gets suspended while holding the lock, all the other domains will spin waiting for it (without ever sleeping).
This time isn't accounted for in the GC events reported by OCaml 5.2.
Since the sleeping mechanism will be changing in OCaml 5.3,
it would probably be worthwhile checking how that performs too.
I think there are some opportunities to improve the GC, such as letting idle domains opt out of GC after one collection,
and it looks like there are opportunities to reduce the amount of synchronisation done
(e.g. by letting late arrivers start the GC without having to wait for the leader,
or using a lock-free algorithm for becoming leader).
For the solver, it would be good to try experimenting with CPU affinity to keep a subset of the 160 cores reserved for the solver.
Increasing the minor heap size and doing work in the main domain should also reduce the overhead of GC,
and improving the version compare function in the opam library would greatly reduce the need for it.
And if my goal was really to make it fast (rather than to improve multicore OCaml and its tooling)
then I'd probably switch it back to using processes.
Finally, it was really useful that both of these blog posts examined performance regressions,
so I knew it must be possible to go faster.
Without a good idea of how fast something should be, it's easy to give up too early.
Anyway, I hope you found some useful new tool in these posts!
Update 2024-08-22
I reported above that using chrt to make the process high priority didn't help on my machine.
It also didn't help on the ARM server using the real service.
However, it did help a lot when running the simplified version of the solver on the ARM server!
Some more investigation showed that the real service had an additional problem:
it spawned a git log subprocess after every solve, and this was causing all the domains to pause
for about 50ms during the fork operation.
Use OCaml code to find the oldest commit
eliminated the problem (and is also faster, as it can cache the history, although that doesn't matter much).
Here are some benchmarks of various combinations of fixes:
Processes, sched-other is the original version using multiple processes.
Oddly, using sched-rr to make it high-priority actually slows this one down!
Domains, sched-other is the original version using domains.
The results are noisy but using sched-rr doesn't seem to help.
ocaml-git indicates that the git log subprocess has been replaced by OCaml code.
And with that, the new multicore solver service is finally faster than the old process-based one!
]]>OCaml 5 performance problems2024-07-22T10:00:00+00:00https://roscidus.com/blog/blog/2024/07/22/performanceLinux and OCaml provide a huge range of tools for investigating performance problems.
In this post I try using some of them to understand a network performance problem.
In part 2, I'll investigate a problem in a CPU-intensive multicore program.
While porting capnp-rpc from Lwt to Eio,
to take advantage of OCaml 5's new effects system,
I tried running the benchmark to see if it got any faster:
$ ./echo_bench.exe
echo_bench.exe: [INFO] rate = 44933.359573 # The old Lwt version
echo_bench.exe: [INFO] rate = 511.963565 # The (buggy) Eio version
The benchmark records the number of echo RPCs per second.
Clearly, something is very wrong here!
In fact, the new version was so slow I had to reduce the number of iterations so it would finish.
time
The old time command can immediately give us a hint:
(many shells provide their own time built-in with different output formats; I'm using /usr/bin/time here)
time's output shows time spent in user-mode (running the application's code on the CPU),
time spent in the kernel, and the total wall-clock time.
Both versions ran for around 2 seconds (doing a different number of iterations),
but the Lwt version was using the CPU 98% of the time, while the Eio version was mostly sleeping.
eio-trace
eio-trace can be used to see what an Eio program is doing.
Tracing is always available (you don't need to recompile the program to get it).
$ eio-trace run -- ./echo_bench.exe
eio-trace run runs the command and displays the trace in a window.
You can also use eio-trace record to save a trace and examine it later.
I've zoomed the image to show the first four iterations.
The first is so quick it's not really visible, but the next three take about 40ms each.
The yellow regions labelled "suspend-domain" show when the program is sleeping, waiting for an event from Linux.
Each horizontal bar is a fiber (a light-weight thread). From top to bottom they are:
Three rows for the test client:
The main application fiber performing the RPC call (mostly awaiting responses).
The network's write fiber, sending outgoing messages (mostly waiting for something to send).
The network's read fiber, reading incoming messages (mostly waiting to something to read).
Four rows for the server:
A loop accepting new incoming TCP connections.
A short-lived fiber that accepts the new connection, then short-lived fibers each handling one request.
The server's network write fiber.
The server's network read fiber.
One fiber owned by Eio itself (used to wake up the event loop in some situations).
This trace immediately raises a couple of questions:
Why is there a 40ms delay in each iteration of the test loop?
Why does the program briefly wake up in the middle of the first delay, do nothing, and return to sleep?
(notice the extra "suspend-domain" at the top)
Zooming in on a section between the delays, let's see what it's doing when it's not sleeping:
After a 40ms delay, the server's read fiber receives the next request (the running fiber is shown in green).
The read fiber spawns a fiber to handle the request, which finishes quickly, starts the next read,
and then the write fiber transmits the reply.
The client's read fiber gets the reply, the write fiber outputs a message, then the application fiber runs
and another message is sent.
The server reads something (presumably the first message, though it happens after the client had sent both),
then there is another long 40ms delay, then (far off the right of the image) the pattern repeats.
To get more context in the trace,
I configured
the logging library to write the (existing) debug-level log messages to the trace buffer too:
Log messages tend to be a bit long for the trace display, so they overlap and you have to zoom right in to read them,
but they do help navigate.
With this, I can see that the first client write is "Send finish" and the second is "Calling Echo.ping".
Looks like we're not buffering the output, so it's doing two separate writes rather than combining them.
That's a little inefficient, and if you've done much network programming,
you also probably already know why this might cause a 40ms delay,
but let's pretend we don't know so we can play with a few more tools...
strace
strace can be used to trace interactions between applications and the Linux kernel
(-tt -T shows when each call was started and how long it took):
On Linux, Eio defaults to using the io_uring mechanism for submitting work to the kernel.
io_uring_enter(4, 2, 0, 0, NULL, 8) = 2 means we asked to submit 2 new operations to the ring on FD 4,
and the kernel accepted them.
The call at 11:38:58.079637 timed out after 19ms.
It then woke up some futexes and then waited again, getting woken up after a further 21ms (for a total of 40ms).
Futexes are used to coordinate between system threads.
strace -f will follow all spawned threads (and processes), not just the main one:
The benchmark connects to "127.0.0.1" and Eio uses getaddrinfo to look up addresses (we can't use uring for this).
Since getaddrinfo can block for a long time, Eio creates a new system thread (pid 48451) to handle it
(we can guess this thread is doing name resolution because we see it read resolv.conf).
As creating system threads is a little slow, Eio keeps the thread around for a bit after it finishes in case it's needed again.
The timeout is when Eio decides that the thread isn't needed any longer and asks it to exit.
So this isn't relevant to our problem (and only happens on the first 40ms delay, since we don't look up any further addresses).
However, strace doesn't tell us what the uring operations were, or their return values.
One option is to switch to the posix backend (which is the default on Unix systems).
In fact, it's a good idea with any performance problem to check if it still happens with a different backend:
(to reduce clutter, I removed calls that returned EAGAIN and ppoll calls that returned 0 ready descriptors)
The problem still occurs, and now we can see the two writes:
The client writes 40 bytes to its end of the socket (FD 7), after which the server's end (FD 8) is ready for reading (revents=POLLIN).
The client then writes another 136 bytes.
The server reads 40 bytes and then uses ppoll to await further data.
After 39ms, ppoll says FD 8 is now ready, and the server reads the other 136 bytes.
bpftrace
Alternatively, we can trace uring operations using bpftrace.
bpftrace is a little scripting language similar to awk,
except that instead of editing a stream of characters,
it live-patches the running Linux kernel.
Apparently this is safe to run in production
(and I haven't managed to crash my kernel with it yet).
In this output, the order is slightly different:
we see the server's read get the 40 bytes before the client sends the rest,
but we still see the 40ms delay between the completion of the second write and the corresponding read.
The change in order is because we're seeing when the kernel knew the read was complete,
not when the application found out about it.
tcpdump
An obvious step with any networking problem is the look at the packets going over the network.
tcpdump can be used to capture packets, which can be displayed on the console or in a GUI with wireshark.
$ sudo tcpdump -n -ttttt -i lo
...
...041330 IP ...37640 > ...7000: Flags [P.], ..., length 40
...081975 IP ...7000 > ...37640: Flags [.], ..., length 0
...082005 IP ...37640 > ...7000: Flags [P.], ..., length 136
...082071 IP ...7000 > ...37640: Flags [.], ..., length 0
Here we see the client (on port 37640) sending 40 bytes to the server (port 7000),
and the server replying with an ACK (with no payload) 40ms later.
After getting the ACK, the client socket sends the remaining 136 bytes.
Here we can see that while the application made the two writes in quick succession,
TCP waited before sending the second one.
Searching for "delayed ack 40ms" will turn up an explanation.
ss
ss displays socket statistics.
ss -tin shows all TCP sockets (-t) with internals (-i):
$ ss -tin 'sport = 7000 or dport = 7000'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 127.0.0.1:7000 127.0.0.1:56224
ato:40 lastsnd:34 lastrcv:34 lastack:34
ESTAB 0 176 127.0.0.1:56224 127.0.0.1:7000
ato:40 lastsnd:34 lastrcv:34 lastack:34 unacked:1 notsent:136
There's a lot of output here; I've removed the irrelevant bits.
ato:40 says there's a 40ms timeout for "delay ack mode".
lastsnd, etc, say that nothing had happened for 34ms when this information was collected.
unacked and notsent aren't documented in the man-page,
but I guess it means that the client (now port 56224) is waiting for 1 packet to be ack'd and has 136 bytes waiting until then.
The client socket still has both messages (176 bytes total) in its queue;
it can't forget about the first message until the server confirms receiving it,
since the client might need to send it again if it got lost.
This doesn't quite lead us to the solution, though.
offwaketime
offwaketime records why a program stopped using the CPU, and what caused it to resume:
offwaketime records a stack-trace when a process is suspended (shown at the bottom and going up)
and pairs it with the stack-trace of the thread that caused it to be resumed (shown above it and going down).
The taller column on the right shows Eio being woken up due to TCP data being received from the network,
confirming that it was the TCP ACK that got things going again.
The shorter column on the left was unexpected, and the [UNKNOWN] in the stack is annoying
(probably C code compiled without frame pointers).
gdb gets a better stack trace.
It turned out to be OCaml's tick thread, which wakes every 50ms to prevent one sys-thread from hogging the CPU:
Having multiple threads shown on the same diagram is a bit confusing.
I should probably have used -t to focus only on the main one.
Also, note that when using profiling tools that record the OCaml stack,
it's useful to compile with frame pointers enabled.
To install e.g. OCaml 5.2.0 with frame pointers enabled, use:
magic-trace allows capturing a short trace of everything the CPUs were doing just before some event.
It uses Intel Processor Trace to have the CPU record all control flow changes (calls, branches, etc) to a ring-buffer,
with fairly low overhead (2% to 10%, due to extra memory bandwidth needed).
When something interesting happens, we save the buffer and use it to reconstruct the recent history.
Normally we'd need to set up a trigger to grab the buffer at the right moment,
but since this program is mostly idle it doesn't record much
and I just attached at a random point and immediately pressed Ctrl-C to grab a snapshot and detach:
The output is a bit messed up because magic-trace doesn't understand that there are multiple OCaml fibers here,
each with their own stack. It also doesn't seem to know that exceptions unwind the stack.
In each 40ms column, Eio_posix.Flow.single_read (3rd line from top) tried to do a read
with readv, which got EAGAIN and called Sched.next to switch to the next fiber.
Since there was nothing left to run, the Eio scheduler called ppoll.
Linux didn't have anything ready for this process,
and called the schedule kernel function to switch to another process.
I recorded an eio-trace at the same time, to see the bigger picture.
Here's the eio-trace zoomed in to show the two client writes (just before the 40ms wait),
with the relevant bits of the magic-trace stack pasted below them:
We can see the OCaml code calling writev, entering the kernel, tcp_write_xmit being called to handle it,
writing the IP packet to the network and then, because this is the loopback interface, the network receive logic
handling the packet too.
The second call is much shorter; tcp_write_xmit returns quickly without sending anything.
Note: I used the eio_posix backend here so it's easier to correlate the kernel operations to the application calls
(uring queues them up and runs them later).
The uring-trace project should make this easier in future, but doesn't integrate with eio-trace yet.
Zooming in further, it's easy to see the difference between the two calls to tcp_write_xmit:
Having identified a load of interesting events
I wrote summary-posix.bt, a bpftrace script to summarise them.
This includes log messages written by the application (by tracing write calls to stderr),
reads and writes on the sockets,
and various probed kernel functions seen in the magic-trace output and when reading the kernel source.
The output is specialised to this application (for example, TCP segments sent to port 7000
are displayed as "to server", while others are "to client").
I think this is a useful way to double-check my understanding, and any fix:
$ sudo bpftrace summary-posix.bt
[...]
844ms: server: got ping request; sending reply
844ms: server reads from socket (EAGAIN)
844ms: server: writev(96 bytes)
844ms: tcp_write_xmit (to client, nagle-on, packets_out=0)
844ms: tcp_v4_send_check: sending 96 bytes to client
844ms: tcp_v4_rcv: got 96 bytes
844ms: timer_start (tcp_delack_timer, 40 ms)
844ms: client reads 96 bytes from socket
844ms: client: enqueue finish message
844ms: client: enqueue ping call
844ms: client reads from socket (EAGAIN)
844ms: client: writev(40 bytes)
844ms: tcp_write_xmit (to server, nagle-on, packets_out=0)
844ms: tcp_v4_send_check: sending 40 bytes to server
845ms: tcp_v4_rcv: got 40 bytes
845ms: timer_start (tcp_delack_timer, 40 ms)
845ms: client: writev(136 bytes)
845ms: tcp_write_xmit (to server, nagle-on, packets_out=1)
845ms: server reads 40 bytes from socket
845ms: server reads from socket (EAGAIN)
885ms: tcp_delack_timer_handler (ACK to client)
885ms: tcp_v4_send_check: sending 0 bytes to client
885ms: tcp_delack_timer_handler (ACK to server)
885ms: tcp_v4_rcv: got 0 bytes
885ms: tcp_write_xmit (to server, nagle-on, packets_out=0)
885ms: tcp_v4_send_check: sending 136 bytes to server
The server replies to a ping request, sending a 96 byte reply.
Nagle is on, but nothing is awaiting an ACK (packets_out=0) so it gets sent immediately.
The client receives the data. It starts a 40ms timer to send an ACK for it.
The client enqueues a "finish" message, followed by another "ping" request.
The client's write fiber sends the 40 byte "finish" message.
Nothing is awaiting an ACK (packets_out=0) so the kernel sends it immediately.
The client sends the 136 byte ping request. As the last message hasn't been ACK'd, it isn't sent yet.
The server receives the 40 byte finish message.
40ms pass. The server's delayed ACK timer fires and it sends the ACK to the client.
The client's delayed ACK timer fires, but there's nothing to do (it sent the ACK with the "finish").
The client socket gets the ACK for its "finish" message and sends the delayed ping request.
Fixing it
The problem seemed clear: while porting from Lwt to Eio I'd lost the output buffering.
So I looked at the Lwt code to see how it did it and... it doesn't! So how was it working?
As I did with Eio, I set the Lwt benchmark's concurrency to 1 to simplify it for tracing,
and discovered that Lwt with 1 client thread has exactly the same problem as the Eio version.
Well, that's embarrassing!
But why is Lwt fast with 12 client threads?
With only minor changes (e.g. write vs writev), the summary script above also worked for tracing the Lwt version.
With 1 or 2 client threads, Lwt is slow, but with 3 it's fairly fast.
The delay only happens if the client sends a "finish" message when the server has no replies queued up
(otherwise the finish message unblocks the replies, which carry the ACK to the client immediately).
So, it works mostly by fluke!
Lwt just happens to schedule the threads in such a way that Nagle's algorithm mostly doesn't trigger with 12 concurrent requests.
Anyway, adding buffering to the Eio version fixed the problem:
An interesting thing to notice here is that not only did the long delay go away,
but the CPU operations while it was active were faster too!
I think the reason is that the CPU goes into power-saving mode during the long delays.
cpupower monitor shows my CPUs running at around 1 GHz with the old code and
around 4.7 GHz when running the new version.
Here are the results for the fixed version:
$ ./echo_bench.exe
echo_bench.exe: [INFO] rate = 44425.962625 # The old Lwt version
echo_bench.exe: [INFO] rate = 59653.451934 # The fixed Eio version
60k RPC requests per second doesn't seem that impressive, but at least it's faster than the old version,
which is good enough for now! There's clearly scope for improvement here (for example, the buffering I
added is quite inefficient, making two extra copies of every message, as the framing library copies it from
a cstruct to a string, and then I have to copy the string back to a cstruct for the kernel).
Conclusions
There are lots of great tools available to help understand why something is running slowly (or misbehaving),
and since programmers usually don't have much time for profiling,
a little investigation will often turn up something interesting!
Even when things are working correctly, these tools are a good way to learn more about how things work.
time will quickly tell you if the program is taking lots of time in application code, in the kernel, or just sleeping.
If the problem is sleeping, offcputime and offwaketime can tell you why it was waiting and what woke it in the end.
My own eio-trace tool will give a quick visual overview of what an Eio application is doing.
strace is great for tracing interactions between applications and the kernel,
but it doesn't help much when the application is using uring.
To fix that, you can either switch to the eio_posix backend or use bpftrace with the uring tracepoints.
tcpdump, wireshark and ss are all useful to examine network problems specifically.
I've found bpftrace to be really useful for all kinds of tasks.
Being able to write quick one-liners or short scripts gives it great flexibility.
Since the scripts run in the kernel you can also filter and aggregate data efficiently
without having to pass it all to userspace, and you can examine any kernel data structures.
We didn't need that here because the program was running so slowly, but it's great for many problems.
In addition to using well-defined tracepoints,
it can also probe any (non-inlined) function in the kernel or the application.
I also think using it to create a "summary script" to confirm a problem and its solution seems useful,
though this is the first time I've tried doing that.
magic-trace is great for getting really detailed function-by-function tracing through the application and kernel.
Its ability to report the last few ms of activity after you notice a problem is extremely useful
(though not needed in this example).
It would be really useful if you could trigger magic-trace from a bpftrace script, but I didn't see a way to do that.
However, it was surprisingly difficult to get any of the tools to point directly
at the combination of Nagle's algorithm with delayed ACKs as the cause of this common problem!
This post was mainly focused on what was happening in the kernel.
In part 2, I'll investigate a CPU-intensive problem instead.
]]>Lambda Capabilities2023-04-26T10:00:00+00:00https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities"Is this software safe?" is a question software engineers should be able to answer,
but doing so can be difficult.
Capabilities offer an elegant solution, but seem to be little known among functional programmers.
This post is an introduction to capabilities in the context of ordinary programming
(using plain functions, in the style of the lambda calculus).
Even if you're not interested in security,
capabilities provide a useful way to understand programs;
when trying to track down buggy behaviour,
it's very useful to know that some component couldn't have been the problem.
We have some application (for example, a web-server) that we want to run.
The application is many thousands of lines long and depends on dozens of third-party libraries,
which get updated on a regular basis.
I would like to be able to check, quickly and easily, that the application cannot do any of these things:
Delete my files.
Append a line to my ~/.ssh/authorized_keys file.
Act as a relay, allowing remote machines to attack other computers on my local network.
Send telemetry to a third-party.
Anything else bad that I forget to think about.
For example, here are some of the OCaml packages I use just to generate this blog:
Having to read every line of every version of each of these packages in order to decide whether it's safe
to generate the blog clearly isn't practical.
I'll start by looking at traditional solutions to this problem, using e.g. containers or VMs,
and then show how to do better using capabilities.
Option 1: Security as a separate concern
A common approach to access control treats securing software as a separate activity to writing it.
Programmers write (insecure) software, and a security team writes a policy saying what it can do.
Examples include firewalls, containers, virtual machines, seccomp policies, SELinux and AppArmor.
The great advantage of these schemes is that security can be applied after the software is written, treating it as a black box.
However, it comes with many problems:
Confused deputy problem
Some actions are OK for one use but not for another.
For example, if the client of a web-server requests https://example.com/../../etc/httpd/server-key.pem
then we don't want the server to read this file and send it to them.
But the server does need to read this file for other reasons, so the policy must allow it.
Coarse-grained controls
All the modules making up the program are treated the same way,
even though you probably trust some more than others.
For example, we might trust the TLS implementation with the server's private key, but not the templating engine,
and I know the modules I wrote myself are not malicious.
Even well-typed programs go wrong
Programming in a language with static types is supposed to ensure that if the program compiles then it won't crash.
But the security policy can cause the program to fail even though it passed the compiler's checks.
For example, the server might sometimes need to send an email notification.
If it didn't do that while the security policy was being written, then that will be blocked.
Or perhaps the web-server didn't even have a notification system when the policy was written,
but has since been updated.
Policy language limitations
The security configuration is written in a new language, which must be learned.
It's usually not worth learning this just for one program,
so the people who write the program struggle to write the policy.
Also, the policy language often cannot express the desired policy,
since it may depend on concepts unique to the program
(e.g. controlling access based on a web-app user's ID, rather than local Unix user ID).
All of the above problems stem from trying to separate security from the code.
If the code were fully correct, we wouldn't need the security layer.
Checking that code is fully correct is hard,
but maybe there are easy ways to check automatically that it does at least satisfy our security requirements...
Option 2: Purity
One way to prevent programs from performing unwanted actions is to prevent all actions.
In pure functional languages, such as Haskell, the only way to interact with the outside world is to return the action you want to perform from main. For example:
12345
f::Int->Stringfx=...main::IO()main=putStr(f42)
Even if we don't look at the code of f, we can be sure it only returns a String and performs no other actions
(assuming Safe Haskell is being used).
Assuming we trust putStr, we can be sure this program will only output a string to stdout and not perform any other actions.
However, writing only pure code is quite limiting. Also, we still need to audit all IO code.
Option 3: Capabilities
Consider this code (written in a small OCaml-like functional language, where ref n allocates a new memory location
initially containing n, and !x reads the current value of x):
Can we be sure that the assert won't fail, without knowing the definition of f?
Assuming the language doesn't provide unsafe backdoors (such as OCaml's Obj.magic), we can.
f x cannot change y, because f x does not have access to y.
So here is an access control system, built in to the lambda calculus itself!
At first glance this might not look very promising.
For example, while f doesn't have access to y, it does have access to any global variables defined before f.
It also, typically, has access to the file-system and network,
which are effectively globals too.
To make this useful, we ban global variables.
Then any top-level function like f can only access things passed to it explicitly as arguments.
Avoiding global variables is usually considered good practise, and some systems ban them for other reasons anyway
(for example, Rust doesn't allow global mutable state as it wouldn't be able to prevent races accessing it from multiple threads).
Returning to the Haskell example above (but now in OCaml syntax),
it looks like this in our capability system:
123
letfx=...letmainch=output_stringch(f42)
Since f is a top-level function, we know it does not close over any mutable state, and our 42 argument is pure data.
Therefore, the call f 42 does not have access to, and therefore cannot affect,
any pre-existing state (including the filesystem).
Internally, it can use mutation (creating arrays, etc),
but it has nowhere to store any mutable values and so they will get GC'd after it returns.
f therefore appears as a pure function, and calling it multiple times will always give the same result,
just as in the Haskell version.
output_string is also a top-level function, closing over no mutable state.
However, the function resulting from evaluating output_string ch is not top-level,
and without knowing anything more about it we should assume it has full access to the output channel ch.
If main is invoked with standard output as its argument, it may output a message to it,
but cannot affect other pre-existing state.
In this way, we can reason about the pure parts of our code as easily as with Haskell,
but we can also reason about the parts with side-effects.
Haskell's purity is just a special case of a more general rule:
the effects of a (top-level) function are bounded by its arguments.
Attenuation
So far, we've been thinking about what values are reachable through other values.
For example, the set of ref-cells that can be modified by f x is bounded by
the union of the set of ref cells reachable from the closure f
with the set of ref cells reachable from x.
One powerful aspect of capabilities is that we can use functions to implement whatever access controls we want.
For example, let's say we only want f to be able to set the ref-cell, but not read it.
We can just pass it a suitable function:
12345
letx=ref0inletsetv=x:=vinfset
Or perhaps we only want to allow inserting positive integers:
Or log each time it is used, tagged with a label that's meaningful to us
(e.g. the function to which we granted access):
12345678
letlog=ref[]inletsetnamev=letmsg=sprintf"%S set it to %d"namevinlog:=msg::!log;setvinf(set"f");g(set"g")
Or all of the above.
In these examples, our function f never got direct access (permission) to x, yet was still able to affect it.
Therefore, in capability systems people often talk about "authority" rather than permission.
Roughly speaking, the authority of a subject is the set of actions that the subject could cause to happen,
now or in the future, on currently-existing resources.
Since it's only things that might happen, and we don't want to read all the code to find out exactly what
it might do, we're usually only interested in getting an upper-bound on a subject's authority,
to show that it can't do something.
The examples here all used a single function.
We may want to allow multiple operations on a single value (e.g. getting and setting a ref-cell),
and the usual techniques are available for doing that (e.g. having the function take the operation as its first argument,
or collecting separate functions together in a record, module or object).
Web-server example
Let's look at a more realistic example.
Here's a simple web-server (we are defining the main function, which takes two arguments):
12
letmainnethtdocs=...
To use it, we pass it access to some network (net) and a directory tree with the content (htdocs).
Immediately we can see that this server does not access any part of the file-system outside of htdocs,
but that it may use the network. Here's a picture of the situation:
Initial reference graph
Notes on reading the diagram:
The diagram shows a model of the reference graph, where each node represents some value (function, record, tuple, etc)
or aggregated group of values.
An arrow from A to B indicates the possibility that some value in the group A holds a reference to
some value in the group B.
The model is typically an over-approximation, so the lack of an arrow from A to B means that no such reference
exists, while the presence of an arrow just means we haven't ruled it out.
Orange nodes here represent OCaml values.
White boxes are directories.
They include all contained files and subdirectories, except those shown separately.
I've pulled out htdocs so we can see that app doesn't have access to the rest of home.
Just for emphasis, I also show .ssh separately.
I'm assuming here that a directory doesn't give access to its parent,
so htdocs can only be used to read files within that sub-tree.
net represents the network and everything else connected to it.
In most operating systems, directories exist in the kernel's address space,
and so you cannot have a direct reference to them.
That's not a problem, but for now you may find it easier to imagine a system where the kernel and applications
are all a single program, in a single programming language.
This diagram represents the state at a particular moment in time (when starting the application).
We could also calculate and show all the references that might ever come to exist,
given what we know about the behaviour of app and net.
Since we don't yet know anything about either,
we would have to assume that app might give net access to htdocs and to itself.
So, the diagram above shows the application app has been given references to net and to htdocs as arguments.
Looking at our checklist from the start:
It can't delete all my files, but it might delete the ones in htdocs.
It can't edit ~/.ssh/authorized_keys.
It might act as a relay, allowing remote machines to attack other computers on my local network.
It might send telemetry to a third-party.
We can read the body of the function to learn more:
Note: Net.listen net is typical OCaml style for performing the listen operation on net.
We could also have used a record and written net.listen instead, which may look more familiar to some readers.
Here's an updated diagram, showing the moment when Http.serve is called.
The app group has been opened to show socket and handler separately:
After reading the code of main
We can see that the code in the HTTP library can only access the network via socket,
and can only access htdocs by using handler.
Assuming Net.listen is trust-worthy (we'll normally trust the platform's networking layer),
it's clear that the application doesn't make out-bound connections,
since net is used only to create a listening socket.
To know what the application might do to htdocs, we only have to read the definition of static_files:
We see that handle_connection has no way to share telemetry information between connections,
given that handle_request never stores anything.
We can tell these things after only looking at the code for a few seconds, even though dozens of libraries are being used.
In particular, we didn't have to read handle_connection or any of the HTTP parsing logic.
Now let's enable TLS. For this, we will require a configuration directory containing the server's key:
OCaml syntax note: I used ~ to make tls_config a named argument; we wouldn't want to get this directory confused with htdocs!
We can see that only the TLS library gets access to the key.
The HTTP library interacts only with the TLS socket, which presumably does not reveal it.
Updated graph showing TLS
Notice too how this fixes the problem we had with our original policy enforcement system.
There, an attacker could request https://example.com/../tls_config/server.key and the HTTP server might send the key.
But here, the handler cannot do that even if it wants to.
When handler loads a file, it does so via htdocs, which does not have access to tls_config.
The above server has pretty good security properties,
even though we didn't make any special effort to write secure code.
Security-conscious programmers will try to wrap powerful capabilities (like net)
with less powerful ones (like socket) as early as possible, making the code easier to understand.
A programmer uninterested in readability is likely to mix in more irrelevant code you have to skip through,
but even so it shouldn't take too long to track down where things like net and htdocs end up.
And even if they spread them throughout their entire application,
at least you avoid having to read all the libraries too!
By contrast, consider a more traditional (non-capability) style.
We start with:
1
letmainhtdocs=...
Here, htdocs would be a plain string rather than a reference to a directory,
and the network would be reached through a global.
We can't tell anything about what this server could do from looking at this one line,
and even if we expand it, we won't be able to tell what all the functions it calls do, either.
We will end up having to follow every function call recursively through all of the server's
dependencies, and our analysis will be out of date as soon as any of them changes.
Use at different scales
We've seen that we can create an over-approximation of the reference graph by looking at just a small part of the code,
and then get a closer bound on the possible effects as needed
by expanding groups of values until we can prove the desired property.
For example, to prove that the application didn't modify htdocs, we followed htdocs by expanding main and then static_files.
Within a single process, a capability is a reference (pointer) to another value in the process's memory.
However, the diagrams also included arrows (capabilities) to things outside of the process, such as directories.
We can regard these as references to privileged proxy functions in the process that make calls to the OS kernel,
or (at a higher level of abstraction) we can consider them to be capabilities to the external resources themselves.
It is possible to build capability operating systems (in fact, this was the first use of capabilities).
Just as we needed to ban global variables to make a safe programming language,
we need to ban global namespaces to make a capability operating system.
For example, on FreeBSD this is done (on a per-process basis) by invoking the cap_enter system call.
We can zoom out even further, and consider a network of computers.
Here, an arrow between machines represents some kind of (unforgeable) network address or connection.
At the IP level, any process can connect to any address, but a capability system can be implemented on top.
CapTP (the Capability Transport Protocol) was an early system for this, but
Cap'n Proto (Capabilities and Protocols) is the modern way to do it.
So, thinking in terms of capabilities, we can zoom out to look at the security properties of the whole network,
yet still be able to expand groups as needed right down to the level of individual closures in a process.
Key points
Library code can be imported and called without it getting access to any pre-existing state,
except that given to it explicitly. There is no "ambient authority" available to the library.
A function's side-effects are bounded by its arguments.
We can understand (get a bound on) the behaviour of a function call just by looking at it.
If a has access to b and to c, then a can introduce them (e.g. by performing the function call b c).
Note that there is no capability equivalent to making something "world readable";
to perform an introduction,
you need access to both the resource being granted and to the recipient ("only connectivity begets connectivity").
Instead of passing the name of a resource, we pass a capability reference (pointer) to it,
thereby proving that we have access to it and sharing that access ("no designation without authority").
The caller of a function decides what it should access, and can provide restricted access by wrapping
another capability, or substituting something else entirely.
I am sometimes unable to install a messaging app on my phone because it requires me to grant it
access to my address book.
A capability system should never say "This application requires access to the address book. Continue?";
it should say "This application requires access to an address book; which would you like to use?".
A capability must behave the same way regardless of who uses it.
When we do f x, f can perform exactly the same operations on x that we can.
It is tempting to add a traditional policy language alongside capabilities for "extra security",
saying e.g. "f cannot write to x, even if it has a reference to it".
However, apart from being complicated and annoying,
this creates an incentive for f to smuggle x to another context with more powers.
This is the root cause of many real-world attacks, such as click-jacking or cross-site request forgery,
where a URL permits an attack if a victim visits it, but not if the attacker does.
One of the great benefits of capability systems is that you don't need to worry that someone is trying to trick you
into doing something that you can do but they can't,
because your ability to access the resource they give you comes entirely from them in the first place.
All of the above follow naturally from using functions in the usual way, while avoiding global variables.
Practical considerations
The above discussion argues that capabilities would have been a good way to build systems in an ideal world.
But given that most current operating systems and programming languages have not been designed this way,
how useful is this approach?
I'm currently working on Eio, an IO library for OCaml, and using these principles to guide the design.
Here are a few thoughts about applying capabilities to a real system.
Plumbing capabilities everywhere
A lot of people worry about cluttering up their code by having to pass things explicitly everywhere.
This is actually not much of a problem, for a couple of reasons:
We already do this with most things anyway.
If your program uses a database, you probably establish a connection to it at the start and pass the connection around as needed.
You probably also pass around open file handles, configuration settings, HTTP connection pools, arrays, queues, ref-cells, etc.
Handling "the file-system" and "the network" the same way as everything else isn't a big deal.
You can often bundle up a capability with something else.
For example, a web-server will likely let the user decide which directory to serve,
so you're already passing around a pathname argument.
Passing a path capability instead is no extra work.
Consider a request handler that takes the address of a Redis server:
1
Http.servesocket(handle_requestredis_url)
It might seem that by using capabilities we'd need to pass the network in here too:
1
Http.servesocket(handle_requestnetredis_url)
This is both messy and unnecessary.
Instead, handle_request can take a function for connecting to Redis:
1
Http.servesocket(handle_requestredis)
Then there is only one argument to pass around again.
Instead of writing the connection logic in handle_request, we write the same logic outside and just pass in the function.
And now someone looking at the code can see "the handler can connect to Redis",
rather than the less precise "the handler accesses the network".
Of course, if Redis required more than one configuration setting then you'd probably already be doing it this way.
The main problematic case is providing defaults.
For example, a TLS library might allow us to specify the location of the system's certificate store,
but it would like to provide a default (e.g. /etc/ssl/certs/).
This is particularly important if the default location varies by platform.
If the TLS library decides the location, then we must give it (read-only at least) access to the whole system!
We may just decide to trust the library, or we might separate out the default paths into a trusted package.
Levels of support
Ideally, our programming language would provide a secure implementation of capabilities that we could depend on.
That would allow running untrusted code safely and protect us from compromised packages.
However, converting a non-capability language to a capability-secure one isn't easy,
and isn't likely to happen any time soon for OCaml
(but see Emily for an old proof-of-concept).
Even without that, though, capabilities help to protect non-malicious code from malicious inputs.
For example, the request handler above forgot to sanitise the URL path from the remote client,
but it still can't access anything outside of htdocs.
And even if we don't care about security at all, capabilities make it easy to see what a program does;
they make it easy to test programs by replacing OS resources with mocks;
and preventing access to globals helps to avoid race conditions,
since two functions that access the same resource must be explicitly introduced.
Running on a traditional OS
A capability OS would let us run a program's main function and provide the capabilities it wanted directly,
but most systems don't work like that.
Instead, each program requires a small trusted entrypoint that has the full privileges of the process.
In Eio, an application will typically start something like this:
Eio_main.run starts the Eio event loop and then runs the callback.
The env argument gives full access to the process's environment.
Here, the callback extracts network and filesystem access from this,
gets access to just "/srv/www" from fs,
and then calls the main function as before.
Note that Eio_main.run itself is not a capability-safe function (it magics up env from nothing).
A capability-enforcing compiler would flag this bit up as needing to be audited manually.
Use with existing security mechanisms
Maybe you're not convinced by all this capability stuff.
Traditional security systems are more widely available, better tested, and approved by your employer,
and you want to use that instead.
Still, to write the policy, you're going to need a list of resources the program might access.
Looking at the above code, we can immediately see that the policy need allow access only to the "/srv/www" directory,
and so we could call e.g. unveil here.
And if main later changes to use TLS,
the type-checker will let us know to update this code to provide the TLS configuration
and we'll know to update the policy at the same time.
If you want to drop privileges, such a program also makes it easy to see when it's safe to do that.
For example, looking at main we can see that net is never used after creating the socket,
so we don't need the bind system call after that,
and we never need connect.
We know, for instance, that this program isn't hiding an XML parser that needs to download schema files to validate documents.
Thread-local storage
In addition to global and local variables, systems often allow us to attach data to threads as a sort of middle ground.
This could allow unexpected interactions. For example:
123
letx=ref0infx;g()
Here, we'd expect that g doesn't have access to x, but f could pass it using thread-local storage.
To prevent that, Eio instead provides Fiber.with_binding,
which runs a function with a binding but then puts things back how they were before returning,
so f can't make changes that are still active when g runs.
This also allows people who don't want capabilities to disable the whole system easily:
It looks like f () doesn't have access to anything, but in fact it can recover env and get access to everything!
However, anyone trying to understand the code will start following env from the main entrypoint
and will then see that it got put in fiber-local storage.
They then at least know that they must read all the code to understand anything about what it can do.
More usefully, this mechanism allows us to make just a few things ambiently available.
For example, we don't want to have to plumb stderr through to a function every time we want to do some printf debugging,
so it makes sense to provide a tracing function this way (and Eio does this by default).
Tracing allows all components to write debug messages, but it doesn't let them read them.
Therefore, it doesn't provide a way for components to communicate with each other.
It might be tempting to use Fiber.with_binding to restrict access to part of a program
(e.g. giving an HTTP server network access this way),
but note that this is a non-capability way to do things,
and suffers the same problems as traditional security systems,
separating designation from authority.
In particular, supposedly sandboxed code in other parts of the application
can try to escape by tricking the HTTP server part into running a callback function for them.
But fiber local storage is fine for things to which you don't care to restrict access.
Symlinks
Symlinks are a bit of a pain! If I have a capability reference to a directory, it's useful to know that I can only access things beneath that directory. But the directory may contain a symlink that points elsewhere.
One option would be to say that a symlink is a capability itself, but this means that you could only create symlinks to things you can access yourself, and this is quite a restriction. For example, you might be forbidden from extracting a tarball because tar didn't have permission to the target of a symlink it wanted to create.
The other option is to say that symlinks are just strings, and it's up to the user to interpret them.
This is the approach FreeBSD uses. When you use a system call like openat,
you pass a capability to a base directory and a string path relative to that.
In the case of our web-server, we'd use a capability for htdocs, but use strings to reference things inside it, allowing the server to follow symlinks within that sub-tree, but not outside.
The main problem is that it makes the API a bit confusing. Consider:
1
save_to(htdocs/"uploads")
It might look like save_to is only getting access to the "uploads" directory,
but in Eio it actually gets access to the whole of htdocs.
If you want to restrict access, you have to do that explicitly
(as we did when creating htdocs from fs).
The advantage, however, is that we don't break software that relies on symlinks.
Also, restricting access is quite expensive on some systems (FreeBSD has the handy O_BENEATH open flag,
and Linux has RESOLVE_BENEATH, but not all systems provide this), so might not be a good default.
I'm not completely satisfied with the current API, though.
Time and randomness
It is also possible to use capabilities to restrict access to time and randomness.
The security benefits here are less clear.
Tracking access to time can be useful in preventing side-channel attacks that depend on measuring time accurately,
but controlling access to randomness makes it difficult to e.g. randomise hash functions to
help prevent denial-of-service-attacks.
However, controlling access to these does have the advantage of making code deterministic by default,
which is a great benefit, especially for expect-style testing.
Your top level test function is called with no arguments, and therefore has no access to non-determinism,
instead creating deterministic mocks to use with the code under test.
You can then just record a good trace of a test's operations and check that it doesn't change.
Power boxes
Interactive applications that load and save files present a small problem:
since the user might load or save anywhere, it seems they need access to the whole file-system.
The solution is a "powerbox".
The powerbox has access to the file-system and the rest of the application only has access to the powerbox.
When the application wants to save a file, it asks the powerbox, which pops up a GUI asking the user to choose the location.
Then it opens the file and passes that back to the application.
Conclusions
Currently-popular security mechanisms are complex and have many shortcomings.
Yet, the lambda calculus already contains an excellent security mechanism,
and making use of it requires little more than avoiding global variables.
This is known as "capability-based security".
The word "capabilities" has also been used for several unrelated concepts (such as "POSIX capabilities"),
and for clarity much of the community rebranded a while back as "Object Capabilities",
but this can make it seem irrelevant to functional programmers.
In fact, I wrote this blog post because several OCaml programmers have asked me what the point of capabilities is.
I was expecting it to be quite short (basically: applying functions to arguments good, global variables bad),
but it's got quite long; it seems there is a fair bit that follows from this simple idea!
Instead of seeing security as an extra layer that runs separately from the code and tries to guess what it meant to do,
capabilities fit naturally into the language.
The key difference with traditional security is that
the ability to do something depends on the reference used to do it, not on the identity of the caller.
This way of thinking about security works not only for controlling access to resources within a single program,
but also for controlling interactions between processes running on a machine, and between machines on a network.
We can group together resources and zoom out to see the overall picture, or expand groups to zoom in and get a closer
bound on the behaviour.
Even ignoring security, a key question is: what can a function do?
Should a function call be able to do anything at all that the process can do,
or should its behaviour be bounded in some way that is obvious just by looking at it?
If we say that you must read the source code of a function to see what it does, then this applies recursively:
we must also read all the functions that it calls, and so on.
To understand the main function, we end up having to read the code of every library it uses!
Resources diagram (simplified)
