Universal Bits

Sandboxing local models on macOS

2026-03-12T00:00:00+00:00

The recently released Qwen3.5 family advances the capabilities of open-weights multi-modal models: rumors are they’re exceptionally good for their size. The 35B-A3B and 27B variants are small enough to run on 64GB of unified memory (especially if quantized). So, I decided to give them a try.

Sandboxing

Being wary of prompt injection and the probabilistic nature of LLMs, I usually confine AI agents to ephemeral VMs, where they can wreak havoc in YOLO mode (--dangerously-skip-permissions). With hosted models, this is easy: the VM just needs internet access and remains isolated from the host. To run local models on the VM instead, we’d need to let it access the GPU and install compatible drivers. My only device with 64GB of RAM is a MacBook, and I run Linux VMs on it, so I’d need to find Linux drivers compatible with Apple hardware (if they exist).

A pragmatic approach is to:

Sandbox model inference (since parsing GGUF models has led to code execution vulnerabilities);
either sandbox the client as well, for quick explorations; or
keep the agents in a VM and forward their API access to the local model.

The go-to sandbox on macOS is sandbox-exec. Apple has marked it as deprecated years ago, but it continues to work because it powers App Sandbox, the sandboxing mechanism used by the Apple Store. It’s a weird beast, configurable through arcane profiles written in a Scheme dialect (which got a lot easier to write with LLMs). Now that the big AI companies are using it to sandbox their tools¹, it is seeing a renaissance. I have been using it for a while to sandbox lazyvim when running outside a VM, so that it cannot reach the internet or make changes outside ~/work.

Here, we’ll take a similar approach to sandbox model inference down to its minimal capabilities (essentially, GPU and driver access) and then sandbox the agents to run offline and within the well-defined boundaries of a workspace. Importantly, all sandbox profiles have default deny, so we can allow exactly what we need and deny the rest.

Sandboxing profiles

If you want to jump to the code, you’ll find everything at the sandboxed-ai GitHub repository.

Server

We’ll use llama-server to do model inference, which also provides OpenAI-compatible REST APIs for the clients. The sandbox file I came up with through trial and error only allows access to the executable, the model weights, the GPU and its drivers, and some cache directories. In addition, llama-server can bind to port 8080 to serve its APIs, but cannot reach the network (outbound).

The result is a simple sandbox.sh script (no external dependencies) that calls sandbox-exec and spins up the server. All additional arguments are forwarded to llama-server. Because the sandbox prevents network access, the script does some special handling of the --model parameter: if a model doesn’t exist locally, it will fetch it (outside the sandbox) through curl.

./sandbox.sh llama-server --model unsloth/Qwen3.5-9B-GGUF:Qwen3.5-9B-Q8_0.gguf

We can now run Qwen3.5. Here, I am using the unsloth quantized model and including their recommended sampling parameters:

$ ./sandbox.sh llama-server \
    --model unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'

Starting sandboxed llama-server:
  binary:        /nix/store/l4xdm13zilm71n1jad95rpzk49h57is5-llama-cpp-metalkit-0.0.0/bin/llama-server
  model:         /Users/aldur/Work/local-opencode/.opencode/models/unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
  alias:         Qwen3.5-35B-A3B-GGUF
  port:          8080
  extra:         --ctx-size 16384 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 --chat-template-kwargs {"enable_thinking":false}

...

Local clients

With the model ready, we can now chat with it. To do that, I have also prepared sandbox profiles for:

simonw/llm, which I use to run quick queries;
opencode, which I use interactively.

$ ./sandbox.sh llm
Hello there!
^D
Hello! How can I help you today?

The sandbox.sh script automatically takes care of the configuration files that both tools require to interface with llama-server (you’ll find an opencode.json in the script directory).

The sandbox profiles are quite hardened: they don’t allow network outbound, restrict all writes to a single workspace directory (-w), and whitelist only selected executables (restricting allowed tools). Cache files are written alongside the script. Because of the sandbox restrictions, a few things will break: opencode web, for instance, because it requires remote access to load the frontend assets. On the other hand, the sandbox guarantees both integrity and confidentiality on bare macOS, guaranteeing that the computation remains local and preventing opencode from leaking your prompts².

In a QEMU VM

When I need to do anything more than quick checks/chats, I just spin up an ephemeral QEMU VM. My qemu-vm script uses SLiRP user network³, so llama-server from the host is available at 10.0.2.2:8080:

qemu-vm -d $(pwd) -- -ephemeral
Starting VM...
...
qemu-nixos login: aldur (automatic login)
[I] aldur@qemu-nixos ~> curl http://10.0.2.2:8080/health
{"status":"ok"}
[I] aldur@qemu-nixos ~> cat opencode.json
{
  "$schema": "https://opencode.ai/config.json",
  "model": "llama/Qwen3.5-35B-A3B-GGUF",
  "provider": {
    "llama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://10.0.2.2:8080/v1",
        "apiKey": "dummy"
      },
      "models": {
        "Qwen3.5-35B-A3B-GGUF": {
          "name": "Qwen3.5-35B-A3B-GGUF",
          "tool_call": true
        }
      }
    }
  },
  "autoupdate": false
}
[I] aldur@qemu-nixos ~> nix run github:nixos/nixpkgs#opencode
                                   ▄
  █▀▀█ █▀▀█ █▀▀█ █▀▀▄ █▀▀▀ █▀▀█ █▀▀█ █▀▀█
  █  █ █  █ █▀▀▀ █  █ █    █  █ █  █ █▀▀▀
  ▀▀▀▀ █▀▀▀ ▀▀▀▀ ▀▀▀▀ ▀▀▀▀ ▀▀▀▀ ▀▀▀▀ ▀▀▀▀

  Session   Greeting and quick check-in

Hello Qwen! How are you doing today?
I'm doing well, thanks for asking! How can I help you today?

Test drives

On a MacBook Pro with an M3 Max CPU and 64GB of RAM, Qwen3.5-35B-A3B-Q8_0 is quick enough that any prompt feels snappy and interactive, even with thinking enabled. When using opencode, it churns for a few seconds through the initial big prompt (on a cold cache) and is then able to use tools and read files quickly enough. Although the quality of results is lower than frontier hosted models, it’s a big step forward: this relatively small model can make small, interactive changes. Plus, the OCR capabilities of the whole family are impressive.

I also gave Qwen3.5-27B-Q8_0.gguf a shot, which trades inference speed for better accuracy. To test drive it, I stuck to the default parameters, fed it a draft of this blog post, then asked for edits⁴. Here’s the transcript of the session, the server logs and the model info. Producing the output required more than 10 minutes of computation and used about 15% of battery charge, with fans spinning and GPU at 100% utilization. The task was relatively easy, and the results are satisfying: I integrated almost all of its suggestions.

Thank you for reading and ‘til next time! 👋

Footnotes

With the important caveat that often the agent can disable the sandbox at runtime. ↩
In the default configuration, the small_model configuration parameter uploads prompts to OpenCode’s servers to generate session titles. ↩
SLiRP user network isn’t ideal for isolation because it doesn’t firewall the VM, which appears to macOS as a process that can access localhost sockets. The proper solution is to use TAP bridges, but we’ll leave that to another post. ↩
As a reminder, all writing on this blog (typos and weird sentences included) is mine. Writing is thinking and I don’t see the point in having the LLM think for me. ↩

Why my internet dropped like clockwork

2026-02-12T00:00:00+00:00

I recently upgraded my home network to gigabit internet provided by O2, the low-cost brand of Movistar Spain. They run fiber to the home (FTTH) and lease consumers their routers to convert the optical signal into an electric one. In my case, I got a Mitrastar HGU router (GPT-2742GX4X5 v6, running firmware GL_g2.5_100XNT0b23).

ISP routers come with security and privacy concerns: where possible, I replace them with alternatives I can customize and control. At home, I already own a router that supports OpenWRT. It’s much better from a security/privacy standpoint, but it doesn’t have an ONT and so I cannot connect it directly to fiber.

To fix that, I configured the ISP router in bridge network mode and connected one of its LAN ports to the WAN on my router. Then, within OpenWRT, I configured the required PPPoE credentials and created a VLAN. I got online, made a few speed measurements, and ensured that everything was working great.

After some time I noticed that the connection dropped and that the web interface of the router wasn’t reachable anymore. I brushed it off, thinking that the network would eventually stabilize. But it kept happening! So I dug a bit more. At first, I thought of a hardware issue (maybe on the fiber line). Then, I noticed the following pattern:

OpenWRT’s statistics showed connection drops every two hours, like clockwork.

Drops so predictable likely indicated a software issue. After ensuring that the issue wasn’t on OpenWRT’s side, I started looking at the HGU router. Mitrastar firmwares have a history of bugs; it is also likely that bridge mode, isn’t as battle-tested as the rest, because few users enable it.

ISP routers can be a mixed bag in terms of debug access. This one is weirder than usual:

User 1234 can ssh into the router with a password printed on a label on the router’s back.
By default, ssh drops the user into a restricted shell, unless you use ssh 1234@192.168.1.1 /bin/sh to get a better shell.

$ ssh 1234@192.168.1.1
1234@192.168.1.1's password:
>ls
Can't find command: [ls]. Type '?' for usage
>?
>?
# Confused at what's going on

$ ssh 1234@192.168.1.1 /bin/sh
1234@192.168.1.1's password:
ls
rom-t
busybox
BusyBox v1.26.2 (2025-06-19 15:12:31 CST) multi-call binary.
...

The 1234 user is technically root; the shell is based on busybox and it is half-broken: there are no prompts and ls doesn’t show the full directory listing. However, it’s enough to start digging.

After poking around, I discovered the router configuration at /tmp/var/pdm/config.xml. It refers to a watchdog, which seems to be one of the customizations that TELEFONICA (Movistar’s brand in Spain) applied to it:

   PARAMETER="configured" TYPE="boolean">1
  
     PARAMETER="configured" TYPE="string">hgu.rima-tde.net
     PARAMETER="configured" TYPE="uint32">1
     PARAMETER="configured" TYPE="uint32">2
     PARAMETER="configured" TYPE="uint32">900
  
  
     PARAMETER="configured" TYPE="string">pppd;ztr69;zebra;ripd;zywifid;igmpproxy;voiceApp;tefdog
  

The watchdog itself appears to be tefdog. When running in bridge mode some of its health checks likely fail and trigger a restart of the connection. It also seems to leak memory, which explains why the web interface would become unreachable after a while. In particular, the list of monitored processes always includes pppd, even though it doesn’t run in bridge mode.

To test the hypothesis of a badly configured watchdog, I first tried to kill it and wait. That did not work, because PID 1 would restart it after a few minutes. Because the file-system on the router is read-only, I couldn’t edit the executable directly. But I could use a bind mount over it:

echo '#!/bin/sh' > /tmp/tefdog_fake
echo 'exit 0' >> /tmp/tefdog_fake
chmod +x /tmp/tefdog_fake
mount --bind /tmp/tefdog_fake /usr/bin/tefdog

At this point, I killed it again, ensured it would not restart, and waited for a few hours to see if the connection would drop. When it didn’t, I knew I had found the issue!

The bind mount hack, however, wouldn’t survive a reboot. After a power loss, the router would restart and the watchdog would resume killing the connection every two hours. I kept poking around for ways to persist the change or to overwrite the router configuration, until I remembered that a router’s web interface typically allows backing up and re-uploading the configuration. In some cases, modified configurations would also unlock settings not available through the web interface.

I went ahead, exported the configuration through the web UI, and quickly discovered that it is encrypted:

$ file romfile.cfg
romfile.cfg: openssl enc'd data with salted password

I mentioned that this router is a bit weird, didn’t I? ssh provides root access, but only if you know which process to launch. You can read the configuration through SSH, but then the export is encrypted. After a bit more digging, I figured out where the decryption happens in the firmware:

/usr/bin/ccc_preload.sh: openssl aes-256-cbc -md MD5 -k $2 -d \
  -in /usr/etc/smt.cfg -out /var/pdm/config.xml
# NOTE: I added the line break for readability.

Later on, I discovered where the encryption password is stored and that it’s a base64 encoding of 24 (random?) bytes:

$ grep ENCRYPT /etc/MLD_Config.sh
MLD_APPS_ENABLE_ENCRYPT_CONF_FILE_KEY=zRUqIM1VCqPBrlYbf6CXiOZoZwiIAMHJ
# NOTE: this isn't my encryption key, but a similar-looking one.

I don’t have another router to check whether this password is hardcoded into the firmware or generated depending on the hardware (e.g., the MAC address). Out of an abundance of caution, I haven’t shared the exact key I found on my router, but feel free to reach out if you are interested in it.

After testing the decryption key, I modified the configuration by disabling the watchdog:

diff -u config.xml.back config.xml
--- config.xml.back 2026-02-01 18:28:36
+++ config.xml 2026-02-01 18:41:10
@@ -292,7 +292,7 @@
     
   
   
-    1
+    0
     
       hgu.rima-tde.net
       1

While I was at it, I also made a few more changes to disable analytics, prevent the router from phoning home, turn off WPS and block TR069 at the firewall level, preventing remote firmware upgrades or configuration changes.

diff -u config.xml.back config.xml
--- config.xml.back 2026-02-01 18:28:36
+++ config.xml 2026-02-01 18:41:10
@@ -150,7 +150,7 @@
             Permit
             TCP
             Default_TR069
-            1
+            0
             
@@ -602,7 +602,7 @@
             PushButton
             1
             1
-            1
+            0
             0
           
           
@@ -921,7 +921,7 @@
             PushButton
             1
             1
-            1
+            0
             0
           
           
@@ -1502,7 +1502,7 @@
     1
     0
-    1
+    0
     604800
     
       

After re-encrypting the modified configuration, I uploaded it, rebooted the router, and it has since been running smoothly:

No more drops after the fix!

To wrap things up, I also dumped the bootloader and the firmware (just in case, as neither seems to be available online):

cat /dev/mtd0 > /tmp/bootloader.bin
cat /dev/mtd4 > /tmp/tclinux.bin
md5sum /tmp/bootloader.bin
16fa4cafc4e2c412e053e8757af3d60b  /tmp/bootloader.bin
md5sum /tmp/tclinux.bin
8bb6e6d4e1fb231e4fbb0a98a494ccaf  /tmp/tclinux.bin
nc 192.168.1.3 1234 < /tmp/bootloader.bin
nc 192.168.1.3 1234 < /tmp/tclinux.bin

One mystery remains. Why did the drops happen exactly every two hours? I looked at the router configuration and then had an AI agent look at the firwmware through Ghidra’s MCP. The most convincing explanation (so far) is that the watchdog performs a check every 15 minutes. After 4 failed checks, it tries to restart the PPP interfaces (a noop in bridge mode). After 4 more failed checks, it sends a telemetry event to the ISP’s IoT bridge on Azure. Which, in response, requests a reset of the network stack and temporarily drops the connection. Here are the relevant parameters from the decrypted configuration:

Parameter name	Value	Notes
WD_CheckChange	900	seconds, frequency of checks
AlertAfter	2	how many failed checks before resetting PPP (off-by-one, the decompiled shows it resets when ≥ 3, 0-based)
MaxReset	1	how many resets to try before notifying upstream

Thank you for reading so far! Reach out if you’d like to chat. 👋

Impermanent NixOS VMs in ChromeOS

2026-02-01T00:00:00+00:00

We have talked about using NixOS to run VMs under ChromeOS. The VM image doesn’t include any secrets and relies on hardware keys for authentication and signatures (e.g., to push and sign commits on GitHub). This way, even if the VM was compromised, the hardware-backed credentials would remain safe.

In practice, though, the VM slowly accumulates other (possibly) confidential information: source code, authentication tokens, LLM sessions, and even shell history. To clean things up, a user would need to periodically destroy and recreate the VM. But users (myself included) get sloppy, for instance when overworked. Worse, being in a VM might give them a wrong sense of confidence that there is nothing to leak!

What is impermanence

Good safety systems should not rely on user behavior to maintain their safety properties. Neither should VMs meant to be ephemeral working environments. Luckily NixOS can help us remediate that.

Due to its reproducibility, NixOS can boot by exclusively relying on a /init file and the /nix store¹. Everything else can be re-created at runtime (typically, by symlinking files and directories to the appropriate path in the store). We can leverage that to achieve impermanence and ensure a clean system after each reboot.

The Nix community has contributed a few ways to achieve impermanence. Typically, they require configuring the system to erase itself at boot and maintaining a whitelist that will survive reboots (for instance SSH host keys, which would otherwise result in a different remote fingerprint at each reboot).

Erasing `/home`

For the NixOS VMs I use under ChromeOS, I configure impermanence as follows:

/home mounts through tmpfs. This way, anything I don’t explicitly whitelist will be gone at power down. I don’t have to worry about automating its deletion, but I need to cap /home size to (a portion of) the available memory.
The preservation module takes care of safe-keeping a few required files and directories under /home (for instance known SSH hosts or source code I explicitly want to keep across reboots).
The rest of the filesystem is not impermament, for simplicity and to save RAM. My VM user cannot become root, so (assuming correct permissions) should not be able to modify system files anyways.

Settings things up

A minimal Nix module to achieve impermanence looks as follows:

{inputs, ...}:
let username = "aldur"; in {
  imports = [
    # Impermanence: import the preservation module
    inputs.preservation.nixosModules.preservation
  ];

  # Impermanence: tmpfs home with preservation
  fileSystems."/home" = {
    device = "none";
    fsType = "tmpfs";
    options = [
      "defaults"
      "size=4G"
      "mode=755"
    ];
  };

  # Impermanence: whitelist to preserve
  # See `preservation` docs for more configuration options.
  preservation = {
    enable = true;

    preserveAt."/persist".users.${username}.directories = [
      "Documents/"

      {
        directory = ".ssh";
        mode = "0700";
      }
    ];
  };
}

Playing nicely with `home-manager`

While setting things up and testing the result, the VM would occasionally fail to correctly load home-manager’s configuration (which, importantly, configures the fish shell).

After quite some debugging, I figured out that home-manager was racing against garcon, a service that automatically starts and spawns a shell when a Baguette VM starts from ChromeOS through vmc start. When garcon won the race and executed before home-manager, it would launch fish without any customization.

The fix is to delay garcon (a user service) until after home-manager has completed:

_:
let username = "aldur"; in {
  services."user@" = {
    overrideStrategy = "asDropin";
    after = [ "home-manager-${username}.service" ];
    wants = [ "home-manager-${username}.service" ];

    # In case something goes wrong
    serviceConfig.TimeoutStartSec = "90";
  };
}

Wrapping up

I have run with impermanence for a few weeks now and, so far, I haven’t had any issue. With 16GB of RAM, I typically size /home to 4GB. I suspect that this could not be enough to complete artifact-heavy or memory-intensive builds, e.g. Rust workspaces or numeric Python projects. If that happens, I’ll configure the build tools to store assets in /tmp. Similarly, if a project’s cache is wiped on reboot, offline rebuilds would fail (e.g., while on a flight). I avoid that by relying on nix develop for local development shells, which caches requirements in the nix store.

I also see a few security limitations of the approach. Sophisticated malware could persist through the system configuration or in the nix store (to which my user has write access). Although it isn’t a silver bullet, impermanence for /home still adds to defense in depth. It prevents some classes of attacks (e.g., supply chain compromises) from harvesting credentials that accumulated over time. I’ll continue hardening the VM image against the remaining attacks: I would love to enable mandatory access control policies, but doing that in NixOS seems like a deep rabbit hole to explore.

👋 Thank you for reading so far! Shoot me an email if you’d like to comment, discuss, or just say hi. Until next time!

Footnotes

Pretty much what happens when the Baguette NixOS image starts. ↩

Effective salespeople

2025-11-09T00:00:00+00:00

Nick Huber, in “The Sweaty Startup”, writes that sales is the foundation of every business.

Having a tech background and being mostly surrounded by other builders, sales is both one of my weakest skills and one that I want to improve. I’ve witnessed great ideas fail because nobody was selling them, disproving the “build it and they will come” myth, and I have seen what a truly good seller can bring to the table.

To me, good sales boil down to mutual wins. A good salesman will figure out what the other party needs, a way to provide it at a fair price, and manage expectations, while avoiding deception and the psychological biases that trap both buyers and sellers.

A corresponding section of the book describes a few down-to-earth habits to become more effective at sales. I found them particularly valuable because they align with good product management or successful negotiations. I am writing them here with a few reflections so that, eventually, they can become part of my mental models.

1. Realize not everyone wants to buy what you’re selling
2. Get comfortable being uncomfortable
3. Prove that you are an expert
4. Manage expectations
5. Add value first
6. Make scarcity work for you
7. Let the other party sell themselves
Closing thoughts

1. Realize not everyone wants to buy what you’re selling

Some will never be your users.

This is both true and hard to accept at first, especially when we compare ourselves to the hyperscaler of the day, or when technology is so pervasive that almost everyone browses social media through a smartphone.

For those who would be your users, though, remember that innovation and adoption take time.

2. Get comfortable being uncomfortable

Improving at something requires discovering one’s limits and overcoming them, while facing both the unknown and what’s visible around us. Sales require that same attitude, times a thousand.

Especially when cold-calling or finding an initial market fit, sales are a numbers game. You will have dozens of conversations, each equally uncomfortable. Make them worth it: each sale is an opportunity to figure out someone’s needs and an opportunity to adapt your offer.

3. Prove that you are an expert

Experts know about both the upsides and the downsides of an opportunity. They have been in the trenches, and they know that things can and will go wrong, but they also recognize the value that an opportunity can provide.

Some salesmen will only try to sell you on the upside, without mentioning the downsides. This sometimes results in potential buyers getting defensive, feeling that something is “too good to be true” and that they aren’t seeing the full picture yet. They’ll likely stall or simply back out. I know I do, in those cases.

Instead of hiding risks and difficulties, acknowledge them in your pitch and show you know how to manage them. Exposing the negative aspects makes them less frightening. It defuses them while reinforcing what’s positive.

4. Manage expectations

Success depends on how you define it. Pick success criteria that you are very likely to meet, while ensuring to satisfy the customer needs.

The classic examples draw from forecasting how long something will take. Instead of over-promising on a tight timeline that requires everything to go smoothly, be realistic and think about risks, uncertainties, and unknown unknowns. Discuss them upfront with your potential buyers. Together, you’ll pick good success criteria.

Those conversations will be hard but necessary to manage all stakeholders’ expectations. Have them early on: you will also find out that some potential buyers are not the right fit for you.

5. Add value first

Business is a positive-sum game. Someone’s success will enable them to spend more, and their spending is going to be someone else’s income.

Adding value means first providing something that your users need, so they can test it and eventually trust you. At that point, the sale becomes a lot easier.

Successful SaaS companies (e.g., Tailscale) have found a good balance around this, with a free tier generous enough to suit most needs and a paid tier you (or your employer) can purchase after finding trust and value in their service.

6. Make scarcity work for you

Not every customer will provide you the same value: 20% of your users might generate 80% of your sales.

Gently push back potential buyers after you have made it clear that you are an expert, you know how to manage expectations, and how to add value. You don’t need to win every sale and if you do, your prices are probably too low. It will be hard because it looks counter-intuitive.

Pushing back allows you to select customers that are right for you, while turning away potentially bad customers.

7. Let the other party sell themselves

After having defused the negatives and having gently pushed them away, ask your potential buyer “What makes you think I’d be a good fit?”.

In negotiations, open ended questions and well-timed silences will allow information to flow while you discover what the other party needs. In sales, they’ll do something as powerful: they’ll change perspective and framing. Then, the other party will start selling themselves to you, explaining how they’d be a great fit or a good customer. If that happens, they’ll have overcome any objection they might have had and convinced themselves that they want to work with you.

Closing thoughts

I find that sales, negotiations, and product management share these traits. They boil down to a few core principles: being upfront and providing true value. Forecasting what to expect and know what you don’t know. Managing stakeholders’ expectations by defusing the negatives and, as a result, reinforcing the positives. Not everyone will be your user, nor will every user be right for you. Change perspective and re-frame things to let the other party sell to you. You will learn what they really need and they will work with you to overcome objections and biases.

How do you approach sales? Have you found yourself nodding along or violently shaking your head? I’d love to hear about it – shoot me an email!

Thank you for reading and ‘til next time! 👋

NixOS Baguette images in ChromeOS

2025-10-29T00:00:00+00:00

This post extends the one about NixOS containers in ChromeOS to build Baguette images. Give Baguette a try and let me know how it works for you!

The ChromiumOS team is experimenting with Baguette 🥖, a way to run containerless VM images in ChromeOS. By not running through LXC, Baguette gives users more freedom (e.g., to run Kubernetes without KVM or access the GPU).

The nixos-crostini repository already provided the magic glue to build NixOS containers that fully integrate with Crostini. When someone asked if it would be possible to support Baguette as well, I started taking a look at what that would take. This post describes the results:

Background: ChromeOS VMs
Baguette NixOS images
Conclusion

tl;dr: You can build both LXC containers and Baguette images to run your NixOS configuration in Crostini. They provide the same features and UX (e.g., clipboard sharing, Wayland/port forwarding, notifications, file browsing from ChromeOS).

Background: ChromeOS VMs

Under the hood, ChromeOS runs VMs through crosvm, a hardened virtual machine monitor. We already met it when investigating FIDO2 support in Linux ChromeOS guests.

Crostini uses crosvm to run a stripped-down VM called termina that boots quickly to run the user’s containers. It also does a few more things:

It mounts crosvm-tools, made available by the host through crosvm, into a guest directory (and later into containers as well).
It runs vshd, allowing the host to get a shell on the guest.
It handles the lifecycle of the VM and of its processes through maitred.

The default Baguette image is Debian-based and replicates all this. In addition, it configures the VM to run garcon and sommelier directly (while in Crostini they run within the container). Together, they provide URI handling, file browsing, and X/Wayland forwarding: all the things that make Crostini container very pleasant to use.

Baguette NixOS images

Our NixOS Baguette image will need to replicate the Debian image setup. This could turn out to be tricky, because NixOS cannot run non-Nix executables due to the lack of FHS and of a global library path. Luckily, we won’t need to worry about that: crosvm-tools include their own libraries and dynamic linker, so they run without issues in NixOS.

When we prepared NixOS LXC images for Crostini, we already figured out how to run garcon and sommelier at user log in. Mounting crosvm-tools and starting vshd and maitred was straightforward and simply required adding their systemd unit definitions.

A Baguette image is a compressed BTRFS image built from a RootFS tarball. To build the tarball from the NixOS, I took a page from the lxc-container NixOS module. Then, to package it:

I initially tried the Python script used by Google, but it depends on libguestfs-appliance and is not available for aarch64-linux in nixpkgs¹.
I later switched to a QEMU-based approach that works with Nix and supports ARM².

After transferring the compressed image to the Chromebook “Downloads” directory we can run it from crosh:

vmc create --vm-type BAGUETTE \
  --size 15G \
  --source /home/chronos/user/MyFiles/Downloads/baguette_rootfs.img.zst \
  baguette

vmc start --vm-type BAGUETTE baguette

You might have heard about the #crostini-containerless flag: you can actually run vmc start --vm-type BAGUETTE even without setting it. It only affects what happens when you “Configure Linux” in ChromeOS or use the “Terminal” app to launch a Linux guest.

This way, you can try Baguette without losing your Crostini containers.

At boot, maitred relies on /usr/sbin/usermod to configure users and groups. The usermod lives under a different path in NixOS, but I symlinked it to /usr/sbin/ to solve the issue and correctly get to a shell. Within the VM, I configured the DNS to rely on the host and set the environment variables required by crosvm-tools.

X/Wayland and port forwarding were the last pieces of the puzzle. By diving into the source code and the logs, I discovered that the /dev/wl0 device was missing read/write permissions for non-root users. By fixing it with a quick udev rule, clipboard sharing and GUI apps started to work. I also created a systemd unit to start cros-port-listener and enable automated port-forwarding from Baguette to ChromeOS (very handy when writing this blog to preview its HTML in Chrome).

With our image prepared and all issues fixed, Baguette is ready to shine! The baguette.nix file includes all the configuration in details, if you are curious. Here is the result, showing a baguette-nixos VM correctly forwarding a Wayland session to ChromeOS.

Wayland forwarding working in a Baguette VM.

How-to: Make it yours

nixos-crostini can now build both Baguette images and LXC containers. If you give it a try, let me know how it goes through any of the contacts in the footer.

tip: nixos-crostini builds NixOS Baguette images in CI and uploads them as GitHub workflow artifacts. Download them to quickly boot Baguette and then re-build NixOS from your customized configuration.

If you want to change the default username, fork the repository and edit the configuration. The CI will re-build the image for you.

How-to: Additional shell sessions

To get additional shell sessions from new crosh tabs, use:

vsh baguette penguin

We don’t really need the penguin argument, but without it we will get the following error:

if attempting to connect to a containerless guest please use vsh termina penguin.

How-to: USB forwarding

With Baguette, you can easily configure USB devices from Settings → Linux → Manage USB devices:

Click on: Enable persistent USB device sharing with guests.
Enable any USB device you’d like available to Baguette.

Selected devices will automatically be forwarded to Baguette once plugged in.

If you want to enable USB forwarding through crosh, Baguette simplifies the LXC approach because it doesn’t need a container name.

Insert the device and then navigate to chrome://usb-internals. In the devices tab, note the Bus number and Port number of your device. dmesg in crosh will provide the same information, if you prefer.

Now open a crosh shell and attach the USB to the VM:

# Replace  and  with the Bus and Port number from above.
vmc usb-attach baguette :

How-to: Launch NixOS from “Terminal”

This does require setting the #crostini-containerless flag.

The Terminal application will default to launching a VM named termina. To launch our VM, we will need to destroy and replace the default one. From crosh:

vmc stop termina
vmc stop baguette

# Optional: backup `termina`
vmc export termina /home/chronos/user/MyFiles/Downloads/termina.img

# WARNING: This will destroy your existing `termina` VM and any data it contains.
vmc destroy termina

vmc export baguette /home/chronos/user/MyFiles/Downloads/baguette-nixos.img

vmc create --vm-type BAGUETTE \
  --size 15G \
  --source /home/chronos/user/MyFiles/Downloads/baguette-nixos.img \
  termina

# Optional: destroy the other `baguette` VM
vmc destroy baguette

vmc start --vm-type BAGUETTE termina

Using Baguette in “Terminal” is a bit wonky and shows that it is currently under development. The ChromeOS team will not consider it stable until Chrome 143.

In my experiments, I noticed that congierce will request maitred to configure the VM using a default username (the one displayed when “Configuring Linux” from the settings). Trying to use a custom username seems to be hit or miss.

I typically rename the VM to termina and then ditch “Terminal” and just use crosh.

The Debian image allows passwordless sudo. The default NixOS configuration in nixos-crostini replicates the approach, so that you can use escalate privileges to rebuild your configuration from within the VM.

In my configuration, I prefer to SSH as root to passwordless sudo. This way, I can use a hardware key to prove my physical presence and login as root, but an attacker cannot automatically escalate privileges.

Conclusion

Getting Baguette and NixOS to work together required a bit of trial and error to build the image in the right format, figure out a few quirks, and adapt to ChromeOS’ CLI updates. I am now satisfied with the result: I wrote this blog post from Baguette and I couldn’t tell the difference from LXC.

I don’t run Kubernetes (which seems to be one of the biggest pain point for LXC users), but Baguette improves a few things for me as well:

In addition to being able to automatically forward USB devices, Baguette does not hold an exclusive lock on USB hardware keys. So I can use them both in Baguette and in ChromeOS (as a passkey) at the same time without having to fiddle with crosh.
A containerless VM has better access to the underlying hardware and better control of its init. This might make it easier to implement ephemeral storage and seems to fix an issue with pcscd that would make it fail reading from Yubikeys after some time, until restarted.
Using crosh + vsh makes it easy to attach/detach USB devices and manage the VM itself without breaking flow when switching from “Terminal”. I could have done the same with LXC containers, but I didn’t know about the vsh command.

Thanks for reading, and ‘til next time! 👋

I was experimenting with all this on the ARM-based Chromebook that I use for couch-computing. ↩
Because I wanted the build scripts to run within the default penguin image, I also overrode the derivation so that it falls back to emulation when /dev/kvm is missing. ↩

Nuggets of psychology

2025-09-08T00:00:00+00:00

Understanding the world requires seeking good explanations, not fooling oneself, and figuring out why things truly happen. Only this way we can do better next time.

In my quest to achieve this, I have found that many mental models are rooted in psychology. They start right there and lean into marketing, behavioral economics, product management, negotiation, and even how we approach everyday tasks.

This living post collects those nuggets of psychology I wish to remember and add to my latticework of mental models. I have learned them from multiple authors and books: Thinking Fast and Slow, The Paradox of Choice, Poor Charlie’s Almanack, and more. In describing them here, I will make simplifications that might make experts shiver. If you find something that interests you and you want to dive into it, start out with those books – or better yet, the original research. And if you spot any mistake or over-simplification, please do reach out and let me know!

The paradox of choice
Availability bias
Anchoring
Framing
Prospect theory
Sunk costs
Inversion
The man with the hammer
Confirmation bias
The hedonic treadmill
Perception of past experiences
Incentives
Interest, not reason
Seven, plus or minus two

The paradox of choice

It is much easier to spend several hours binge-watching a TV series than to pick a good movie and commit a couple of hours to it. Why? Because freedom of choice can lead to difficult decisions and, sometimes, to less happiness.

If you have already watched a few episodes of a show, you know what to expect from it. If you have previously enjoyed it, you expect to continue doing so. Instead, modern streaming platforms encourage us to pick a movie among hundreds. It is much easier to decide not to pick and just watch another episode of that familiar show.

When we do try picking something, we face uncertainty and we fear regret: “What if I pick a worse movie than the TV show?”. Even when exclusively focusing on movies, we will compare two or more options on multiple features (genre, duration, actors, etc.). No movie will clearly win across all dimensions, but the act of weighting the options will be enough to frustrate us, so that we will be less satisfied, regardless of what we choose.

Forcing people to make a decision along a trade-off will make them unhappy and indecisive. When applied to product management, this teaches us that more choices are not always better. With limitless choices we might achieve better results, but at the cost of significant struggle and energy to sort among the different options. This is particularly bad for maximizers, who, as opposed to satisficers won’t settle for something “good enough”.

Availability bias

Kahneman and Tversky originally termed it availability heuristic because we take this mental shortcut to save energy: when something is readily available or very vivid in our memory, we think it is also frequent or important. I prefer calling it a bias because it hinders our ability to make good decisions unless we take measures to do better.

I think of this bias also when we are blindsided by information or options that are in front of us, but fail to see that with a bit more digging we would find much better options.

Availability bias is also why we misestimate facts. For instance:

That deadly car accidents are more frequent than disease-related deaths (they are not).
Why we tend to form judgements or stereotypes based on extremely small (and likely non-representative) samples.
Why we attribute to our egocentric self more merits or faults than we really should.

Anchoring

Hi dad, can I have 100 dollars?

80 dollars? What do you need 50 dollars for, kid? Here, take 10 and bring back some change.

We make decisions based on relativeness and reference points. That is why anchoring can be powerful or lead to biased decisions. The number we hear first in a negotiation sets a reference point and we judge what comes next against it. If someone successfully manages to place an artificial anchor, then they can bias the decision-maker into making that direction.

Some stores seem to always offer a discount on merchandise. The original price is typically shown prominently and serves as an anchor which makes the discounted price look like a bargain. In a negotiation, an extreme “opening” has better odds to the desired outcome than a moderate proposal, closer to one’s true value.

Framing

The way information gets presented influences how we make decisions about it. This is why it is more effective to advertise “a discount on credit cards” instead of a “surcharge on cash”. This is also why when juries are asked to choose one parent for childcare, they will look for markedly good traits. If they were asked to discard a parent as being ineligible, they would look for markedly bad traits.

Prospect theory

When faced with the choice of getting $100 or having a 50% chance of either getting nothing ($0) or $200, most people will choose the sure choice. Even though they have the same expected value, the “psychological” value of $200 is not twice as much as the one from $100. For this reason, regardless of the same expected value, people will not take the bet. Things are not symmetrical. As humans, we tend to be risk averse.

There is diminishing marginal utility in satisfaction. This is also why $100 provide different satisfaction to someone with a thousand and a million dollars in their bank accounts.

When facing losses, things do work symmetrically, but not the way you would expect. If asked to surely loose $100 or flip a coin and either lose nothing or $200, most will choose the coin flip. Why? Because losing $200 does not hurt twice as much as losing $100: there is a decreasing marginal disutility of losses.

Lastly, losing $100 will produce, in absolute terms, a much stronger reaction than winning $100. We are loss averse. This applies even to small, symbolic losses as small as a few dollars. Loss aversion also explains why, on average, few people return products after they have bought them, even if return is free. They would experience more loss than the pleasure of getting it in the first place. This is the endowment effect.

Sunk costs

You are months into an expensive project at work, only to discover that you are not building the right thing, or that no one will use it. Do you keep working on it, or do you scrap it? The work or money you have put into is already sunk. Dropping the project would be the right thing to do – completing it, would not benefit anyone. But loss aversion will sometimes trick us into continuing that work, worsening an already non-optimal situation.

This recently happened to me. I had invested significant time, money, and effort into training for a bicycle competition. A few weeks before the race, I developed a small injury which prevented me from enjoying cycling and put competitiveness out of the picture. Due to the sunk costs, I considered racing anyways, almost surely dropping out and, possibly, making the injury worse. Luckily, I decided to ignore sunk costs and make a decision based on present conditions. Time and money were already gone and I now had to think about recovery and future rides.

Inversion

This is a tool, not a fallacy, where we think backward rather than forward, succeeding by avoiding mistakes instead of seeking brilliance.

By inverting “How do I get there?”, picture where you want to avoid going and then describe what it would take to get there. This is the principle inversion. It helps because it allows us to look at problems differently and leverage the ability of our minds to reduce and simplify rather than to add. It allows us to first figure out what we would define bad outcomes and then prune any decision-tree branch that would end up there.

When building proofs by contradiction, mathematicians often use something similar to inversion. They assume that what they want to prove is false and use logic to find a contradiction – indicating that what they are trying to prove is true.

The man with the hammer

A Charlie Munger’s favorite, this fallacy warns about how having only a limited set of mental models will encourage you to use the wrong one for the job. Somehow, this relates to availability bias since “shinier”, more available tools will lure you into thinking they are the most appropriate.

This is the reason why I am collecting several tools here and why I like learning from unfamiliar disciplines (e.g., biology, systems thinking, psychology).

Confirmation bias

We often make this mistake when navigating through available facts by prioritising information that supports our belief (or hypotheses, etc.). This applies to search, recall, interpretation and “favoring”. Essentially, we select the information that fits outs view of the world.

It typically happens without us realizing it and we fall for it because it makes us save energy plus it makes us feel good about ourselves. To fight it, we should first be aware it exists (exactly what we are doing here) and then try to actively falsify our hypotheses instead of looking for information that confirm them.

Social media “bubbles” create resonance chambers that make confirmation bias insidious. Social media timelines tend to weigh more shared beliefs, which will make it more likely for users to only be exposed to other users of the same “opinions”. This compounds the effect of confirmation bias, because it actively makes it harder to find conflicting information and debate them with others.

The hedonic treadmill

As humans, we will adapt to everything. The hedonic treadmill conjectures that our levels of pleasure, happiness, and sadness will return to a relatively stable level even after major life events. We “adapt” to those new levels. I think of it like “mean reversion” (described both in finance and mathematics) for our feelings.

A few studies have tested this conjecture, finding out that people would return to their “average” happiness after significant spikes (e.g., when winning the lottery) or downturns (e.g., falling victims of a debilitating accident).

It has strong pragmatic implications: it will make new things shine less over time and it might lead to exaggerate (or unconstrained) spending while seeking additional stimuli – always seeking more.

Knowing about the hedonic treadmill is a powerful way to counteract it. We should not fool ourselves by thinking that a new purchase (or a promotion, or more money) will make us happier, because we will quickly adapt to it. Instead, we can focus on those long-lasting things whose effect compound over time and bring us meaningfulness and purpose (e.g., social relationship or long-term missions).

Perception of past experiences

We rely on past experiences to make decisions about our future, but we often misremember and misjudge the past. Our memories of an event rely on:

The absolute high (or low) we experienced.
What we felt when the experience ended.

This study ingeniously demonstrated this. Two groups of patients underwent a colonscopy. In one group, the exam was made artificially longer by leaving the probe still for a few minutes before ending the procedure. Because a still probe is less unpleasant, the patients fell less discomfort when the colonscopy ended. As a result, they were more likely to come back for additional checkups than those in the other group, despite the actual exam being longer and the absolute discomfort being the same.

Incentives

Incentives are powerful system levers and keeping them in mind is fundamental in decision-making. According to Charlie Munger, we routinely underestimate their effects on behavior and forget that people will do what they are paid to do.

In his almanack, Munger tells two stories about incentives. One is the story of a new Xerox product. Despite being better than older products, it was selling a lot less. Why? Because salespeople were motivated by outdated incentives, rewarding them for each sale of an old product.

In another story, FedEx needed packages to move from A to B, every night. The task was not completed unless all packages had moved. And they needed to move fast! FedEx management tried a whole set of things to incentivize workers. None worked, until someone realized that workers were paid by the hour and were lacking incentives to complete their tasks quickly. When they started to be paid by shift, their incentives aligned with FedEx’s: to quickly and effectively complete their tasks before going home.

Interest, not reason

If you would persuade, appeal to interest and not to reason.

Ben Franklin, Poor Richard’s Almanack

Rather than pure rationality, we are often motivated by self-interest and incentives. Sometimes all this happens subconsciously: we rationalize flawed reasoning without even noticing it.

To change someone’s behaviour through product management, we need to keep all this in mind. Instead of trying airtight logic or reasons, build products or systems that support our users’ interest. Only then their behavior will change.

This has profound implications in business as well. Users won’t buy a new technology because it is shinier, faster, or better than some legacy. They will only adopt it if it aligns with their interests by letting them save or gain money.

Seven, plus or minus two

Our working memory can hold roughly seven (± two) “items” before performance declines. Once that happens, we become unable to effectively remember all the options or even differentiate among them. Grouping related items together by “chunking” them alleviates the burden on our brains.

This cognitive limitation is also known as Miller’s law. It should impact the design of products and features by ensuring that we do not overwhelm users with more than they can effectively work with – remembering that, in the worst case, five choices will already push them to their limit. Structuring information in a hierarchy helps. Simplifying the user journey and reducing the number of choices will do even better.

Code integrity for web apps

2025-09-02T00:00:00+00:00

I used to prefer web-apps to their native equivalents, especially if the “native” one is just wrapping a browser anyway (e.g., through Electron). Here is why:

Web-apps run within the browser sandbox, adding to defense in depth. The browser is designed to execute untrusted code and prevent it from “spilling over” to the rest of the system¹. A malicious web page should not be able to access your documents or read your emails.
Native apps, instead, can access the underlying OS more directly². They can read and write the filesystem, execute arbitrary commands, connect to other machines, start whenever you log-in, etc.

A few days ago, a potential flip side of web-apps made me rethink things more thoroughly. How do we know that we are running a “honest” web-app, instead of one modified by an attacker?

Web vs native apps

The browser downloads web-apps every time we use them (unless cached). If an attacker could modify the web-app being downloaded, then they would control its context. If it’s a messaging app, they could read or send messages. If it’s a password manager, they could access passwords or secrets.

To pull this off, the attacker would have to compromise the web-servers providing the code (including any content delivery networks) or succeed in a man-in-the-middle attack. Realistically, these attacks are relatively hard to pull off at scale, but are possible for sophisticated-enough attackers.

In comparison, native apps deploy counter-measures against all this. They are typically bundled, signed, and then downloaded through a separate channel (e.g., the App Store or a link to a DMG or deb package). This allows users (or their OSs) to validate the bundle’s integrity before executing the app, assuming: 1. a public key infrastructure (PKI) to associate the application signing keys to the application developer; 2. that the signing keys have not been compromised; 3. that the user downloaded the right application in the first place. The bar for an attacker to compromise this process can be higher, e.g., if the developers appropriately protect their signing keys in cold storage.

With E2EE

Code integrity is particularly important around end-to-end encryption (E2EE). With E2EE messaging apps (Signal, WhatsApp, Matrix) and password managers (1Password, Bitwarden), the server cannot decrypt user data, but simply holds encrypted payloads. The client receives them, decrypts them, and displays the result to the user. The integrity of the client is fundamental. A compromised client can completely circumvent E2EE and leak messages, keys, and passwords.

Signal

The lack of code verification in the browser is why Signal does not offer a web client and instead provides a native one based on Electron (essentially, a web page served by a signed native app). It all boils down to verifiable distribution. This way, users download the Signal app only once, right before they install it. They do not need to “blindly” trust it, but can verify its integrity by checking that it was signed by the Signal developers³. With web-apps, instead, users download the code at every visit and they cannot easily verify what is being served to them.

Evolving standards

A better fix would be to systemically solve this by improving things for everyone by providing better security, by default. For this to work, there needs to be a coordinated effort so that browsers validate the code against a “root of trust” before executing the application, similarly to how they ensure the integrity of websites served through TLS.

While I was researching all this, someone on IRC nicely pointed me to Isolated Web Apps, which have been designed to solve this problem but also allow a wider set of capabilities for the browser (e.g., opening raw sockets). As it happens with coordinated efforts, it takes some time to reach consensus on the best approach:

Chrome is experimentally allowing Isolated Web Apps for enterprise Chromebooks.
WebKit hasn’t taken a position yet.
Mozilla has instead declined to adopt them, stating that the additional capabilities introduce new hazards and that a new, in-development standard, would be a better solution.

Despite the time it will take, I consider this great news! Bright people are working so that we will all be able to validate the integrity of web-apps. Eventually, one or more standards will emerge, be implemented, and advance everyone’s security.

For users

What does all this mean for me and you, the users? It depends on the threat model.

E2EE service: If you are protecting against “someone trying to break E2EE for a specific service”, then native apps offer an additional line of defense, preventing a modified client from running. The assumption is that we trust the application developer, because we use their E2EE service and we run their app.
Local device compromise: If instead you are protecting against “someone trying to compromise my endpoint”, then web-apps offer additional protection because of the browser sandbox. Same goes for “I do not trust the application developer and I want to limit their access to my system”.

Now, threat model (2) is a lot more general than (1). If someone manages to compromise my device, chances are that they will also be able to break E2EE for any app I use there⁴. If, instead, someone serves me a malicious WhatsApp web-app, they will be able to read messages or send new ones, but will most likely not be able to also read my emails or steal my passwords.

Password managers are a notable exception, because an attacker can leverage them to try accessing other services and increase the blast radius. But I argue that even then, MFA should prevent most damage – as long as the other factors are not in the password manager, the attacker won’t be able to log in. In addition, most services today notify users about new or unusual logins, alerting them that something might be going on. Compare this with when a device is compromised: an attacker who steals a session token might be able to use it without triggering a new login notification.

Due to all this, I consider web-apps a better fit for a majority of users (myself included), especially when used through a thin client (e.g., a Chromebook). Encouraging users not to download and install native software makes device compromise less likely and localizes the blast radius (e.g., to the specific compromised app). Isolated Web Apps (or similar) will be a welcome addition once they standardize, adding one more layer of protection to the web. Meanwhile, the Code Verify extension probably doesn’t hurt the services it covers.

Threat modeling is not one-size-fits-all, though, and each user should think carefully about what they are protecting from. For instance, if you are a whistle-blower, a leak of your Signal messages could lead to fatal consequences, more severe than someone being able to access your financial data or log-in as you on Facebook. This is why I appreciate the thoughtfulness of Signal developers, providing a client that is secure against those most-severe threat models and prevents those users from making a wrong choice.

As for me, I hope that this deep dive will help you (as it helped me) to define your threat model more explicitly and weight the associated trade-offs. If you have thoughts about all this, I would love to hear them and chat about it. Please, reach out! Meanwhile, thank you for reading and see you next time! 👋

Footnotes

Not a silver-bullet, but still useful. ↩
Most operating systems run native apps in a sandbox as well, but it is harder for users to tell when that is the case. For instance, on macOS apps will run in two different sandboxes depending on whether they are from the App Store or not. ↩
On macOS, Signal.app is signed through Signal’s Apple developer account. Debian packages instead rely on a GPG signing key. The operating system should typically perform the verification on behalf of the user. ↩
As 1Password puts it:

There’s no password manager or other mainstream tool with the ability to guard your secrets on a fully compromised device.

↩

Paddling in circles: lagging indicators

2025-07-12T00:00:00+00:00

Last weekend I went kayaking on the Ardèche river, in southern France. I had kayaked a couple of times before, but never in tandem over a two-day trip. It was a terrific experience, with wonderful people in mesmerizing nature.

As it often happens when I do outdoor sports, my mind put together a few thoughts. This time, about systems, their delays, and lagging indicators. I think they generalize to more than kayaks 🛶, so I wrote about them here.

For most of day one (out of two), my kayak partner and I struggled to navigate in a straight line. We often found the kayak pointing at the wrong angle and had to bring it back on route. Then, we would find it pointing wrongly in the other direction. Unsurprisingly, we reached camp for the night tired and with low morale.

Why were we doing such a bad job? Because of delays!

A kayak:

Gains forward speed when pushed by the paddle, backwards, on either side.
Turns in the opposite direction of the paddle in the water.

These forces complement each other. Paddling on the right side will move the kayak forward and point it slightly to the left. Paddling on the left will offset the change in direction while maintaining the speed. This is why you typically alternate paddling on each side: to maintain momentum and balance the forces that would rotate the kayak.

Occasionally, things did not work that way for us. When the kayak leaned one way, we paddled “harder” on that side to turn in the opposite direction. But, by then, it was too late already! Due to the delay between the energy we transmitted to the water and the kayak rotating, the kayak’s direction was a lagging indicator of the quality of our efforts.

The kayak in the water is a system we can model, reason about, and predict. Because no reaction is truly instantaneous, all systems have delays. Knowing where they are and how to shorten (or lengthen) them means finding a leverage point that can affect the whole system. Therefore, learning how to operate in the absence of timely information is key: both to having fun on a kayak and to making better decisions in business and life. When I realized that, the task of controlling a kayak looked a lot like a scaled-down version of the challenges we face daily and I saw an opportunity for thinking about it and connecting the dots.

We can’t begin to understand the dynamic behavior of systems unless we know where and how long the delays are. And we are aware that some delays can be powerful policy levers. Lengthening or shortening them can produce major changes in the behavior of systems.

Thinking in Systems, A Primer. By Donella H. Meadows

On the kayak, we went through phases of observation and experimentation. At first, we suspected that an asymmetry in our paddling (left/right or front/back) was at fault. We thought that if we could have perfectly symmetric strokes, then the kayak would go straight. But then, how would we account for the remaining factors, e.g. the water currents? We wanted to build a mental model of the kayak so that we could diagnose our troubles and figure out how to fix them. But we were failing, because delays were muddling the effects of our inputs and revealing them only after some time (when the kayak turned).

To get better, we had to first acknowledge that the system was messier than what we were trying to picture in our minds. Both delays and external forces were contributing to the outcome. Once we learned that, we started to master our waters. We started looking at weaker observations that were more readily available: e.g., our bow just hinting at a direction after a paddle stroke. This way, we shortened the feedback loop of actions, measurements, and reactions and we stopped over-correcting our angle when it was too late. With experience, we even started seeing how each paddle stroke would slightly turn the kayak while pushing it forward and we learned how to naturally balance for that in the next stroke. We could “point” the kayak more effectively, control our route and pick between more options in how to face the rapids. And, obviously, have a ton more fun!

So, what did we learn?

The map is not the territory. We have a natural tendency to build tidy mental models, but the world is sometimes messier than that. Acknowledge that to iteratively build a useful model.
All systems have delays, since no reaction is truly instantaneous. Delays lead to oscillations and longer feedback loops, which make systems harder to “hold in the head”.
Delays are powerful leverage points. Controlling them can result in significant (sometimes surprising) changes to the system’s behavior.
A faster loop reduces the effect of delays, which makes it easier to understand the system.
The shorter the loop, the faster we iteratively build system models. Strive for simplicity, get it working, then improve it through experience and insights.
If something is hard to predict (or correct), you might be only looking at lagging indicators. They are more visible but less useful than leading indicators. A weaker signal earlier on is useful: try looking for those leads.

These lessons apply across disciplines. Software development, for instance, combines different loops (write, review, bugfix, or release, deploy, rollback) and leading and lagging indicators (number of developers vs number of bugs released). There are delays: it takes time for a developer to onboard and become an effective contributor; sometimes, bugs only surface years after they have been coded. It is also messy: we measure team velocity and story points, but need to acknowledge how reality changes around us and the humans that compose the team. Endurance training can be another example. It takes weeks to build the aerobic skills required to compete and training and recovery interact in loops (with form and fitness). The heart rate on race day is a leading indicator, while VO₂ max lags behind.

I hope that while reading about delays and indicators you started picturing examples from your own fields where these thoughts apply. Reach out through the contacts in the footer; I would love to hear all about it!

Thank you for reading, and until next time! 👋

Root login in NixOS containers

2025-06-29T15:25:00+00:00

When working in NixOS containers under ChromeOS, the container users root and aldur have no password. This is very convenient, as it avoids managing secrets in the NixOS configuration. However, it makes securely escalating privileges tricky: both sudo and su ask for the user password, which we can’t provide.

This post details my journey to find a secure and usable privilege escalation method for NixOS containers.

Take 0: the baseline

When the container runs via lxc within the termina VM in ChromeOS, we can spawn a root shell directly from Crosh:

# Replace with `bash` if that's what you are using.
lxc exec  fish

This works well and requires no extra configuration. However, it’s inconvenient. It requires ChromeOS, and relies on access to the host VM. If ChromeOS Baguette graduates to production, this approach will not work anymore.

Take 1: insecure, passwordless `sudo`

Enabling passwordless sudo for the users in the wheel group is temptingly simple. NixOS makes it easy by setting security.sudo.wheelNeedsPassword = false. The result, however, also makes it easy for an attacker to escalate privileges and is essentially equivalent to being root all the time. Not ideal.

Take 2: unsupported, `pam-u2f`

Since I already use a Yubikey to authenticate through SSH, sign commits, and even decrypt vaults, I considered using it for privilege escalation as well. This approach would maintain a separation of concerns between the users while providing a reasonable security posture¹.

The pam-u2f module is the modern way to go for this. However, FIDO2 doesn’t currently work under Crostini, due to missing raw USB HID device access. The corresponding kernel changes have been merged upstream last year, so hopefully this will be fixed soon.

Take 3: insecure, `yubico-pam`

Yubico originally maintained the aptly named yubico-pam module, which relies on HMAC-SHA1 Challenge-Response.

This module is now deprecated, but I managed to get it working by digging through the old docs.

Setting up yubico-pam in NixOS

Two SSH keys on a Yubikey

2025-06-26T09:17:00+00:00

With NixOS containers in ChromeOS, I use an SSH key in a Yubikey to authenticate to remote hosts and keep the key separated from the container.

Creating keys on the Yubikey and ensuring good UX while using them can be tricky, but yubikey-agent makes it seamless. There’s a catch though! It supports one SSH key only, while I’d like to use two:

One to authenticate (e.g., to GitHub).
The other to sign git commits.

This way, I can implement separation of duties and can “decommission”¹ each key independently.

A few PRs in the yubikey-agent repository add support for multiple keys and even different policies (touch, PIN). Most try to develop a generic approach, while I only need to support two keys. So, I decided to scratch my own itch and implement a solution.

Luckily, the agent is written in Go, is easy to understand. The piv-go does the heavy lifting. Under the hood:

It asks the Yubikey to generate an elliptic curve (ECC256) key pair in the PIV Authentication slot.
Then it creates a self-signed X509 certificate to wrap the public key and make it available to clients (our SSH agent) through the PKCS#11 interface.

This Yubikey tutorial describes roughly the same process, but I find the Go code easier to read and more precise. In addition, the setup guides the user through the PIN/PUK setup and removes the need for a management key by delegating its control to the PIN/PUK.

To support two keys, I decided to go the simplest approach and make the minimum set of changes to the code to generate a new key (during the setup phase) and then serve it through the SSH agent. I initially considered generating the key through the CLI, but the delegation of the management key made it hard to do, so I just went with code for the setup as well. By poking around the piv-go code and the PIV standard, I decided to use slot 9c, which is used for Signature (the Yubikey docs even mention using it for git commit).

The result is in my fork of the project. I intentionally kept the code as simple as possible, so that it is easy to maintain and follow upstream (in case I need to). If you want to try it out, setup the Yubikey as usual and then run yubikey-agent -setup-sign to generate the additional key. When running the agent, it will gracefully try loading both keys and ignore the one for Signatures if it cannot be found.

I have also ensured that the ordering of the keys doesn’t change and have configured my git client to sign with the second key returned by the agent. This is convenient to use and matches my configuration on other hosts.

Here is the result:

[I] aldur@lxc-nixos ~> ssh-add -L
ecdsa-sha2-nistp256 AAAAE2V...iUkW4JQUDA= YubiKey #25972834 PIV Slot 9a
ecdsa-sha2-nistp256 AAAAE2V...pKYi/Zh/HA= YubiKey #25972834 PIV Slot 9c

To deploy my changes, I wrote a small Nix overlay that applies my patch:

(final: prev: {
  yubikey-agent = (
    prev.yubikey-agent.overrideAttrs (old: {
      # Used a patch instead of overriding the source so that it will keep
      # working (or explicitly break) on upstream updates.
      patches = (old.patches or [ ]) ++ [
        (prev.fetchurl {
          url = "https://github.com/aldur/yubikey-agent/commit/f7a6769fd832a867e62228c8ddb0133174db64bf.patch";
          hash = "sha256-swQb3N89yAJSQ4pkUq2DDKvEFBlzhr/tbNMdC2p60VE=";
        })
      ];
    })
  );
})

Ideally I’d want to revoke keys, but there is no built-in way to do it for SSH (that I am aware of). The PIV standard relies on certificates, so there might be a way to revoke them – but then, consumers would need to check a revocation list, and I don’t think they do. GPG keys can be revoked, but come with other downsides. ↩

Universal Bits

Sandboxing local models on macOS

Sandboxing

Sandboxing profiles

Server

Local clients

In a QEMU VM

Test drives

Footnotes

Why my internet dropped like clockwork

Impermanent NixOS VMs in ChromeOS

What is impermanence

Erasing /home

Settings things up

Playing nicely with home-manager

Wrapping up

Footnotes

Effective salespeople

1. Realize not everyone wants to buy what you’re selling

2. Get comfortable being uncomfortable

3. Prove that you are an expert

4. Manage expectations

5. Add value first

6. Make scarcity work for you

7. Let the other party sell themselves

Closing thoughts

NixOS Baguette images in ChromeOS

Background: ChromeOS VMs

Baguette NixOS images

How-to: Make it yours

How-to: Additional shell sessions

How-to: USB forwarding

How-to: Launch NixOS from “Terminal”

How-to: Root login

Conclusion

Nuggets of psychology

The paradox of choice

Availability bias

Anchoring

Framing

Prospect theory

Sunk costs

Inversion

The man with the hammer

Confirmation bias

The hedonic treadmill

Perception of past experiences

Incentives

Interest, not reason

Seven, plus or minus two

Code integrity for web apps

Web vs native apps

With E2EE

Signal

Meta

Evolving standards

For users

Footnotes

Paddling in circles: lagging indicators

Root login in NixOS containers

Take 0: the baseline

Take 1: insecure, passwordless sudo

Take 2: unsupported, pam-u2f

Take 3: insecure, yubico-pam

Two SSH keys on a Yubikey

Erasing `/home`

Playing nicely with `home-manager`

Take 1: insecure, passwordless `sudo`

Take 2: unsupported, `pam-u2f`

Take 3: insecure, `yubico-pam`