A little over a year ago, I migrated the bulk of my HomeAssistant automations from NodeRED to DigitalAlchemy, a TypeScript based automation system. Since then, I've slowly been adding more and more automations, of varying complexity. For simple ones, the built in event-based system works well enough, but as soon as you start having to track state across a few different entites, it becomes a big, unwieldly mess.

State Machines?

This is a problem encountered in a lot of other domains, and the solution is often to use what is known as a finite state machine. Finite State machines are a whole topic on their own, with plenty of ink spilled, so I will only skim the surface, but essentially they let you model a real world system as a series of states and transitions. States are things like "on", "off", "open", "closed", etc. Transitions are the "in-betweens". "turn on" would be a transition from "off" to "on", for example.

In the Elixir/Erlang world, a lot of things are modeled as state machines. GenServer, which a lot of people consider the building block of most Elixir/Erlang applications, is a kind of state machine. While it lacks the formalities that let you rigidly define transitions and states, the primitives are all there. There's even an erlang module called gen_statem which allows you to very easily build full fledged state machines.

Since DigitalAlchemy is typescript based, I can't use gen_statem, but there are a number of decent Javascript statemachine libraries out there. I settled on xstate, which is rather mature, and ties into the stately.ai platform, which has both a robust VSCode extension, and a decent web interface (more on this later)

Basic XState usage

XState is rather simple, once you get the hang of it. You define a state machine, and then create an actor from said state machine. The actor is an instance of the state machine. You can have multiple instances of the same state machine.

You trigger states by sending events to the actors. Depending on the current state, the events can trigger transitions, or do nothing at all. You can attach actions to transitions, as well as to entering/exiting a state. Using this, you can program in desired behavior, limiting potential states, and ignoring undefined behavior. For something like HomeAssistant, that means you don't have to explicitly code around "undefined" and other states that you don't care about.

Using XState in DigitalAlchemy

DigitalAlchemy doesn't really do anything out of the ordinary that would prevent you from using XState. In most cases, you just install it to your project, import it, write your state machine, and its off to the races.

Mailbox monitor

Recently, I built a simple mailbox notifier, using ESPHome. My mailbox has a top door and a bottom door, and I monitored both using reed switches. The mail carrier will always use the top door, as its the "incoming" mail, and I'll always use the bottom door to retrieve the mail, as its the locking mail bin.

Since the monitor is running on battery power, I used the deep sleep functionality of ESPHome. This means that the device is off most of the time. Since it takes a little bit of time to wake up and connect to HomeAssistant, there is a chance it misses a trigger. If the mail door is opened and closed before we can connect up to HomeAssistant, HomeAssistant will see the door as "closed", and we won't know which door triggered. Since ESPHome has no concept of "store and forward" for events, we have to handle this in a round-about way. We add two additional binary sensors¹, one for each door, that listen to the main door sensors, and if they see them go true, they themselves go tru, and then hold that state until the next shutdown, which triggers at the end of the wakeup period. That way we can trigger off either the main door, or the "sticky" sensors. If you want to see the ESPHome YAML for the mailbox, its available on my github

Over on the HomeAssistant side, we get a nice little device that has 4 entities we need to track:

• binary_sensor.mailbox_top_door: the main top door sensor
• binary_sensor.mailbox_bottom_door: the main bottom door sensor
• binary_sensor.mailbox_top_door_sticky: the sticky top door sensor
• binary_sensor.mailbox_bottom_door_sticky: the sticky bottom door sensor

We will also create a couple entities on the HomeAssistant side, to light up an icon on our dashboard indicating new mail, and to reset the state of our system, should something go wrong. We'll call one binary_sensor.new_mail, and the other button.mailbox_reset.

We can now actually createa a fairly simple state machine, in home assistant, without using DigitalAlchemy or xState, just by listening to the sensors above and using the binary_sensor.new_mail as our state tracker. For this case, it will work about the same. But we're going to use DigitalAlchemy and xState anyways, because for more complex cases, you can't just rely on a single state tracker, and it can get ugly quickly.

If we model the states of our mailbox monitor as a flow chart, we get something like this:

We can see that the only way to get to New Mail is from the Top Door Opened event, and we can get from New Mail to No Mail from either the Bottom door opened event or the Reset Button Pressed event. Pressing the reset button, or opening the bottom door, while we are in a no-mail state does nothing, and so we don't have to deal with it. Our state machine will just ignore the event as an invalid transition

One of the coolest features of xState is that you can actually build your code using a flowchart. You can use the online editor, or the VSCode plugin. I use the VSCode plugin, which is pictured in the above screenshot

On Each state, we encode some actions in the entry. Entry lets you say "Any time this state becomes active, do this". For New Mail, we use Entry actions to send us notifications and to turn on our dashboard indicator. For No Mail, we use it to clear the notifications and turn our indicator off.

The final State Machine code looks like this:

const machine = setup({
  types: {
    context: {} as {},
    events: {} as
      | { type: "Top Door Opened" }
      | { type: "Bottom Door Opened" }
      | { type: "Reset" },
  },
  actions: {
    notify: () => {
      notifier();
    },
    clearNotify: () => {
      clearNotifier();
    },
    indicatorOn: () => {
      new_mail.is_on = true;
    },
    indicatorOff: () => {
      new_mail.is_on = false;
    },
  },
}).createMachine({
  context: {},
  id: "Mailbox",
  // initial: new_mail.is_on ? "New Mail" : "No Mail",
  initial: "No Mail",
  states: {
    "No Mail": {
      on: {
        "Top Door Opened": {
          target: "New Mail",
        },
      },
      entry: [{ type: "clearNotify" }, { type: "indicatorOff" }],
    },
    "New Mail": {
      on: {
        "Bottom Door Opened": {
          target: "No Mail",
        },
        Reset: {
          target: "No Mail",
        },
      },
      entry: [{ type: "notify" }, { type: "indicatorOn" }],
    },
  },
});

In the actions block, we make a few calls to some functions we created for handling notifications, as well as setting some properties on DigitalAlchemy proxies. To tie our state machine into our actual monitor, we just need a bit of glue code, that takes state changes from HomeAssistant and uses them to trigger events on our state machine, which will trigger transitions.

const topDoorAction = ({ state: newState }) => {
  if (newState == "on") {
    mailboxActor.send({ type: "Top Door Opened" });
  }
};

top_door.onUpdate(topDoorAction);
top_door_sticky.onUpdate(topDoorAction);

const bottomDoorAction = ({ state: newState }) => {
  if (newState == "on") {
    mailboxActor.send({ type: "Bottom Door Opened" });
  }
};

bottom_door.onUpdate(bottomDoorAction);
bottom_door_sticky.onUpdate(bottomDoorAction);

reset_mail.onUpdate(() => {
  mailboxActor.send({ type: "Reset" });
});

We create two anonymous functions, topDoorAction and bottomDoorAction, and then use them in the onUpdate handler for the 4 entities that can trigger events in our system. We also use a similar pattern for the reset button. Each handler function just calls our mailboxActor and sends it an event, which our above state machine definition defined. The state machine listens to those events, and if they can trigger a transition, they do, and we call the appropriate actions.

The full, working code example of the mailbox is here, and the previous, non-state-machine driven version is here

3D Printer Automation

My 3D Printer is connected to a smart switch, which I use to turn off power to the printer when it's not in use. This is a mix of a safety precaution and a bit of energy saving. It's a safety precaution to prevent the printer from doing things without my initating them, and an energy saving meeasure to prevent the device from drawing power while idle. I want to have the switch turn the printer off after a few hours of inactivity. We can define inactivity as any time the printer is not printing or running a filament dryer.

Modeling our state machine in xState, we get something like this:

It's a lot more complicated than the Mailbox monitor, but can be broken down into a few main "things"

• The printer is idle when it is not printing or drying
• The printer can be both printing and drying at the same time
• Printing has multiple sub states that should be considered "active"
• Drying has only one state that should be considered active, "drying"
• Both printing and drying can be "idle"
• We want to have an initial state that is unsynced, where we don't actually know the state of the printer
• At any time, we can transition to power_off, because the real world has things like power failures or users hitting the e-stop button
• power_off can only transition to idle, because the printer has to boot up before it can resume a power outage print, or anything else

Key things to call out in this state machine are the use of actions on transitions, guard clauses, "after" transitions, and parallel state machines.

Actions on transitions let us perform something whenever a particular transition is triggered, and only when that transition is triggered. On the startPrinting transition, we reset an energy meter, so I can see how much power the current print has used. Similarly, on the transitions out of the printing state, we typically fire off notification handlers. Note that we are not using any entry/exit actions in this state machine, they simply don't suit our needs here.

Guard clauses are used to check the status of both child state machines (printing and drying) on the active state machine. We have this guard clause set in what's known as an always transition, a transition that will fire on every single other transition involving the active state machine, which is all of the transitions of its children, and transitions on itself. The guard clause prevents it from firing when some condition isn't met. In this case, we have the guard clause set to check if both the printing and drying state machines are idle. If so, we can transition back to our parent machine's idle state.

"After" transitions let us fire a transition if a state machine has been in a particular state for a duration of time. We use it here to shut down our 3D printer's smart plug if we've been in the idle state for a few hours. If we transition out of the idle state at all, for any reason, then the timer is cancelled, and will start from the top next time we enter the idle state. This transition has an explicit action tied to it, which turns off our smart plug. This is the only time the state machine will turn off the smart plug

Finally, there's a top-level transition, which always listens for a turnOff event. Should our smart switch be turned off at any time, for any reason, we can transition our state machine to power_off, and have it match reality.

Before I moved this to a state machine, I had a fairly complicated bit of code to track the status of a 3D printer. Whenever the printer was idle, and the smart plug was on, I would start a timer. When the timer finished running, it would check the state of the 3D printer, and if it was in a "good" state (not printing, not drying), it would turn off. If I started a print job or a drying job, the timer would be cancelled, to be resumed later. This was rather fragile, as "printing" is a whole progression of states of the printer, and handling the corner cases, such as what happens when a print/dry job finishes while a longer print/dry job is still running, became maddening. There was also some speical logic around drying, as that gives us an end time, but it never really worked all that well.

You can see the full state machine and associated digital-alchemy code on github, and what it looked like before.

That looks an awful lot like NodeRED

Kind of! Flowchart based programming can be useful, particularly for things like state machines, but I still find it much easier to reason about this than NodeRED. NodeRED had some weird constructs, that I just haven't had to work around in this. Since there's always "real code" right there, I never felt as constrained as I did in NodeRED, where I'd frequently just toss a Javascript action in there to get something done when I couldn't figure it out.

The door and opener.

I have hens. A decent amount of hens. And one thing you have to do when you have hens is secure them at night. Most people will agree that chicken is tasty, and most predators share that opinion. So a big hen house full of tasty birds is a tempting target for any number of raccoons, foxes, cats, opossums, and more.

When we first got hens, we kept them in a small coop we ordered online. It wasn't particularly well-made, and the doors were flimsy. But it was good enough for our first year of birds, and by year two we knew we needed something better. To build a better coop, I laid a concrete pad, stuck a Costco plastic shed on it, and cut a hole in the side for a door. Our local farm store sold a simple automatic door, which mounted over the hole, had a rudimentary set of programs, and did the job reasonably well. This unit gave us 3 good years of service, and would likely have given more. But there were a few issues that I wanted to address.

The problems

During the summer, we have a lawn care company come by every month to spray various treatments on the lawn. Things like fertilizer, pesticides, the usual suburban lawn treatments. They usually give me 24 hours notice they'll be heading out to our property, so the night before I can disable the next morning's auto open feature, keeping the hens inside for a reasonable time after whatever substances have been applied to the yard have had a chance to soak in. The door has no remote control features, all interaction has to be done at the control module. Its interface is reminiscent of early 2000s electronics, with lots of holding buttons down, navigating menus on a single line dot matrix display, and lots of frustrating repetition.

During the shoulder seasons, the weather can be unpredictable. Sometimes we get snow as early as September, other times we don't have snow on the ground until January. We like to let the birds out as much as possible, but don't want to wake up and have hens freezing in the snow. So we typically disable auto-open in mid-fall, and re-enable it in mid spring. But if we have a long, warm fall, or a warm, early spring, we have to manually open the door every day to let the birds out, until we decide to switch to automatic opening.

As far as programmable open times, the unit has a few modes, aside from manual opening. The obvious mode is time based, which uses a little RTC on the device to keep time, and opens or closes at preset times. Works well enough, even with clock drift. Chickens aren't particularly picky about time. The other mode is sensor based, which uses an internal light sensor to determine the sun rise and set times. We primarily used this one, as the rise/set time changes over the course of the summer, and having it set at something like 9 PM might be too late for the edges of the season, but too early for the middle. But since it's a light sensor, it depends on the ambient lighting. Stormy days, or even just heavily overcast days, would trigger the sensor early. The hens seem to know the difference between a cloudy sky and sunset, and so we'd sometimes find hens waiting patiently outside a locked door.

Things the existing door did well

Looking at how the existing automatic door opener worked, there are a few very clever choices that were made, that I wanted to keep. The controller and motor don't directly drive the door, rather they use a short length of cable to raise/lower the door. The door itself has a latching mechanism that engages when it hits the end of its downwards travel, and disengages when the cable begins to lift. So all the motor has to drive is a winch. An additional benefit of this is that there is minimal chance of hurting a chicken if she blocks the door. The door itself is rather light, a couple pounds at most, and so if the full weight of it lands on a bird she can shrug it off.

I've seen some other door opener designs that use direct-drive motors (like a garage door) or linear actuators, and then have to have added complexity in detecting door blockages and handling them.

This should be simple, right?

So, keeping what the existing system did both right and wrong in mind, the solution, to someone who has recently been a bit obsessed with Home Automation, particularly with ESPHome, was obvious. Build a new door opener, one that could be controlled via Home Assistant. Building atop this platform gives me full remote control, from a system I already use, as well as some robust scheduling primitives, like sunrise/sunset times, ambient temperature, and anything else you can think of.

To enable such an opener, I'd need a few things:

• A control module: Some ESP32 variant. It would need to have an external Wi-Fi antenna, since this will sit out in my yard, and I wasn't sure about how strong my Wi-Fi signal would be that far. Seeed Studios Xiao ESP32C3 fit the bill nicely.
• A Motor: I probably overestimated the specs on the motor, but I wanted something that would turn reasonably fast, have a fair bit of torque, and use a gearbox to get as much power as possible. Amazon is covered in motor modules for DIYers, and so I picked this one, based on little more than the fact that it was 12V, 100RPM, and small-ish.
• A way of controlling the motor: I know you can't drive that sort of motor directly from a microcontroller, and so I needed a way to control it. A simple on-off relay wouldn't work here either, as I needed to be able to reverse the polarity to drive the motor both clockwise and counter-clockwise. A h-bridge fits the bill perfectly, and this particular unit also outputs 5V, for powering a microcontroller
• Wi-Fi Antenna: for obvious reasons
• A weatherproof button:, to control the door manually
• A power supply: More on this one later, but I had to revisit this point a couple of times.
• An Enclosure: I have a 3D printer. I have lots of filament. This was probably the easiest part.
• A spool for the cable: Again, 3D printer

Components!

The enclosure

The enclosure started off as a big box, and ended up as a big box. There's not that much you can do different in this space. But there are some subtleties.

The spool sits in its own little mini-enclosure in the box, with a hole for the cable to pass-through. While this won’t prevent everything from getting in, it does separate the electronics from the outside.

For all the external connections the unit will need, I created a pass-through for a Deutsch Connector. I have a bunch of these on hand from other projects, and they're a personal favorite. Deutsch sells a bulkhead connector, but I didn't have any, so I made my own bulkhead, which can be anchored into the main enclosure using some M2.5 screws and a few heat-set inserts. For weatherproofing, a TPU gasket is used, that sits between the bulkhead and the enclosure. The antenna and the button both have pass-through holes in the side. The button has its own rubber gasket, while the antenna just uses an o-ring. I also added a little "roof" over each, to slightly increase the weatherization. Probably not necessary, but it looks nice. Finally, the top of the enclosure has a shadow line where it meets the top piece, with a gasket, to further aid in gasket. And then the top itself is held down by 4 M3 screws, engaging with heat-set inserts in the main enclosure.

Since I wasn't entirely sure about how well the 3D printed parts would fit the Deutsch connectors, I made that bulkhead its own part. While my initial print fit the 2-position connector I was initially using, having the bulkhead be a separate part proved to be unusually prescient.

Putting it all together

Getting everything put together on a breadboard was rather easy, and after a quick flashing of the ESP32 with ESPHome, a bit of yaml, and some inhalation of solder fumes, I had a primitive prototype. I'd click the door open button in HomeAssistant, the motor would turn on, spin for a bit, then turn off. Attaching a spool and string let it actually wind and unwind the string, and putting a 5lbs weight on the end let me estimate how much current it might draw in the wild. Everything was working far too well.

The initial ESPHome component I used was a Template Cover. The motor controller is tied to a couple GPIO pins, and so this template simply turns the appropriate pins on and off, with some delay actions to turn the respective pin off after a time delay. This is pretty much exactly how the existing door opener worked, but while mulling things over, I realized that I'd prefer to have some form of feedback from the door, so I can tell if it was open or closed. Adding this feedback to the system lets you use the aptly named Feedback Cover, which handles most of the automation I had written by hand in the template sensor, as well as letting you do things like open the door an arbitrary percentage. The feedback cover is still mostly time based, but lets you put some end stops in, that will stop the motor when it reaches the end of its travel, regardless of how long it took (within limits, I always left some upper level time limit on there to prevent suck doors from breaking things).

But adding endstops means I needed to add more components. I decided to use magnetic contact sensors, like you'd use for a door or window in a burglar alarm system. I could stick the magnet on the door, and stick the sensors on the rail it travels along, and get a good indication of when the door is open or closed. These sensors are simple, when a magnet is near, they complete a circuit. Wiring them up to everything is rather simple. One side connects to a GPIO pin, the other connects to ground, and then with a bit of ESPHome yaml, you now have a sensor. I wanted all the connections to be inside the weatherproof box, and so I needed a way to bring in 4 more wires. The 2-position Deutsch connector I was using wouldn't cut it. So I grabbed a 6 position connector, printed a new bulkhead plate, crimped some connectors on the end of the sensor lines, and wired their corresponding positions to the board, and now had feedback. The motor would spin until the endstop for the direction of travel closed, and then it would stop.

Everything worked, so I moved all the connections from a breadboard connection to a hard-solder connection, put everything in the enclosure, sealed it up, and then waited for the weather to warm up, so I could install it on the hen house.

Getting power to it

The trench, sitting empty.

I'd decided early on that this opener would run off mains power. I didn't want to have to deal with batteries, deep sleep, or anything else. I needed to get power out to the hen house, the old extension-cord across the yard wasn't going to cut it, for running a water heater in the winter, and a fan in the summer, and so this served as a good excuse. I have an exterior outlet with a junction box on the side of my house, and so getting power from the house across the yard to the henhouse was a fairly simple problem, with a simple solution: Dig a ditch, run some conduit in the bottom, and pull power.

Simple is only a conceptual word here, as the actual project proved a bit more involved. I rented a trenching machine to dig a 3' deep trench from the house to the coop, and in typical trenching machine fashion, the clutch was busted. Judicious use of a spring clamp and a piece of rebar managed to keep the clutch engaged, and within an hour I had a nice deep trench across my yard. And, of course, I hit some sprinkler lines. 3 of them to be exact. Fortunately, I didn't hit the sprinkler control wires, just a main water line and two zone lines. I keep the system depressurized when it's not in use, and so fixing the breaks wasn't complex, but it was a dirty project that took the better part of an afternoon. Once the sprinkler line was fixed, laying the flexible conduit was trivial, and filling in the trench wasn't much harder.

Filled in, but still ugly. Grass will eventually move in.

Pulling 3 conductors through the conduit was the usual pain, fish tape and pulling lubricant made it a bit easier, but I'm never a fan of pulling wire through conduit. Beats pulling it through walls, but not by much. I also ran a line of CAT-8, unterminated, if I ever want to add a Wi-Fi extender, cameras, or anything else out at the coop. At the coop side, I put in a large box, put a few outlets inside it, and then I put a small box with an in-use cover on the outside, so powering heaters and fans is a matter of plug and play. Connecting up the wires at the house side, everything read green.

Mounting the opener, and power troubles

Mounting the opener was easy enough, I just unscrewed the old one, screwed the new one in, tied the cable to the door, ran the power lines from the large electrical box, where a wall-wart transformer turned the 120VAC to 12VDC, and voilà, a working chicken door opener.

Or so I thought. The door closed just fine the first night, and opened just fine the next morning. But the next evening, the door sat firmly open past sunset. Checking the device in HomeAssistant, it showed several log entries of the device rebooting. At first, I worried it might be an issue with an electrical short, so I manually closed the door, and checked the wiring the next day. Everything was fine, and so I enabled some debug logging, specifically setting up a text sensor to tell me the previous reboot reason, and just left the unit to do its thing. The door opened and closed just fine for a few days, and then failed to open one morning. Checking the logs, I saw the same rebooting issue, and this time the debug sensor told me the unit rebooted due to brownout.

The cheap wall wart power supply I used, which claimed to deliver 30W of power was capping out at 15.6W of power, meaning it was seeing voltage drop whenever the motor tried to move. Frustrated, I grabbed a spare 50W Meanwell power supply I had left over from some xmas light projects, wired it up, and put everything back together. Since then the door hasn't given me any troubles.

The Code

I've more or less glossed over the code up to this point, as its only marginally interesting. The full version is on my github, which should always be updated, but here are some highlights I thought could be useful to others.

The primary motivational factor of this whole project was being able to turn off auto open at any time. This is handled through a simple template switch, which shows up in HomeAssistant:

switch:
  - platform: template
    id: auto_open
    name: "Automatic Open"
    optimistic: true
    restore_mode: RESTORE_DEFAULT_OFF

This switch's value is checked during the auto-open automation on the sun component, which also handles a delay. Since the hens don't always go to bed exactly at astronomical sunset (when the sun is just below the horizon), I wanted to add a bit of delay to the door closing, so that the birds get in there. Initially, I hardcoded this delay, but after a few tweaks, I got tired of having to recompile and reflash every time I needed to tune it. So I added a few number inputs, which let me set the delay in HomeAssistant, and then have the ESPHome use them for delays when sun events happen:

number:
  - platform: template
    name: "Open Delay"
    id: open_delay
    min_value: 0
    max_value: 600
    unit_of_measurement: "min"
    mode: box
    step: 1
    optimistic: true
    restore_value: true
    initial_value: 30
    icon: mdi:weather-sunset-up
  - platform: template
    name: "Close Delay"
    id: close_delay
    min_value: 0
    max_value: 600
    unit_of_measurement: "min"
    mode: box
    step: 1
    optimistic: true
    restore_value: true
    initial_value: 15
    icon: mdi:weather-sunset-down

Finally, the sun component is where the actual automation happens. The on_sunset event is fairly simple, and fires unconditionally. If the door is ever open, I want it closed at sunset (accounting for the delay, of course). The on_sunrise event has a logic check to see if automatic open is on, and if so, it waits for the delay from the number component, then closes

sun:
  latitude: !secret latitude
  longitude: !secret longitude
  on_sunrise:
      - then:
        - if:
            condition:
              lambda: "return id(auto_open).state;"
            then:
              - delay: !lambda 'return id(open_delay).state * 60000;'
              - cover.open: chicken_coop_door
  on_sunset:
    - then:
        - delay: !lambda 'return id(close_delay).state * 60000;'
        - cover.close: chicken_coop_door

For diagnostic information, and because it's fun to see, I also added a couple text sensors that return the next open/close time. These are a wee bit more complex, as they have to translate the text timestamps the sun text_sensor component returns into epoch time, do some time math, and then convert them back to timestamps, but it's still fairly basic:

text_sensor:
  - platform: sun
    name: Next Auto Open
    type: sunrise
    format: "%Y-%m-%d %H:%M:%S"
    entity_category: "diagnostic"
    update_interval: 10min
    filters:
      lambda: |-
        const std::string input_time(x);

        ESPTime parsed_time;
        ESPTime::strptime(input_time, parsed_time);

        parsed_time.recalc_timestamp_local();
        parsed_time = ESPTime::from_epoch_local(parsed_time.timestamp + id(open_delay).state * 60);

        char buffer[64];
        parsed_time.strftime(buffer, sizeof(buffer), "%Y-%m-%d %H:%M:%S");
        return std::string(buffer);

  - platform: sun
    name: Next Auto Close
    type: sunset
    format: "%Y-%m-%d %H:%M:%S"
    entity_category: "diagnostic"
    update_interval: 10min
    filters:
      lambda: |-
        const std::string input_time(x);

        ESPTime parsed_time;
        ESPTime::strptime(input_time, parsed_time);

        parsed_time.recalc_timestamp_local();
        parsed_time = ESPTime::from_epoch_local(parsed_time.timestamp + id(close_delay).state * 60);

        char buffer[64];
        parsed_time.strftime(buffer, sizeof(buffer), "%Y-%m-%d %H:%M:%S");
        return std::string(buffer);

Finally, the interface for controlling this is just a simple card on my dashboard, making use of a pop-up for the more involved settings. You can open, close, toggle auto-open, and view all the information in one place. And its already proven its worth. As I was writing this article up, I got a message from the lawn-care company stating they'll be out tomorrow to spray, and so I turned off the auto-open, which will keep the door shut tomorrow until I remotely trigger the opening.

**** Controls on my main dashboard. A toggle for auto-open, then Open, Stop, and Close controls

Clicking the icon on the main controls pops up this dialog, giving you a bit more information.

I've thought about adding some more "smarts" on the HomeAssistant side, like disabling auto-open if the temperature is too low, or if its raining, but those are projects for another day.

The chickens themselves.

Updates

2025-06-22

I was having a few issues with brown-outs, even after upgrading the power supply (as mentioned earlier). They were fewer and further between, but for an appliance like this, we don't want any issues. It had yet to leave the door open at night, but there were a few mornings when I'd look outside and not see any hens running about. From what I can gather online, the H-Bridges I use, the ones with the built-in 12V-5V converter, occasionally have issues where the converter will output less than 5V intermittently. It seems to be hit or miss if you get one that has this problem, so I'll continue to recommend them, but if you do get one that has these issues, then you can get a simple little buck converter, wire it in series with the H-Bridge unit, and connect your ESP32 to the output of the buck converter instead.

I've been running this setup for a few days now, some of the hottest we've had this year, and it's yet to give me any trouble.

I've got a chest freezer that I like to keep an eye on. It's never failed me, but it has several thousand dollars worth of foods and meats in it, and sits out of the way, so it's not something you check every day. If it did fail, it could be catastrophic. Since I have Home Automation, this was entirely unacceptable. So I set out to monitor it.

State of the market, 2024

At the time of this project, there were a variety of fridge/freezer products on the market, some of which even had remote monitoring, which was a requirement for this project. But, of the ones I'd found, there weren't any that were HomeAssistant or "DIY-er" friendly.

Both ThermoWorks and FireBoard offer fridge/freezer monitoring tech, and I like products from both companies. But their monitoring tech, at the time, was unsuitable. Both companies products required internet connections; there was no LAN capability, and both, being commercial oriented, were not meant for 24/7 monitoring¹

Looking outside them for a while, a common approach many people seemed to take was to shove a simple Zigbee temperature probe, the kind you'd leave outside, in their fridge. This, of course, barely works in a small apartment, and in my big ol' house, wouldn't work at all. The signal attenuation through the walls of a big chest freezer alone means you have to put a repeater practically on the other side of the walls, and the cold environment isn't kind to cheap batteries. I probably could have made it work, but it seemed unreasonably complex.

When they won't build it, you have to

Dissatisfied with what I saw on the market, I went about building my own. I laid down some requirements first:

• Must be local only
• Must tie into HomeAssistant
• Must have replaceable and removable remote probes, ideally K-Type thermocouples.

With these in mind, the obvious candidate was to use an ESP32 device with ESPHome. At the time, the only ESPHome project I'd done was the cat feeder, which was mostly just a bit of GPIO and some timing. This would be more complicated.

The ESPHome platform supports use of external modules to translate the weak signals from a K-Type thermocouple to something the ESP32 can use (i.e. a temperature). The module I wound up using was the MAX31855-based module, specifically this unit off Adafruit. ESPHome supports this natively, and so it was a shoo-in.

Wanting to also monitor the conditions of my Garage, I grabbed a AM2301B ATH based Temp/Humidity sensor. These are supported in ESPHome via the AHT10 module.

Finally, I needed the probe itself. ThermoWorks sells a probe called a food simulant probe, which is exactly what it sounds like. It's a square block of plastic, with a probe embedded in the middle. You stick it in your fridge/freezer, and instead of just getting the point reading of metal probe, you get a "diffuse" reading, what a piece of food in the freezer would actually measure as a temp.

Other odds and ends, of minimal importance to the project, were the power supply, which was just a cheap multi-voltage one I picked up at radioshack, the enclosure, and a K-Type extension cord, which was used as the "terminal" for the connection to the PCB

Assembly and testing

Assembly was extremely straightforwards, basically electronics lego. Just solder some headers on a protoboard, solder some headers on the MAX31855 and ESP32 (mine came without them), solder leads from the power cord to the VCC and GND of the protoboard, and then solder the leads from the AHT20 to the protoboard. The K-Type extension cord got its male end terminal removed, exposing the bare wires, which happily set into the screw terminals on the MAX31855.

Once it's all assembled, take your ESP32 over to your computer, flash it with the loader, pair it with HomeAssistant, and write a short bit of ESPHome yaml to configure it. Then mount it on your protoboard, and close up your case.

I glued the AHT module to the side of the case, and put some magnets on the back, so I could stick it to the side of my freezer.

I'd previously stuck the food simulant probe in my chest freezer, and fished the cord out through the back, so it wouldn't interfere with normal operation of the lid. I'd let it sit for 24 hours, so it was good and cold, and measuring it with a normal, non-connected thermometer reader (A ThermoWorks ThermaQ, in this case), saw my freezer was hovering around -1ºF, where I like to keep it. Disconnecting from the ThermaQ, and connecting to the extension that connects back to the MAX, I should have been good to go.

Calibration

Unfortunately, the temps I was seeing in my HomeAssistant were wrong. But they were consistently wrong at the temperature ranges I wanted to read. I didn't need to calibrate for a large set of different temperatures, my probe was likely to only be reading temps around -5ºF to +5ºF, and so testing and finding the inaccuracies at this range is a lot easier.

I used a K-Type thermocouple simulator I spent far too much money on to generate temps around this level, and noted the difference that HA read from what the sim was generating.

Once I'd measured a decent enough cloud of points, I took the average of the offsets, which turned out to be -1º. Fortuitous.

ESPHome has a few different ways to calibrate a device. All the calibration systems use the filters system, which is a way to change or modify how a sensor in ESPHome works. There are linear calibrations, logarithmic calibrations, and more. They all take some data points, and use them to shift the signal accordingly. For my use case, since I was really just happy shifting the temp by a degree, I could just use a simple offset filter.

For all the sensors according to this board, I didn't want them to update too frequently, as you'd get noise on the graph. I had the update intervals set to 10 and 20 seconds, for various sensors, but even thats a little more frequent than I cared about. One nice way to smooth out a signal, and get accurate readings, is to use an exponential moving average, which ESPHome provides as a nice filter! Installing it on each module was simple, and I set it to basically average all the reads from the sensor over a minute. I had to tune the alpha factor a bit for each of them, but that was mostly trial-and-error until I got a result that I liked.

State of the market 2025

Since building this, there are a few new products on the market that might be interesting to people wanting to do the same thing, without going the DIY approach. Sonoff has come out with a Zigbee based temperature probe, the SNZB-02LDSNZB-02LD² that has a remote probe. You'd stick the transmitter/screen on the wall or side of your fridge, and the probe inside. No clue how well it works, but it's an option if you want something easy, and already are invested in the Zigbee universe.

You'll want to make sure you get one that has the external probe. The amazon listing seems to be an "updated" use of an old listing, for a probe-less model.

I've been enthusiastic about ESPhome for a while now. It seems to be the perfect medium between turn-key home-automation appliances, and 100% DIY stuff. I've used it to measure the temperature of my chest freezer, to fix up a cat feeder no longer supported by the manufacturer, and to add an optical rain sensor to my weather station.

So naturally, when reading through the documentation, and coming across the Sprinkler controller, I thought "This would be an excellent project."

Enter RainMachine

Back in 2020, when we purchased this house, I did a bit of research, and came across two real contenders for the irrigation controller I wanted. RainMachine, and OpenSprinkler. Both were aimed more at the diy-ish type, but the RainMachine was glossier than the OpenSprinkler, and seemed the better investment. Their software was slicker, and the controller seemed just smart enough to do what I wanted, but not get in the way.

Others, such as Rachio, or the big names like Orbit's B-Hyve, Rainbird's Wi-Fi enabled one, and offerings from K-Rain and Hunter, just didn't appeal to me. All were either dumb controllers with some smarts crudely bolted on, cloud-locked, or some similar combination of negatives.

I wasn't looking for a tremendous amount of intelligence in my sprinkler controller, rather, I wanted a way to just set the schedule from a website or my phone, have it adjust the watering depending on how much summer rain we get, and allow me to start and stop stations remotely, probably from my phone, so I can work on the sprinklers as problems come up. HomeAssistant integration was an added bonus, but ultimately not something I was terribly concerned with at the time.

Exit RainMachine

The RainMachine has served me well for the last 5 years, but over the last few, the company behind it has seemingly been in rather dire straits, and, as far as I can tell, has gone out of business now.

A few years ago, they transitioned their online "cloud" services to a premium plan. The sprinkler controller still works completely fine without their cloud, but the cloud gives you nice things like no-fuss remote access, more weather providers, and (purportedly) priority support. And the price for it wasn't too bad, so I paid for it for a couple of years. I could have had my own remote access working fine; I've got tailscale running on my home network, but if a few bucks a year kept the company going, I was willing to pay for it.

Sadly, it didn't keep them going. Over time, the connections to the controller became more and more unreliable. Sometimes the app would just sit there, loading infinitely, without connecting. It didn't matter if I was connected to the LAN the controller was on, or was remote, it just wouldn't load. Sometimes even attempting to load the (lan IP) website in a browser wouldn't work, requiring a trip to the controller in the garage to reboot it. And even when it did work, you'd randomly get sluggish behavior, such as not turning on a zone when given a command to do so, or worse, not turning off a zone when instructed to.

Finally, in late summer of 2024 (last year), my grass began to die off in places. At first, I dismissed it as the usual hot-summer browning, confident the grass would turn back to green when the temp fell a bit. But one night, up late, I noticed I didn't hear the sprinkler running. Checked the app. Non-responsive. Checked the website. Non-responsive. Went out to the physical controller. Completely locked up. Had to power cycle it, and then I found out that it hadn't watered in a week. It had failed receiving some weather update, and just crashed and not recovered.

I was unable to renew my cloud subscription earlier that summer, and now it appears that the controller wasn't reliable at all.

I could have just taken the device completely offline, firewalled it to not connect to the internet, and pushed weather updates into it from HomeAssistant, but I was already a bit tired of some of its other drawbacks, and so wanted to explore other options.

Getting started with a DIY controller

I was very interested in making a sprinkler controller with ESPHome. As I mentioned in the opening paragraph, I've used it for a few tasks around the house, and have been rather impressed. And so it seemed like the perfect fit.

I went online and ordered a relay board, specifically a KinCony KC868-e16s. This board has an ESP32, 16 relays, and some IO multiplexers to allow easy control of all the relays and inputs. It seemed like a perfect fit for an irrigation controller; just wire up 24VAC to the relays, connect each station to the other side of the NO contact, and then connect the stations common wire to the other terminal of the transformer. Presto, nice simple irrigation controller.

Software wasn't too much harder to connect up. The ESPHome Devices page has a nice sample of what a basic configuration for this board would look like, and so I just started with it, editing the parts I needed. Initial testing was promising; I was able to switch every relay on and off from HomeAssistant, trigger the piezoelectric buzzer, and listen to inputs on the various input pins.

Basic relay clicking working, I set up the sprinkler component, and got the various stations set up. I wanted to have two different "controllers", one that would run my high-pressure rotor sprinklers, and one that would run the low pressure drip lines. One of the more disappointing parts of my RainMachine was that it had a singular master controller, so if you wanted to control a pump AND a master valve, it was all or nothing. For the low pressure drip lines, the supply water pressure would be strong enough to give them what they need, and the boost pump was unnecessary. With the rain machine, I had to install pressure regulators at the head of each dripline, just to avoid blowing out the drip plugs.

Depending on how you set up your controllers, you can have as many pumps, master valves, whatever, on any zone, and there's not actually a need to split them up into separate controllers. As the system progresses through a "program", it will turn on and off the various pumps and valves as needed. However, I wanted to have different run cycles for these, as well as running multiple short drip lines at once, to cut down cycle time, and so separate controllers made the most sense for me.

After a bit of YAML, I had what I thought would be a passible configuration. Flashing the device, I opened up HomeAssistant to test, and started a program that would run through each station, 5 seconds per station. Station 1 and 2 fired off just fine, but when it came time for Station 3, the "pump" relay switched off, then the station switched off, while leaving the master valve on, and then the whole thing rebooted.

I fiddled with my configuration a few times, figuring one of the proxy switches I'd made to turn on/off both the pump and master valve from a single switch was the culprit. But even removing it and going down to a single, physical master, I was still having the reboot issues.

Connecting up a serial terminal, so I could actually watch the logs, not whatever the ESPHome sent over the network, I tried to run another cycle, and saw something that made my heart sink. Every time it got to the third station, it had a panic, and rebooted.

I'm not tremendously well versed in embedded software, and so was well out of my depth here. So I collected some logs, opened an issue with what I'd found, and essentially consigned the controller to the "future project" parts bin.

I could have removed the sprinkler component, and just used a bunch of switches to control the zones. There are homeassistant addons that can automate sprinkler systems that are little more than collections of switches. But the itch at the back of my head was saying "this is becoming a fractal of complexity".

I want my irritation system to be something that I can tinker with when I want to, but will more or less work unless I actively break it. And tying the whole thing to HomeAssistant seems to be introducing an unnecessary point of failure. I like using HomeAssistant to augment systems that already exist, to make them better. Any time I have absolute reliance on it, I feel a little uneasy. It hasn't let me down in years, but I started HomeAutomation in the dark ages, when things were extremely fragile, and so I remain wary of those.

OpenSprinkler

Since my RainMachine was essentially non-viable at this point¹, and I needed a new controller, I looked to OpenSprinkler. They offer a few different models, but for my needs the basic, 24VAC one, without latching DC solenoids, was what I needed. Since I've got 14 zones, I also needed to buy one of the expansion boards. The base OpenSprinkler controller only has 8 controllable zones, and they solve this limitation by selling expansion boards; little boxes that just have more zone terminals on them, that connect up to the main controller via a short ribbon cable and use I2C.

OpenSprinkler lets you do nearly everything I've wanted with my irrigation systems. It lets you have 2 "master" controllers, I'm using one for a master valve and one for a pump. Each zone can use one, both, or neither master control. It lets you group your zones into 5 "groups" for exclusivity control; all the zones in one group run sequentially, but you can run zones from different groups in parallel. There are 4 main groups and one special, "parallel" group, in which all zones can run parallel to any other zones, including others in the parallel group.

I was able to quickly get my zones set up, a few basic watering programs set, and so far have yet to encounter any actual limitations that matter to me.

There is a 3rd party homeassistant integration, which I'm running; it brings a ton of control into HomeAssistant, which is useful for a few of the programs I've used in the past. During the summer, I have a little automation set that runs a few zones for a short period of time, 5 minutes or so, when the outdoor temperature gets above 90º. This gives my chickens a bit of mud to go sit in and cool off, and seems to keep them pretty happy. The HomeAssistant integration allows me to fire off individual zones OR a program, using the Actions system, so I've got some flexibility on how I approach it.

The OpenSprinkler system isn't without its flaws, and some of them are rather notable to me. But none of them are dealbreakers, or even really things that prevent me from doing what I want to do, they just make a few things more complex than I feel they should be.

The UI isn't particularly nice to look at. Its serviceable, and by no means non-functional, but compared to the gloss of something like the RainMachine, it's not as good. That said, it actually works consistently, and fast, which isn't something I can say for the RainMachine. Taking the theme of UI considerations, there's a bit of an inversion when it comes to splitting up a run cycle². With both the RainMachine and the ESPHome sprinkler component, you would set your total run time for a zone in a program, and then use a divider to choose how many cycles there were. A zone set to run for 30 minutes, with a divider of 2, would run for 15 minutes in each cycle, with the cycles typically being back to back. OpenSprinkler does the inverse of this, they use a multiplier approach. You set the run time of each zone, and then set a cycle count and delay time. Frustratingly enough, the delay time lacks a "resume after the first cycle has completed" feature. This isn't an issue if you have your zones set to the same exclusivity group, their run times will just queue up, but if you make heavy use of parallel groups, and allow for weather influenced changes to cycle times, you could get overlap

The weather system for OpenSprinkler is rather powerful, but getting your own weather sources, such as my own personal weather station, into the system isn't terribly convenient. You need to run a weather data provider program on your own server, and use that to push data into the controller. OpenSprinkler provides one that uses several online data sources, and swapping over to your own isn't terribly hard, but compared to the RainMachine, which allows for a simple HTTP push of weather data, which can be triggered by things such as WeeWX or HomeAssistant, it does stick out.

Fortunately, all the weather system is used for is doing weather level adjustments. You can actually set these to completely manual adjustments, and then use something like the smart-irrigation hacs package to calculate and push adjustments from HomeAssistant.

But other than those little bits of discomfort, I'm overwhelmingly impressed by OpenSprinkler. Let's hope I stay impressed throughout the year.

And just because I had a bad experience with a particular ESPHome system, doesn't mean I'm swearing off it either. I've got another project coming along soon, and when finished I'll get a blog post about it up.

As I've aged, my interest in competitive multiplayer games has waned. I find that I have no interest in fighting other players online, that I don't really care about the score I get, and that having those things present in a game tends to pair me up with players I'd rather not play with. At the same time, my interest in PvE and Co-Op games has grown substantially. While recent games have been rather good on the PvE front, they're lacking in co-op campaigns. And I wish that wasn't the case. Many modern games feature stories that lend themselves particularly well to having a "squad" of Protagonists; rarely is your character alone in any of the missions. Yet, nearly all of these games won't allow you to replace one of your AI squadmates with a real flesh and blood human.

Co-op is a rather recent invention

We can call this the "history" section of the article. Back in the sixth generation of game consoles (Xbox, PS2, GameCube) was really when we saw the beginnings of true co-op, and they were all offline affairs. I wasn't aware of any games having meaningful online co-op at the time, and didn't play them. But this wasn't a terribly big deal, online play was almost exclusively for things like deathmatch, or MMOs like EverQuest. And even then, online play was generally the domain of PC games, not consoles.¹

What was rather common on games of this era was split-screen, both for competitive multiplayer, and for co-op. Halo: CE shipped with split-screen co-op, and I spent many a night replaying the same levels, over and over again, with my friends, clustered around a tiny 13-inch Emerson TV. Yeah, the split screen co-op of these early games was generally limited to two players, but it was something to do together, a nice pace change from the 4 player deathmatch games.

This, and the subsequent generation of game consoles, really is the heyday of co-op. We saw entire games spend a substantial amount of development time figuring out how to make the game more co-op friendly. Games like Gears of War designed entire levels around having multiple people playing them together, with AI following a scripted path for single play.

In this generation, we also saw the rise of online co-op. Gears of War, as far as I remember, had online co-op from the get-go. Halo 3, which came out a year after Gears of War, had 4 player online co-op, but only 2 player local.² You could play your favorite games with your friends, working together to beat a difficult level, playing along with someone on their first playthrough, or just goofing off and trying to break the game in ways that the developers never accounted for.

Things were good for a while, but around the time the 8th generation arrived, online co-op began to fade. Games like Dead Space 3 were extensively built around co-op gameplay, and critics felt the single player experience suffered because of this. By the time the ninth generation (the current one) arrived, online co-op was a rarity, and often delivered well after the game launched, as was the case with Halo: Infinite. Couch co-op is almost completely dead, with rare exceptions being more "casual" oriented games, such as Untitled Goose Game.

Why can't my buddy be my squad mate?

Since modern game AI has improved tremendously, we're seeing many modern games move towards having a "squad," one member being the player's character. These squad mates exist to help drive the story, provide guard-rails to assist players who might not know what they're doing, and make the game feel more "alive."

In almost every case, you can't replace the AI squadmate with your buddy. Games like Call of Duty: Black Ops 6 feature expansive campaigns, where your character is generally paired up with a partner, at the very least. Driving around desert-storm era Iraq in a humvee full of soldiers, I wondered why my buddy couldn't step in behind the eyes of one of them, and play a bit of the campaign with me.

Other games, such as S.T.A.L.K.E.R. 2, or Starfield, are much more "single player" oriented, but still feel as if co-op would dramatically improve them. Starfield has the player running about with companions, unless they choose to forgo them, so why can't I run around with my friends? Stalker 2, you are a "loner" yes, but walk anywhere in the game and you're bound to encounter small squads of NPCs, two to four of them, moving through the zone, same as you do. This would be great if your buddy could drop into your game and squad up with you.

Arguments against co-op

A common argument behind co-op is that it's technically demanding. While it does come with its own set of headaches, yes, its more or less a solved problem in most game engines nowadays. If you use Unreal, there are primitives ready to go for online co-op, that handle synchronization of data and events between participants. If you use other engines, these primitives likely exist as well. If you're worried about players venturing too far apart and causing too much of the game world to be loaded into memory, there are solutions to that problem as well, some of which are 24 years old. You can put a leash between players, so they don't have the ability to wander too far apart, or choose "softer" options, like the teleporting Halo has done since Halo: CE (with the exception of Halo: Infinite).

Another argument I've heard is that the game would become unbalanced should you allow players to team up. This is an argument entirely without merit. If the players want to team up to beat something too difficult for one or both of them, that's their prerogative. Make the game too hard, and players will drop off and lose interest, regardless of how much trash-talk and "get gud" crap one might see on message boards. Modern games have received criticism of being "too easy," but they almost all feature a difficulty slider, and accessibility features that allow a wider range of players to enjoy the game. And even still, ignoring these factors, it's a simple matter of just scaling internal difficulty systems up should you toss in more players. Make encounters longer, spawn more enemies, spawn higher health enemies, etc.

Finally, there are games where co-op doesn't make sense. The recent Indiana Jones video game (Indiana Jones and the Great Circle) wouldn't make sense to have two Indies cavorting around, and the previous attempts of the franchise to introduce sidekicks (Crystal Skull) have fallen flat.

Closing

We have these amazing single player experiences that almost scream to be enjoyed co-operatively, with squad based mechanics or amazingly stunning gameplay. Yet they very rarely are. I would love to run through the zone in STALKER 2, trading artifacts with my buddy, or setting up complex ambushes of enemies, but I can't. I would love to have my friend explore remote planets in Starfield with me, but I can't. I would love to have a fellow bandit ride along with me in Red Dead Redemption 2, but I can't³

I've some ideas what a "proper" online co-op experience should have, but at this point I'll take whatever I can get, since its so rare. But I'd love to see the following features:

• Drop-in-drop-out co-op. Since it's a more "casual" experience, and most of my online game friends are adults as well, we can't always carve out large blocks of time. As such, I should be able to join in my friends game at any point, help them along, and then leave whenever I have to go.
• An inventory that follows me. If I find rare artifacts while playing in my friend's game, those shouldn't just be forfeit because I wasn't host. Similarly, if I have something rather powerful that could help them out, or multiple of said something, I should be able to bring it with me, and use it or give it to them.
• Instanced loot. When you're fighting some boss, or looking for loot, or whatever else, and you find 1 really high tier item, it becomes somewhat of an issue of who gets it. Does the host get priority? What if they're the better player? Does the guest? What if they already have it in their play? The best solution to this is instancing the loot, so each player has the option to grab it.
• Progression tracking. If I finish missions in a friend's game that I haven't finished in mine, I should still be able to take credit for them. With more linear campaigns this can become an issue, so it's probably the weakest of the options here, but it's nice for the type of game that can use it
• Separate difficulty sliders for each player. Gears 5 does this particularly well, and older entities have had it as well. Each player in the lobby can choose what difficulty they are playing on. So if you're a seasoned player, you can dial up the difficulty, and vice-versa.

For some games, some or even all of these suggestions won't work. Some games that are far more story driven allow you to choose a save slot as your gameplay, and save any progress you make to that save slot. If the save slot is at a certain point, or you start the game from a particular level, then that can be saved to the slot.

But as I said earlier, this is a wishlist. I'd settle for just having some co-op at all.

I've been using the Kagi search engine for a bit over a year now, and while it's not fundamentally changed my life, it has made me reflect and think about some things, mainly where the internet has gone, where it was, and what we've lost along the way. It's safe to say that most of us spend most of our waking hours doing something with the internet. It's in our pockets, on our wrists, in our ears, in front of our eyes, and sometimes, quaintly enough, on our computers. Yet, as many people have noticed, the internet has gotten really irritating to use over the last few years.¹

Part of what makes it immensely irritating is that the tools we used to rely on for navigation through this wilderness, the search engines and content crawlers, have become actively hostile towards us as users. Google no longer tries to show you what you're looking for, instead it tries to show you things that it wants you to see. And what Google wants you to see is ads. Additionally, Google is under the somewhat perverse incentive not to give you what you want, as the more time you spend searching, the more ads you see. Couple that with the amount of shit they shove at the top of a search result page, and it's a miserable experience.

Gone are useful tools like the older knowledge graph; it's been replaced by a cyborg facsimile, an AI box that hallucinates its own insane, twisted reality. From things like telling people to put glue on pizza to eating rocks to just flat out inventing things that don't exist.

And it's been trending downwards for a while now, seemingly gaining steam and becoming shittier as time goes on.

Enter Kagi

Mid-2023 I started hearing some interesting things about a new search engine. Comments on HackerNews, reddit, and in some private chatrooms, were talking about a new search engine, called Kagi. At first, I ignored these discussions, figuring it was another flash-in-the-pan startup (remember Cuil?). But over the summer of 2023, the talk about this tool didn't wane, as usually happens with these things, but rather intensified.

Finally, in September, I'd repeatedly heard enough good things from enough different sources, I decided to check it out. At first, I wasn't terribly impressed. A paid search engine? Come on. Who the hell is going to pay for search? Yeah, Google isn't great, but Bing is marginally better, and Bing pays you to use it². But Kagi, and commenters across the internet, implored me to try it, to use the 300 free searches, and see what it was all about. If I didn't like it after the first 300, I could go back and not pay a cent.

So I tried it. And I loved it. At first, I just treated it like another search engine. I put in my query, got back results. Boring, right? Well, yeah, boring by 2007 standards. But by 2023 standards? The results were actually what I wanted. And things that Google and others long since cast aside, like boolean operators? Available, in all their glory. I found myself doing something I hadn't done in nearly a decade: aimlessly browsing the web, finding new and exciting pages, written by small bloggers, authors, whatever; people who were passionate about their work, and not underwritten by some faceless content mill.

I burned through my 300 searches in about a week. The moment I hit my limit, I jumped and bought one of their ultimate plans, for a year. I have no regrets on that purchase. Looking at my usage page, I average about 950 searches a month, with some outliers being as low as 400 a month or as high as 2000 a month. And this doesn't include searches that use DDG style !bangs, which Kagi not only supports, but lets you add your own custom ones³. Yeah, you can add custom search engines to most browsers worth their salt these days, but the bangs somehow feel more useful. Maybe It's that you can shove em at the end of a search to change what the search is doing.

Over the year, my usage of Kagi changed as well. At some point, I discovered the ? bang, which will force Kagi to generate an AI chatbot answer at the top of your search results. At first, when I read about this, I was dismissive. The Google and Bing versions of this weren't terribly useful, often providing worse than wrong answers, but I went in open-minded, and was impressed. While I don't use it all the time, It's nice to be able to search something like When did Duke Ellington die ? or Is Atomic Heart on Xbox Gamepass? and get the answer right there, complete with citations to check its work.

I still only use the AI chat feature sparingly. Kagi gives you access to most of the big models, through their own interface, which is useful when trying to research something, prototype a bit of code, or whatever else, but I still find them only about as reliable as a precocious teenager, happy to make up things to make themselves sound smarter than they actually are. I do enjoy that I have them, in a nice interface, that lets me run the same query against different models and compare the results. I do enjoy that, being run through Kagi, instead of a user account associated with me, they're effectively washed in a big stew of all the other AI queries other Kagi users are making, bringing about a thin layer of anonymity. But ultimately I don't see them as being hugely useful. A novelty, for sure, that comes in handy, but if it was turned off tomorrow I probably wouldn't miss it.

And you might be asking, in that whole year (and then some) of using Kagi, how often was it unable to find what you're looking for. And the answer is a bit more complicated than it should be. Most of the time, if I couldn't find it on Kagi, it was a flaw in my search terms, that more refinement was able to sort out. Every time Kagi couldn't find something, using another search engine proved equally fruitless. Since Google and Bing are a single bang away, if something wouldn't turn up, I'd try them, but ultimately stopped doing so, as they never returned what I needed.

The two areas where Kagi is weak, and where I do shell out to other engines for more information, are local (maps) and images. Google Maps is a juggernaut in the local and maps space, and is extremely hard to dethrone. I like that Kagi lets me use Apple Maps and OpenStreetMap⁴, and give it a good college try every time I need to find something and have a bit of time to suss through the results and find what I want. But if I need something now, like closing times for a restaurant, I just immediately append the !gm bang to search in Google Maps.

Image search in Kagi is good, but not great. Image search seems to be rather tricky to conduct properly, as no engine gets it right entirely. Depending on context, I typically find myself using Bing image search or Yandex Image search. Google Images used to be good, but ever since Google got a bug up their ass about content in their search results, it's been rather useless.⁵

And that leads to what was ultimately the most revealing revelation about Kagi, and how it changed how I search the web. Google is all too happy to color their results to align with their particular corporate biases. If you are in agreement with the bland "safe" Bay Area values that Google's corporate structure holds, you probably won't notice this, but the moment you step out of line, on any subject, you'll suddenly find that Google won't show you what you're looking for. Inconvenient facts are buried in a manner that Orwell would dismiss as too heavy handed to be believable. Search results and suggestions will serve to nudge you back onto the "safe" path, and when that nudging fails, just refuse to show you what you're asking, instead just vomiting out SEO garbage or counterpoints that align with the Google values.

Kagi doesn't do that. If you search for disgusting content, and turn off safe search, you get what you ask for. It treats you like the adult using the product that you are, not like some petulant child asking inconvenient questions. And once you get used to the freedom of having a search engine respect your wishes, returning to the same old is an exercise in frustration.

Anyway, that's my pro-Kagi rant for the end of the year. If you haven't tried it, I'd encourage you to try it. Kagi doesn't do affiliates or referrals, so I gain nothing either way, but I want to see them succeed, so they can stay in business and keep making the best search engine I've used in the last decade.

Previously, when I was setting up SDR to read my utility meters, while doing research into various tools for reading data with an SDR, I stumbled across a description of some home burglar alarm systems, and how they send data in an easily readable format. This piqued my interest, but I ultimately ignored it at the time, as I was already down a rabbit hole with the utility meters, and wanted to finish that project first. Now, its several months later, and I dove back down that rabbit hole, and within a few hours was able to get a system working quite well.

My house was built in the mid-90s, when Home Automation meant a mixture of an intercom system, burglar alarm, and maybe some X10 stuff (I didn't find any of that, sadly). State-of-the-art burglar alarm systems at the time used cheap little radio transmitters, to avoid the need of running wires all over a house.¹ These devices typically take the form of a little box next to a door or window, with a reed switch that is triggered by a magnet attached to the door or window. And my house is chock-full of them.

However, looking at these sensors from the outside, they are somewhat hard to decipher. Although their immediate function is readily clear to most people who would care about this sort of thing, how to get the information out of them is somewhat more obscure. Fortunately, smart people on the internet have documented the messages these little boxes put out, and written tools that can decipher them.

One such tool is the excellent rtl_433. This tool is a little Swiss army knife of SDR. You can use it for all sorts of things, like reading cheap wireless temperature sensors, using cheap keyfob remotes for input devices, and more. You'll need an SDR, and the RTLSDR I used before sadly won't cut it this time. These sensors transmit at 319.5MHz, and the RTLSDR just couldn't receive them. Fortunately, there's a low-priced and excellent SDR that can read them: The Nooelec NESDR.

Setup

The nesdr is more or less plug and play. Once you have rtl_433 installed, It's pretty much just a matter of calling the right program with the right arguments, and you should immediately start seeing results.

On the physical side, your nesdr may come with a few different lengths of antennas. This little device can be used to listen in on a very wide variety of frequencies, and so the antennas bundled with it have different uses. To find the ideal, or near ideal, antenna length for a frequency, we use a bit of simple math to get the quarter-length antenna, which tells us we need an antenna about 9.2 inches long. My nesdr came with one about this length, so I used that one

Software wise, it couldn't be easier. Assuming you have rtl_433 installed, simply fire it up, listening on 319.5MHz, with the protocol set to 100²:

rtl_433 -f 319.5M -R 100

After firing up this, you'll get a console that seems rather quiet, after the initial messages. Go over and trigger one of these sensors (open a door). You should see a message appear in the console, describing a sensor, several switches, and maybe even a battery state. When you close the door, you should see a similar message, but one of the switches should have changed state. That's your door sensor. If you don't see anything, check the battery on the unit. Many of my units still have good batteries in them, despite the batteries being well over 10 years old.

Now we need to get that information into HomeAssistant.

MQTT

rtl_433 has built in MQTT support, which is the best way to get the information from the sensors into HomeAssistant. You'll need to have an MQTT broker system running, which is beyond the scope of this article, but not terribly complex. Assuming you do have one running, simply modify the command we ran previously to resemble this one:

rtl_433 -f 319.5M -R 100 -F mqtt://your-broker-hostname:1883,user=broker_user,pass=broker_pass

Fire up a tool like MQTT Explorer, and trigger one of the sensors. You should see a message fairly quickly

Once you see those messages on your MQTT network, we can get them into HomeAssistant. In your HomeAssistant configuration, you'll need to add something similar to this yaml:

mqtt:
  cover:
    - name: "Front Door"
      unique_id: "front_door"
      device_class: "door"
      state_topic: "rtl_433/server/devices/Interlogix-Security/contact/af42c7/switch5"
      state_closed: "CLOSED"
      state_open: "OPEN"
      device:
        name: RTL433
        identifiers: "RTL433 sensor"

You'll want to add one of those for each unit you want to detect. I recommend watching the network with MQTT Explorer and go around triggering each sensor, noting down its ID, and then setting up each cover entry.

Once that's all done, reload your HomeAssistant config, and you're done! You now have your door, window, heat, glass break, and any other sensors you cared to map in HomeAssistant.

When building this site, one thing I wanted to do was do it as cheaply and easily as possible. I didn't want to have to go out of my way to write for it, deploy it, update it, or anything else. And I certainly didn't want to pay for it. As such, I built it in a very particular way.

The site is built using the tableau static site generator, by Mitch Hanberg, and the entire site is open source. Assets are handled using [ESbuild], and consist mostly of some CSS niceties, and the occasional javascript component written in [lit]. Posts are written in markdown ¹, in VSCode, and deployments are simply done as part of a CI/CD process; I push a blogpost up, and the blog updates.

The past

Initially, the site was built using vuejs, but I migrated over to tableau. The details as to why I did the migration are in the linked blogpost, but suffice to say all these months later I don't regret the migration to tableau.

GitHub pages were a natural fit for what I was trying to do. They're free, easy to use, support basically anything that can generate plain HTML, support and automate HTTPS, and more or less just work. You miss out on some fancier features, but at the time I didn't really need those (more on that later).

Deploying to ghp in an automated fashion was rather simple. Just use GitHub actions, and the built-in deploy to GitHub pages action, and you're good to go. You can see how I used to do it here. This workflow builds the elixir site, builds the assets, minifies the HTML files, and deploys them to GitHub pages. Simple, easy to reason about, and free.

Analytics

For a long time, I didn't really care about how much traffic my site got. Sure, I wanted to have people read it, but I didn't really care how much read it, or where they came from. I've got a search keyword monitor for links back to my site, and so would see when various things were published to various discussion forums and other blogrolls, and where appropriate I'd join the conversation. But recently, I had a post get a decent amount of attention, being featured on sites like [Hackaday], and suddenly I found I wanted to know where my visitor traffic was coming from.

I'm generally wary of most analytics systems. They're a nasty part of the surveillance reality we all live in, and most of them are horribly intrusive when it comes to privacy. They're also essential parts of web advertising, and so they want to track where a user goes across the web. I use a handful of systems to block as many trackers as is reasonable, and so was hesitant to add any to my personal site, as it felt a bit hypocritical.

Over the past few years, I'd been hearing good things about plausible analytics. I first heard about them through an elixir based discussion, although I can't remember which. They're an open-source analytics platform, that you can run entirely yourself, as well as a hosted option that, while not free, is well within affordable. They have a strong focus on privacy, so much so that they use no persistent tracking mechanisms at all.

If I were hosting my site on a more traditional webserver, like Nginx or Apache, I could analyze the logs to generate site data. Back in the days of Drupal, Joomla, and e107, I did this a lot for my pet sites, and its honestly astonishing how much information you can get just by scanning the logs and comparing them to something like the Maxmind database. Since I'm not, I don't get that data for free. But using something like plausible seems a good compromise.

And so this site now uses plausible for its analytics. You can view the analytics the site collects (probably rather unimpressive) at any time here, which is also available in the footer of every page. There is still no persistent tracking mechanisms on this site, and you can double-check that (no cookies, no localstorage, nada).

Adblock

One of the problems with any third party analytics script is that some popular blocklists take a stance that no tracking at all is permissible. This is fine, and it's the right of users to decide what connections their computer does and doesn't make, but it throws a wrench into the works when trying to get something that resembles accurate analytics, even if they're non-invasive, particularly for more technically oriented sites like this one. Plausible themselves estimate that on techincal sites, up to 60% of viewers may block their tracker. Not great for me.

The simplest way around this is to proxy the analytics script through a server you control. Great idea, but I don't control my server. GitHub does. And GitHub pages offer nothing when it comes to things like this, which is fair, as GitHub pages is more or less just serving up HTML and related assets.

The lazy thing to do was to just stick with GitHub pages, and accept that only 40% or so of my viewers would be accounted for in my analytics. That's fine, but I was feeling motivated, so I started investigating my options, and very quickly settled on Netlify.

Netlify

[Netlify] is one of a handful of companies that have cropped up to cater to the "new" web, where sites are built using technologies like JAMstack, and served as more-or-less static assets, with very lightweight server computational requirements, if any. Essentially, they work like most other static site serving services (i.e. GitHub pages), but offer a few extra features. One such feature was configurable proxies and rewrites 👐. This is exactly what I needed for more accurate analytics.

Setting up an account with them was, expectedly, pretty simple. Click a few buttons, fill out a form, and you've got an account, ready to deploy some sites. And that's where things get complicated.

I initially tried to deploy the site using their automatic GitHub linking, and the first build failed almost immediately. Looking through the logs of the build/deploy, It's pretty clear as to why. Their builder saw that I've got a package.json in the root of my site².

Digging into their builder a bit more, it quickly proved to be unsuitable for what I needed. Doing builds that involve differing languages, such as nodejs and elixir, are possible, but tricky, and from what I could find, poorly documented. Doing builds that use modern versions of Elixir and Erlang were even more difficult; they are running 1.9 for Elixir, which came out in 2019, and so would require me to figure out a way to install newer versions on their build system.

Ultimately, I wound up keeping the build system I was using for GitHub pages, but instead of pushing to GitHub pages, pushing to Netlify instead.

The build script itself is rather straightforwards. The assets and the elixir site are built separately, to get a performance boost through parallelism, and both make use of the upload-artifact action to store their generated outputs. Then another job, dependent on both previous builds, downloads these assets using download-artifact, merges them together, runs minification, installs the Netlify cli, and deploys.

Simple.

Moving to Netlify also netted me a few other positives. Overall global performance has improved, slightly, although one of my friends in India reported that their latency went up a bit, as Netlify no longer has any servers there. And I can now finally modernize a bit of how images are served. Previously, all the images in the site were served verbatim, as they appeared in the source, which usually meant as a Jpeg. Now, thanks to Netlify, images are compressed a bit better, and served depending on what the user's platform supports, be it avif, or webp. Maybe in the future we can serve JpegXL.

Here in Utah, an increasingly common sight is houses with permanent "Christmas" lights. These are usually installed using aluminum channels, along the roofline, and make use of color-changing LEDs. Several of my neighbors have had them installed professionally, and I've always wanted something similar. However, the prices, and limitations, that come with professional installations, are often prohibitive.

Professional installations

Several companies have nation-wide and local footprints for installing these lights. Companies such as Jellyfish lighting, everlights, and trimlight seem to be the biggest players in the space, but I'm sure most roof and gutter companies will offer similar products as a service. They have the installations of these things down to a science; a contractor will come out, measure the areas you want lit, and leave. A while later, usually about a week, they'll come back with some workers, precut channels and lights, and a control system, and get the whole thing installed in an afternoon. Their products will usually come with a warranty, and will generally work pretty reliably. If you just want some lights now, they're pretty much unbeatable in that regard.

But all that convenience comes with some very major limitations. Limitations that, for a tinkerer like myself, are game breakers. First, they typically only offer one type of lighting, and that's pixel lighting. Generally these pixels can only do RGB, no white channel, meaning any whites you get are the odd looking not-white-white we see from other RGB leds, such as those in gaming keyboards and mice. Second, they very rarely have open control software. All of my neighbors installs are limited to only the official app, and require internet access to work. Some of them have mumblings about APIs on their website, but a few hours of research turned up very little more than "contact us for API questions."¹ You can forget about controlling those lights with HomeAssistant.

Finally, they are all absurdly costly. Several thousand dollars at the bottom end, up to tens of thousands at the upper. I can do better, for less.

The state of DIY LED control circa 2024

Fortunately, unlike a lot of other DIY projects, this is a firmly entrenched, established, and catered-to market. There is a massive nation-wide (and international) community of people who do large holiday light shows, complete with music synchronization, permanent led installs, and more. There are many vendors, and products, targeting this segment, and most of them are open, ranging between just using standardized control protocols, to being fully open-source, with the ability to literally build your own controllers using simple, off the shelf parts, like ESP32s.

The biggest utilities in this space are a few software packages, that drive everything else; xLights, WLED, and ESPixexStick. The former is useful for sequencing light shows, and is beyond the scope of this article, but well worth a look if you're into that thing. The latter two are the most interesting. WLED is ultimately what I wound up using, as ESPixelStick is meant more for interfacing with an xLights defined show.

For control hardware, there are a variety of options to choose from, but by far the most widespread and established are the Dig series of controllers from Intermittent Tech. These packages provide a nice set of common requirements for LED control, built around an ESP32. You flash them with either WLED or ESPixelStick, depending on your needs.

As for the lights themselves, you have a dizzying amount of options. There are the common WS281x series LED packages, which are what commonly drives the pixels, packages that support RGBW, such as the SK6812, COB packages (aka LED Neon, very cool), strips, and even newer ones that not only have white LEDs on-board, but multiple white LEDs in addition to multiple color LEDs, so you can adjust the temperature of the white light as well. They all come in various packages too, from simple strips, to strings of "pucks", to stringers (hanging bulbs).

For attaching them to structures, there are nearly as many options as there are light configuration. If you're using pixels, one of the best options is probably PermaTrack, although plenty of other track systems for pixels exist. If you're doing strips, you can buy channels with diffusers that sit over them, spreading the lights out. And some don't even need a channel, such as the larger pucks, which can be screwed directly into a soffit.

My Setup

What I wanted to accomplish.

At the outset, I had a few goals in mind. I wanted lights along the eaves of the entire front of my house. I didn't need lights along the windows, garage doors, or pillars, just trim. Same as if I were brave enough to hang up old-fashioned C9s every year. They had to have a white light channel, in warm white (3000k or lower), as I frequently just want to light up the trim with simple white lights, and they had to be supported by WLED. You might think this constrained me, but in reality, it didn't. This is a fairly common ask, and nearly every single LED configuration out there has some capacity to be used in this.

Initially I wanted to use strips. I wasn't a tremendous fan of the bullet "pixel" look that the commercial installers went with, and thought having a higher density of lights might look better. I'd seen some youtubers I follow talk about how they liked using strips as well, and so it seemed a forgone conclusion for me. I bought a roll of COB LEDs, just to see how they looked, and while they look very cool, the color density was still somewhat low, with one color every 7cm or so.

I've kept these for a future project, as they just look amazing.

From there, I needed a controller, and so I purchased a Dig-Octa. This neat little board is stackable, meaning you can put multiple power routing boards alongside multiple LED controller boards, and have everything working in a nice package. I only needed the capacity of one "brain board" and one power board.

Finally, I needed some power supplies. I bought a pair of Mean Well UHP-350-5 350W 5vdc power supplies. These were based on a back-of-the-envelope calculation on how much power I'd need at peak, based on some power tables published by the guy behind the Dig-Octa.

All in all, here's what I initially planned to use:

• 55m of SK6812 RGBWW (warm white) strips
• 2 power supplies
• 1 Dig-Octa brain board and power board
• Aluminum channels and diffusers for the strips to be screwed to my soffits
• Plastic enclosure purchased from amazon

Changing tracks

Wanting to make sure I was doing everything right, I joined a community for this, and asked some questions there. I relayed what I was planning to do, and was quickly advised against using 5v and using strips. Strips are no-good because they are rather difficult to work with, particularly up on a roof, and tend to have somewhat high failure rates. If you get an IP68 rated strip, which you should if you're using them outside, splicing out a bad pixel is a tedious affair, that involves razors, soldering, wire cutters, a hot glue gun, and heat shrink tubing. And using 5v, while functional, means you have to do a lot more power injection² than you would with higher voltages.

When asking what I should do instead, I was advised to check out a particular LED puck product, that came with channels and pucks. These pucks are available in RGBWW, have 3 LEDs per puck, and run at 12V. The 3 LEDs per puck makes these suckers bright, and the packages themselves are fairly easy to work with. And the channels are a fairly simple aluminum affair, with a backplate that you attach to your soffit, and then a front, where the pixels snap into holes, that snaps into the back plate.

Fortunately, I hadn't bought much more than a single strip of LEDs and the two power supplies, and so this pivot wasn't too difficult. I couldn't use either for this project, but thats acceptable, as they were useful enough they wouldn't spend too long in the spare parts bin.

So, with these changes, my new parts list wound up looking like this:

• 55m of LED pucks
• 55m of aluminum channels
• 1 12v power supply
• 1 Dig-Octa brain board and power board
• 1 Plastic enclosure purchased from amazon

I placed the order, and awaited delivery, which came a surprisingly short period later.

Staging everything

Upon receipt of the huge box of tracks, pucks, and other assorted items, I set to work on the floor of the living room. I flashed an updated version of WLED to my dig-octa, and started testing each string of LEDs I'd received.

Once they were tested, my wife and I installed them into the aluminum tracks

While testing them, I read more about the Dig-Octa, and found out that it supports a relay and an auxillary power source. This lets you turn off the big 12v 600W power supply, and thus turn off the LEDs, while still keeping the brain board active. The brain board will actuate the relay whenever it is toggled.³

I purchased a bunch of generic Chinese solid-state "relays" off amazon, and got one of them wired up to toggle the power supply on and off:

Finally, when all the LEDs were tested, I moved everything into an enclosure, and got it all bolted down nice and secure. This enclosure will keep things dry and clean, although I mounted it in a location where there wouldn't be much risk of dust or water ingress

A period after taking this picture, I actually wired in a small plug inside the box⁴, to plug a USB module into, so as to minimize the amount of wires going in and out of the box.

Finally, it was time to start the installation

Installation

Installation was fairly simple, albeit somewhat labor intensive. We started off by taking the backing of the tracks outside, lining them up along the roof, climbing up on a ladder, and screwing them into the soffit using self-cutting screws. I have metal soffits, so this worked reasonably well, but if I'd wooden soffits it would have been even easier. For corners, where a single 1m long track wouldn't fit, we cut the tracks, and the channels that snapped into them, with a miter saw. Aluminum is a nice soft material.

Once we got all the back tracks installed, it was time to install the front tracks. Given we'd need a couple power-injections for the longer runs, we started on the far end, and worked our way back. As before, we laid the tracks out, and where necessary, attached a power injection using heat-shrink solderless connectors. Soldering and then applying heat shrink isn't a terribly large deal, but doing it all in one pass is efficient, and the gel these connectors use to secure themselves around the wire adds extra waterproofking, akin to filling a heat shrink tube with hot glue or grease, and so its hard to beat.

.

For the segments that were less than 1m long, such as the corners, we popped the LEDs out of the tracks, using a little tool I 3D printed, cut the tracks, and either wrapped the string of pixels around the bend, or cut it and attached a connector. More heat-shrink solder splices were involved in this process.

Finally, when all was done, we turned it on, and tested it. They are very bright, and you could see them in the middle of the day, at middling brightness:

And at night, they look amazing in both color (the first image in this post) and white:

Closing

This was an amazingly rewarding project, and I cannot state how good they look at night. Some of my neighbors, who previously balked at the prices commercial operations have charged, are now interested in DIY. Ultimately I paid just shy of $2000 for this project, which includes all the components AND bringing out a contractor to help with some of the upper roof segments (I'm not comfortable on that tall of a ladder). Since everything about the control side of this is open-source, I have it wired up into HomeAssistant, and already have some automations set up, such as automatically turning them off around midnight, the ability to turn them on automatically around sunset/sunrise, and voice control ("Hey google, turn on the roof trim"). I plan to set up even more automations, such as turning them on in various team colors should they win a game in their associated sports, as well as a calendar integration, so I can schedule them to turn on at arbitrary dates, without worrying about doing it manually.

And I've already had to deal with a single puck failing, which caused all the subsequent ones to start behaving erratically. Fixing this one failed puck was rather simple; I just detached the segment with the faulty puck, took it inside, cut the bad puck out, spliced a new one in, and was up and running again in under an hour. Had I gone with strips, I couldn't have brought it inside to do that, and would have had to sit there splicing with a headlamp as my sole illumination, atop a ladder.

Updates:

• Linkrot ate the original link to the pixels I used, so I've updated it. The new link is https://www.aliexpress.us/item/3256807344866216.html

Running a minecraft server can quickly become an expensive endeavor. Even a small server needs a reasonably powerful machine to run on, and with the landscape of hosting providers, that can quickly rack up in costs. Ideally, we'd run a server when we want to play, and not when we don't, minimizing the amount of useless time we pay for. And Fly.io makes it possible to do just that.

Fly.io

Fly.io is my favorite of the "new" cloud companies. They let you quickly spin up and down virtual machines to do what you want, have a great networking stack, and aren't the most expensive. I use them a lot for little applications, as well as some of my professional endeavors on Elixir. They've kind of moved into the hole that Heroku left when they went fully corporate.

One of the particularly nice features of Fly is that you can make a server automatically start and stop, depending on traffic. So you can have a minecraft server that boots up when you want to play, shuts down when you don't, and is still publicly accessible.

Complications of running on Fly

Running a minecraft server on a more traditional VPS, like DigitalOcean, is fairly straightforwards. You just boot up the machine, shell into it, install minecraft, and configure things. This is great for how easy it is to get started, but its a very manual, interactive process. You have to do everything by hand, and then rely on the persistence (storage volume) to maintain your changes across reboots, with the general assumption that your virtual private server will always exist ¹

With Fly, or any other similar hosting provider, you don't have that same persistence. Every single boot of your Minecraft server will be "different". Sure, when you mount a volume, which you need to do for world and other persistent data, it comes along for the ride, and you can fake a persistent style manual setup. But you shouldn't have to. If you can extract all the configuration out to a file, or set of files, that can be applied to the server at boot, your server becomes more resillient; instead of having to remember how you edited what file when, you just have it in your configuration. This is the same philosophy behind systems like Nix

Other complications arise when we want to tell Fly how to run our server. The easiest way to get arbitrary binaries up on Fly is by providing a docker image. We could build our own Docker image, that contains a minecraft server and some other stuff, but that's a whole endeavor of its own, one that we might not wish to undertake. Fortunately, there's an absolutely excellent docker image for running minecraft servers already.

Enter docker-minecraft-server

docker-minecraft-server is an amazing docker setup for running a minecraft server. It's got a ton of features, including automatic plugin installation, config patching, auto-stop, and more. It's perfect for what we want to do.

Getting it installed is rather straight forwards, and the docs are excellent, but I've made a repository that reflects the way I got it set up.

My changes

If you looked through my repository, you might notice that I make my own docker container, based off the one created by itzg. Minecraft still requires a bit of manual finagling, and so I wanted to make the ecosystem within my server's deployments more pleasurable to ssh into. So I add a few utilities to the base image, set up my config patches, so they can be versioned with the git repo, and tweak a few other system settings. Most of the changes are simple things that just fit my workflow better; you probably don't need them and can run the pure itzg container.

Building my fly.toml dynamically

To deploy on Fly.io, you use a configuration file called fly.toml. This file contains almost all the information your server needs to run, barring a few things like secrets.

The trouble with writing a fly.toml by hand is that certain niceties are absent, notably when dealing with environment variables.

docker-minecraft-server makes use of envars to configure many aspects of how it runs and boots, including where and which plugins it downloads and installs. You specify these as either a newline or comma separated list of URLs or other references, which are picked up at boot, installed, and synchronized. TOML allows for multiline strings, so configuring the list of plugins isn't terribly difficult, however more dynamic lists, such as the SPIGET_RESOURCES variable, which points to resources hosted on Spigot plugin repos, are cumbersome to use.

Specifically, SPIGET_RESOURCES wants a comma-separated list of id numbers, and nothing else. And spigot resource urls are rather descriptive, but the id number is not. I wanted a solution that would let me use the "full" spigot urls, but take advantage of the spiget downloader feature, which manages version updates for me.

Finally, I wanted to make use of fly's PROXY_PROTOCOL support. PROXY_PROTOCOL allows passing of proxy information to servers and other applications, and is relevant to our usecase here because, if turned off, all incoming connections to our minecraft server won't resolve as their "real" IP, but rather a fly internal IP. But I wanted to be able to turn this off and on, and it requires configuration in a few places to do so.

Aiming to solve all these problems, I wrote a simple little deno script. This script is fairly simple, and mostly just does some string concatenation, but it does let me do things that the plain old TOML wouldn't.

I can set a single value, enableProxy, and have it set up both the fly port setup AND the envar, which is used by a patch to enable the proxy support on paper, and by docker-minecraft-server, to enable the auto-stop system to monitor our server.

I can also take full spigot URLs, strip the non-numeric-id portions, and render them out to a format that the docker container is happy to use.

I don't make use of this feature, but since this is just a script file, I could also move the configuration to be generated in a much more composable manner, or to use local .env files, or any number of things.

Since I used deno, we also get the advantage of the script being "self contained". By using a custom shebang, I've made my script executable, so to build a new toml file you just have to run ./fly.ts. No need to install packages (other than Deno), no need to remember which runner to use.

Deploying, and things to note

Once the new fly.toml is generated, deployment and running the server is an absolute breeze.

./fly.ts && fly deploy gets the server up and running, and I can connect to it in Minecraft, as expected. All the plugins and configuration I've specified in config files have been loaded onto the server, and any appropriate config patches have been applied.

Manual configuration, as always with Minecraft, takes a long time, but isn't tremendously difficult, just tedious. Things like LuckPerms, WorldEdit, WorldGuard, and your "basic" plugin of choice (I use CMI) need to be configured, same as always. You can do these configurations in a variety of ways; adding them as custom COPY commands to the Dockerfile, using docker-minecraft-server's patching system, or just by shelling into the server (via fly ssh console) and editing them by hand.

If you use premium plugins, you won't be able to automatically download them, as they likely require authentication to download. You can make use of the /plugins attach point to load these plugins into your docker container, at which point the minecraft scripts will pick them up and put them in the right place.

The server makes use of a few values that shouldn't be exposed in your plaintext config, but rather set as "secrets". Fly has a feature for this, where you simply set them via a command

fly secrets set RCON_PASSWORD=$(openssl rand -base64 32)

This will be exposed as an envar in the container, which handles all the RCON stuff for you, and works nicely

AutoStarting, and why I didn't set it up

One of the more powerful things about this config is that you can make use of both autostart and autostop.

Autostop is handled by the minecraft server container; it monitors connections to minecraft, and kills the process after a configured duration, causing the Fly vm to shut down. This is "safe", and is what I'm using. It lets you quit the game and not rack up a big bill because you forgot to quit the server. When configured with powerful enough anti-afk features in Minecraft itself, you can prevent issues where a player causes the server to be up endlessly through negligence.

However, AutoStart is a more complicated beast. AutoStart exists as a part of fly's systems, not as something that's minecraft aware. It works fine with minecraft, but has one very large caveat: any TCP traffic on your server's exposed port(s) will boot the server. If you're trying to save money, this isn't great, because it means a random server scraper, a nefarious script, or even someone just letting the minecraft server listing sit open, will keep booting your server again and again and again.

Because of this, I elected to manually start my servers, and let autostop handle the rest. Starting is trivial enough, simply run fly apps restart and your server boots almost immediately.

If you run your servers entirely on a private network, then this isn't so much of an issue, as you have less risk of bad actors. However, you still have risk, as negligence on behalf of one of your players could keep the server alive for a very costly period of time.

If you run HomeAssistant in a libvirt-based VM, such as in a qemu backed system, and you want to forward USB dongles (such as for z-wave or zigbee) from the host to the guest, you might run into issues where the dongle doesn't reconnect when the VM restarts, or when the host restarts. This can be quite frustrating, as any devices and automations you have that are tied to that dongle will not work until you manually reconnect it to the guest.

Manually

For a while, I was just doing the reconnects manually, via a simple little script:

#! /usr/bin/env fish

if test (whoami) != "root"
  echo "Must run as root" >&2
  exit 1
end

set -l cyme "/home/jeffs/.local/share/mise/installs/rust/latest/bin/cyme"
set -l serials XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

for serial in $serials
  set -l temp (mktemp /tmp/hass.XXXXXXXXXX)
  set -l usb_info ($cyme --filter-serial $serial --json)
  set -l busnum (jq -r '.[].location_id.bus' (echo $usb_info | psub))
  set -l devnum (jq -r '.[].location_id.number' (echo $usb_info | psub))

  echo -e "
<hostdev mode='subsystem' type='usb'>
  <source>
    <address type='usb' bus='$busnum' device='$devnum' />
  </source>
</hostdev>" > $temp

  while true
    virsh detach-device hass $temp; or break
  end
  virsh attach-device hass $temp
end

This script isn't really anything special; it uses cyme instead of lsusb, as a previous incarnation of the script proved brittle around parsing the output of lsusb, and attempts to get the bus and device # of my two dongles (z-wave and zigbee), writes a quick temporary xml file for virsh to consume, and then asks virsh to detach any devices at that bus/device number, and then asks it to reattach them.

It runs through the detach step multiple times, as I've had issues with virsh getting multiple attachments to a single bus/dev# in the past, exhausting the number of device passthroughs, and getting into an error state. It's probably not necessary, but I keep it around, as its harmless if it does nothing.

Running this script manually still works, but I don't want to have to shell into the host and run it every time there's a reboot to either the host or the guest, and so I looked at automating it

UDEV rules

Initially, I tried running a variation of this script via udev rules, with the following udev rules file:

# ZIGBEE
SUBSYSTEM=="usb", ATTRS{serial}=="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", RUN+="/usr/sbin/hass-reattach-usb.fish"
# ZWave
SUBSYSTEM=="usb", ATTRS{serial}=="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", RUN+="/usr/sbin/hass-reattach-usb.fish"

Those rules would dispatch the following script

#! /usr/bin/env fish

set -l logFile "/var/log/hass-usb-reattach.log"
set -l virshDomain "hass"
set -l tmpfile (mktemp)

set -l deviceType

switch $ID_SERIAL_SHORT
  case "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    set deviceType "Zigbee"
  case "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    set deviceType "Z-Wave"
  case "*"
    exit 0
end

if not set -q BUSNUM; or test -z "$BUSNUM"; or not set -q DEVNUM; or test -z "$DEVNUM"
  echo "NO SUCH LUCK BUDDY" >> $logFile
  exit 0
end


set -l hostdev "
<hostdev mode='subsystem' type='usb'>
  <source>
    <address type='usb' bus='$BUSNUM' device='$DEVNUM' />
  </source>
</hostdev>
"
set hostdev (string trim $hostdev)

printf $hostdev > $tmpfile

echo (date) ": (Re-)attaching $deviceType (B:$BUSNUM D:$DEVNUM)" >> $logFile

virsh detach-device "$virshDomain" "$tmpfile" &>1 >> $logFile || true
virsh attach-device "$virshDomain" "$tmpfile" &>1 >>

rm $tmpfile

exit 0

This worked in the case that the USB devices were added after the VM was running. But it didn't work for initial boot of the host. Which was the biggest and originating problem I was trying to solve.

libvirt hooks

libvirt has a hooks feature, where it will run certain hook scripts on the host OS at various points. Of interest to us is the started point, for qemu.

I wrote up a variation of the first script, with some wrapper code to only dispatch on started events. Unfortunately, it never actually did what I needed it to do.

The script would run, and attempt to mount a device to the guest OS. But it would run too soon after the guest was started, and was unwilling to take new devices, and so would just hang there, eventually timing out, and not adding the new device, requiring a manual intervention.

I meant to post the script example I used here, but found out that I actually deleted it in frustration when it didn't work. You're not missing out on much, as the script was mostly just an if test to see if we're in a started event, an if test to see if the domain is one we care about, and then a dispatch to the manual script.

Attaching the devices when HomeAssistant is online

If we look at what we're actually trying to do here, we're trying to get these devices mounted in HomeAssistant. We don't really give a hoot about the underlying state of the guest OS, and so a libvirt status of started, the equivalent to a power light being illuminated on a physical machine, is of no real importance. If we could get HomeAssistant to somehow tell the host OS when it was ready, we could then run the attach script, and have everything just work.

A lot of people will use SSH to make their HomeAssistant guests talk to the host. I was a bit squeamish about this, it seemed like an easy way to open an attack vector, should my HomeAssistant installation get compromised somehow. I could ssh into a user with limited permissions on the host, but "limited" is a misnomer, as the user would still need access to libvirt, via the virsh command and the ability to query all the USB devices on the system. Not exactly a light set of permissions.

I also thought about setting up a small http server, which would handle dispatching the script when HomeAssistant calls it; essentially a webhook. While this would undoubtably be far more secure than full-blown SSH access, it wasn't really something I wanted to muck about with. I've had enough experience trying to tighten down a webhook to only respond to a "real" client, and ignore fake clients, that I didn't really want to muck with it for what should just be a simple process.

Finally, I realized that HomeAssistant's MQTT integration sends birth and last-will messages to homeassistant/status. I use MQTT to monitor my power and gas meters, as well as bringing real-time data from my WeeWX weather station into HomeAssistant, and so using it for something else was great. Having the homeassistant/status messages automatically emitted from HomeAssistant itself means I wouldn't have to create an automation to emit a message at boot, simplifying the number of moving parts.

Running a simple MQTT client, to listen to a particular topic, and run a small program when a message comes in on that topic, is fairly trivial. I used the mosquitto_sub program, which comes with the mosquitto-clients on most linux distros. This program simply connects to a server, listens to a topic, and emits messages to stdout. A script to consume these messages, and dispatch the attach script, was rather simple:

#! /usr/bin/env fish
# You can run this by hand, but it expects to be run by a daemon
# see hass-device-passthrough-listener.service
set -l host homeassistant.local
set -l username vm-host
set -l password 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
set -l topic homeassistant/status

echo "Starting listener service"

mosquitto_sub -R -h $host -t $topic -u $username -P $password | while read -l line
  echo "[MSG] $line"
  if test $line = "online"
    echo "Attempting to run passthrough script"
    /home/jeffs/homeassistant/device-passthrough.fish
  end
end

Running the script, and sending fake online messages with the HomeAssistant MQTT publish service, worked beautifully. Since I wanted this to run all the time, from boot to shutdown, I whipped up a systemd service file, that starts it and keeps it running:

[Unit]
Description="Hass device passthrough listener"
Requires=libvirtd.service
Requires=network-online.target
After=libvirtd.service

[Service]
Restart=always
RestartSec=30
ExecStart=/home/jeffs/homeassistant/mqtt-listener.fish

[Install]
WantedBy=multi-user.target

After getting this loaded and started with HomeAssistant, I brought down the VM by hand, and watched via journalctl as the script received a message, kicked off the attach script, and then went back to idling. After this test was successful, I restarted the whole host, and saw it work with success as soon as HomeAssistant was ready.

For my home automation needs, I use HomeAssistant. And for more complex automations, I used to use NodeRED. Recently I undertook some effort to move over to DigitalAlchemy, which lets you write automations in Typescript. Here's what I learned in the process.

Home Automations via HomeAssistant

The platform I use for most of my home automation things is HomeAssistant. It's a powerful platform, and will probably show up in other blogposts in the future. HomeAssistant is great because it lets me integrate many services into a single automation framework. It's easy to extend, and serves my needs well. I moved over to it from HomeSeer a few years ago, and the migration has been for the better in almost all regards.

Out of the box, HomeAssistant has a fairly powerful automation system built in. It is comparable to that of HomeSeer, or more widely known systems such as Tasker on Android and Shortcuts on iOS. Essentially, you have a collection of triggers, conditions, and steps for your automation.

For example, if you wanted to turn on a light every time the sun came up, you can do that fairly easily. You'd set a trigger that listens for a state change in the sun integration, and when the trigger sees that sun has changed to up, you'd fire your automation to turn on the light.

You can do far more complicated automations with the built-in system, although it tends to get somewhat unwieldly. Automations are, at the core, a set of discrete steps that flow one into the next. You can add parallelism with some useful constructs, but they very much adopt a trigger-condition-action structure, and breaking out of this becomes confusing quickly.

Enter NodeRED

Many people in the HA community make use of an excellent package called NodeRED. NodeRED aims to make more powerful and complex automations far easier. It works through a flowchart paradigm. You drag out blocks that represent either triggers, actions, or primitive control logic, and literally "wire" them together, via lines in the UI, to do what you want. Automations that are hopelessly complex in pure HomeAssistant become simpler in NodeRED.

However, it isn't without its flaws. The developers and community around NodeRED seem averse to introducing too many "programmer-y" constructs. There are no primitive boolean gates, if/else objects, or simple comparators. You can do all these things, but it isn't immediately obvious, particularly if you come from a programming background.

Additionally, there aren't many ways to "dry" up a NodeRED flow. Say you have many similar, but ultimately different, flows. In programming, you'd just make a common function that handles what it can, and use it where appropriate. You can try to make a sub-flow, but this is limited, and often winds up being more complicated than just copy+pasting nodes everywhere.

Variables are kind of a mess; each flow can have its own, and there are global variables as well. They work well enough, but they're not really surfaced too much by the UI, and so if you don't remember (or leave a note) where a variable comes from or goes, it is largely opaque.

Finally, NodeRED flows are difficult to store in a version management system. They're not generally human-readable when serialized out to JSON, and ultimately insignificant edits to the "shape" of a flow can result in massive diffs.

DigitalAlchemy Cometh

For a few years now, I've shrugged my shoulders and accepted that NodeRED was the best we could get. There are other script-based automation platforms based around Python, but I really don't care for python, and so don't want to use it for my HomeAutomation if I can avoid it.

Earlier this year, a developer named Zoe released DigitalAlchemy, which strives to be a powerful automation framework for HomeAssistant, centered around use of TypeScript for its automation.

I don't mind JavaScript and its friends. It's not my favorite languate family, not by a long shot, but its not repellent to me either. For something like home automation, it's event oriented system, easy async, and friendlyness to functional programming paradigms make it an obvious choice.

DigitalAlchemy is a wonderful little framework, that uses many of the features TypeScript exposes to give you a very powerful way of writing and maintaining automations. It generates TypeScript types for all of your HomeAssistant integrations, so when you're editing your scripts, you get both code completion, but also sanity checks. No more trying to pass a fan to the Light: Turn On service.

And since its real TypeScript, you can use real JavaScript techniques when programming. Want to use a powerful state machine to handle your automations? You can do that, trivially. Just include one of the ones you find on npm, write the glue code, and away you go.

Over the course of a month, I ported all of my automations out of NodeRED and into DigitalAlchemy. The process was rather straightforwards, and the resultant automations are far simpler than their predecessor. I can open one up in VSCode and immediately start working, without having to remember too many weird quirks about how the system works. Feedback is fast, and so iteration times are equally fast. There's even a repl for testing ideas out on the fly.

And the lead developer, Zoe, has been spectacularly responsive. Questions and suggestions get met with near-immediate fixes or implementations, sometimes multiple iterations within the hour of first being brought up. They've recently been working on some tremendous improvements to the DigitalAlchemy synapse system, which lets you create and expose virtual entities in HomeAssistant from your DigitalAlchemy scripts, allowing a more two-way flow of control for scripts. It's really quite powerful; I've used it in several of my scripts to provide "circuit breakers," so I can do things like turn off the smart-away automations when I've got guests over.

Use SyncThing to make editing far more enjoyable

If you are running Digital Alchemy through the HomeAssistant platform (quickstart), you might get tired of editing your automations in the browser based VSCode interface. While you can use systems like SMB or SSHFS to mount your HomeAssistant filesystem for editing, this does impose a performance penalty.

Instead, I suggest using the SyncThing addon for HomeAssistant, and setting up a sync between your development machine and your HomeAssistant install. You can then edit and build your automations locally, and just restart the DA addon in HomeAssistant when you're ready to run them.

Addendum

If you're interested in seeing how DigitalAlchemy scripts look in practice, I encourage you to check out my implementations.

I recently needed a new garden hose reel for the front yard, and wanted something a bit more convenient than what I had before. I've had a hose cart I bought from Home Depot years ago, and its worked reasonably well, but its ugly as sin. I've had rather good luck with an Eley hose reel in the back, where I need 200 feet of hose, but don't want something that large and cumbersome for my smaller front yard. Enter the Hoselink hose reel.

What I wanted

I didn't really want too much more than my old hose reel provided, and a lot of these things are what I feel should be standard for any hose reel. It shouldn't leak. It should be easy to spool hose out, and easy to reel it back in. If it comes with a hose, the hose should be durable and "feel" nice. Finally, the physical construction of the reel should inspire confidence; it shouldn't feel rickety and cheap.

Features in the "would be really nice to have" column are things like the ability to mount the reel on the wall, while being able to easily bring it into the garage during winter, and a self-retracting hose reel. The last one is a point of contention, as (before I bought the HoseLink) I've never found one that held up particularly well, and was quite averse to them, despite wanting them to work.

Visiting big box stores, the options are scarce. There are usually a few cheap, plastic hose reels for sale, and maybe a single metal one. None of them inspire any confidence. The plastic ones look like they'd crack after a single summer in the sun, and the metal ones look as if they'd rust up in a few months. The winding mechanisms look labor-intensive, not something a small child could handle. And, more than anything, they're all fantastically ugly. I don't really expect too much in the way of appearance from something as utilitarian as a hose reel, but it shouldn't look like a piece of cheap trash either.

So I looked online. Loads of wall mount, self retracting reels exist, but most of them have mediocre to terrible reviews. But one does stand out. The HoseLink reel. Youtubers such as SilverCymbal rave about the quality of the HoseLink reel, and his taste has generally been good in the past, so I decided to try one out. They are somewhat expensive, nearly the same price as my all-metal Eley, so I was hoping it would impress me with its quality.

The Reel itself

Ordering was simple enough, and the package came with the usual expediency of modern shipping. A large, somewhat heavy reinforced cardboard box arrived, and had my hoselink, wall-mount kit, a sprayer attachment, Y-fitting, a few quick disconnects, and a well written and easy to follow manual inside. The box immediately became a play fort for my daughter, while I was mounting the unit to the outside wall.

Mounting was easy enough; there's an included template, some brick anchors, and simple instructions. I have brick walls, and so set up with a fresh masonry bit and a hammer drill, and went to town. The anchors worked better than some I have used; I didn't have to inject any epoxy into the holes to ensure fastness. Part of what makes mounting so easy is that you don't actually have the big heavy hose reel hanging off the wall while you mount it. You mount an L-Shaped bracket, and the HoseLink has a post that slots into the bracket, allowing for 180º of movement.

The reel itself is rather spartan and unobtrusive. Nothing screams "fancy hose reel", which is nice. It looks far better than what was for sale at the big box stores. The lead hose is a bit long for my use case, but I'm sure that's attractive when you don't have a faucet so conveniently located. There are no external protuberances on the reel itself either, it's a very compact, clean affair.

The back of the reel, above the mounting pin, has a fold down handle, which is useful when lifting the reel out of the bracket for storage. When not in use, the handle practically disappears up against the body of the unit.

The hose is of decent quality, with brass fittings at either end, and no visible leaks from anywhere. A large ball is mounted about a foot from the business end of the hose, ostensibly to prevent it from retracting all the way into the reel should you wind it back up without an attachment.

It works pretty much as you'd expect it to. Grab the hose, pull out the length you need, and it retracts a couple inches and then locks in place. When it's time to reel it back in, just give it a gentle tug, and the reel will start to wind the hose up, like the cord on a vacuum cleaner. To stop it at any length along the way, just pull it back out a little, and the latch will reengage and leave you with a shorter length of hose. This is quite useful when transitioning between watering tasks, as you can easily prevent coils of hose from just sitting out.

The Sprayer and Quick Disconnects

HoseLink's website gushes about their quick disconnects and the included sprayer wand, both of which I was skeptical about when purchasing, but since they were add-ins at no extra cost, I didn't quibble.

The sprayer is well-made, and feels nice in your hand. The shut-off valve sits comfortably under your thumb, and spray patterns are the usual ones you'd expect to find. The shower is a bit intense for very delicate plants, but that isn't unique to the HoseLink sprayer; for seedlings my wife and I like to use a Dramm spray head, which generates an exceptionally gentle stream.

The quick disconnects are well-made too, although they aren't my favorite. They work using a symmetric bayonet style latch, locking together when you line up the barbs and give a quarter turn. They didn't leak while I was using them, but I did find the bayonet design somewhat cumbersome; once I thought I'd seated it well, only to have it spray water everywhere, as one barb never seated into its channel on the opposing face. There is a raised indicator in the plastic for lining up the two halves before locking together, but it's something you'd have to watch and feel for, not something you can just do blind.

I did like that they included a quick disconnect that has a water shut-off valve built in. I've had some brass shut off valves living on the end of my other hoses for a few years now, and find having the ability to control the amount of water at the business end of the hose invaluable. I would have moved one over to the new hose reel, but in the few years I've had it, it managed to permanently attach itself to the end of the old hose, so a new one was in order.

I bought the somewhat expensive Eley quick disconnects and an Eley shutoff+swivel. While these are far more expensive than the competition, even ones from Dramm, they are made far better than any I have ever used. They are big, well-made brass units, with full flow-through, so you don't impede the water flow, and the quality of the machining is apparent, their operation is incredibly smooth. You can connect and disconnect attachments without having to look and watch what you're doing, one-handed. The plug end is somewhat demure, adding very little bulk to whatever it's attached to, and slotting into the socket end extremely securely. You can see the plug attached to the sprayer in the earlier image. When attached, they kind of act like a swivel, although they aren't 100% free spinning, there's a bit of friction that makes a swivel a nice accessory.

The eley shutoff is equally well-made, and very easy to thumb on and off. It just works so well its unremarkable, and therefore difficult to write about.

As I dive deeper into HomeAssistant automations, one thing has been nagging at me: my energy usage statistics. I have a solar array, battery, and associated systems installed on my house, and they integrate somewhat well with HomeAssistant. However, due to the way my house was wired, the solar/backup system is only aware of power usage for roughly half the house; the air conditioner and some other heavy appliances are on a separate circuit, that I chose not to have backed up due to their heavy loads. As such, any statistics in HomeAssistant related to power were only partially true; they were missing an entire chunk of my annual power consumption, leading to the values reported in HA differing from the values reported on my bill. I wasn't terribly happy with this, and so over time I've been investigating solutions.

Background

HomeAssistant's energy dashboard and integrations

Back in 2021, HomeAssistant added a pretty robust energy integration. Their system could integrate with gas and electric systems commonly found throughout Europe, via a standard called P1. Unfortunately, we don't have that standard here in the USA. When that post came out, I looked it over with interest, saw that it wasn't applicable to the USA, and filed it away under the "some day" file.

There were approaches for pulling power usage information from US systems; some electric utilities provide an API for access to their data, and there are many such integrations built into HomeAssistant, ripe for the pickings. Other people use current-sensing clamps, placed around the power as it enters their breaker box, or, for more granular data, around each circuit as it leaves the breaker box. Still other solutions exist, such certain smart home products that integrate and try to read electrical usage signatures to determine what is using power, or OCR systems that use a camera to watch their electric meter directly. But none of these really appealed to me; I didn't want to shove a bunch of current sensing clamps around each wire in my breaker box¹, and my meter is visible from the street, so a camera in a place that could accurately read the meter was a non-starter. So my usage mostly just stuck with the half accurate measures I was getting.

Smart Meters in the USA

In the USA, most smart meters send data along the ISM band, in a protocol called Encoder Receiver Transmitter, by Itron corporation. This applies to electric, gas, and water meters. Said protocol has a variety of message formats, but they're all fairly easy to work with. The meters generally have a barcode or similar tag on them, usually near the word/logo for "Itron", with an 8 digit numeric ID.

RTLSDR

I've been aware of RTLSDR for a while, always minorly interested in it, but never actually doing anything with it. I'd heard rumblings over the years that people were using it for all sorts of interesting use cases, such as modernizing old 433MHz burglar alarm systems, reading data from weather stations, and so forth. But what really piqued my interest was reports of people using it to read data sent from smart meters.

Basically, you use a cheap radio dongle, like a DVB-T television tuner, as the raw radio, and then software (the S in SDR) to process the data you get and make it useful.

Since the smart meter is going to send the data regardless of who is reading it, and it's the actual data the electric company uses for billing, it seemed a prime candidate for my usage.

RTLAMR

There's an excellent GitHub project, which is the backbone of my setup, called rtlamr. It works with data output via rtl_tcp from the rtl-sdr project, to talk to one of the aforementioned cheap dongles, listening in for messages from meters, optionally filtering by meter IDs. When it sees a message it understands, it decodes it, and prints it out as a formatted JSON object.

If you're not sure of your meter, you can just listen for all packets, and then pick yours out of the soup you get back.

Taking the outputted JSON data and sending it to HomeAssistant is a mostly trivial step, and there are already a few projects that do it.

My Setup

Prerequisites

First things first, you'll need a HomeAssistant installation running. You can still graph and chart all the data provided by rtlamr without it, but you're on your own.

With said HomeAssistant installation running, you'll also need to have the MQTT integration running with an MQTT Broker. I use EMQX becuase I like Erlang and graphs and everything else, but Mosquitto works just fine. HomeAssistant claims RabbitMQs MQTT features don't work well for their use cases, and I never spent the time to prove them wrong.

The dongle

I purchased an RTLSDR Blog v4 from Amazon.com, as these have had good reviews over the years, and provide a nice clean data stream. People have had good luck with DVB-T shields for RPis, or USB DVB-T shields, but the price difference isn't that great, and so I went with a device I knew would work. I purchased the dongle without an antenna, as I wasn't quite sure where I was going to operate it. I have a few SMA connector antennas lying around, from other projects, but ultimately I went and bought a small antenna from a local RadioShack² for this. There are many antennas that will work with this on Amazon, such as this one, which looks very similar to the one I purchased from RadioShack. You just want to make sure the antenna you purchase works well in the 900MHz frequencies.

When you actually get the dongle, it's rather uneventful for setup. Plug it into your computer of choice, be it an RPi, server, or whatever else. You don't have to run this on the same machine HomeAssistant is running on. If you can't see the meter's readings where your server is, you can put the dongle/antenna on an RPi or other small computer in a place that can see the readings, as it will talk to HomeAssistant via MQTT.

HomeAssistant Integration

Searching around the internet lead me to this repo by RagingComputer. It looked like it might do exactly what I wanted; that is, package up RTLSDR, RTLAMR, and a bit of code to send messages across MQTT. However, in experimentation I was unable to get it running. There was an issue reported to the GitHub repo already, from two years ago, and the last commit was four years ago, so I figured the project was dead.

Feeling a bit lazy, I didn't really want to fork and fix it myself, so I set out to see if anyone else had solved it. Enter Allan Gomez GooD. He's got an excellent repo that does the same thing as the amridm2mqtt repo, but has gone the extra mile and built it out as a custom HomeAssistant addon. If you're running the dongle on the same system as your HomeAssistant installation, this is perfect. You can install the whole addon with a couple of clicks, and there are even big friendly buttons in the readme for doing just that.

If you run it on a separate computer, It's not much more complex, and the readme covers that as well

Configuring RTLAMR

Once you get it installed, either via HomeAssistant or as its own thing, you need to configure it. Configuration is rather straightforwards, and well documented.

For me, I went into EMQX and set up a custom user account for RTLAMR to send data to my MQTT system with. It can read and use the credentials HomeAssistant already has in its built in MQTT integration, but I like to keep things separated out; makes debugging and security easier.

Once I had that part configured, I had to figure out what meters were mine. I knew the number of my meter, it's printed on the front of the device, but that's where my knowledge stopped. I didn't know what format of messages it would send, and I didn't know how it would represent the data for import vs export (remember, I sell solar power back to the electric company). How would I figure this information out?

Fortunately, RTLAMR has you covered. There's a mode for listening to all meters, that can be dispatched as a one-off docker command:

docker run --rm -ti -e LISTEN_ONLY=yes -e RTL_MSGTYPE="all" --device=/dev/bus/usb:/dev/bus/usb allangood/rtlamr2mqtt

Running that on the machine with the dongle attached, you'll see the raw JSON messages as they are decoded. In my case, I saw big clusters of about 10 messages every 2 minutes, and then every 10 minutes or so I saw an even larger cluster of different messages.

Looking into the messages, I saw one message that had my meter number on it, along with a reading that was pretty close to what was on the front of the meter:

{
  "Message Type": "SCM",
  "ID": #######4,
  "Type": 8,
  "TamperPhy": 0,
  "TamperEnc": 0,
  "Consumption": 1552536,
  "ChecksumVal": #####
}

But immediately following that, I saw two more messages, with their IDs just incremented off mine by 1 each time

{
  "Message Type": "SCM",
  "ID": #######5,
  "Type": 8,
  "TamperPhy": 0,
  "TamperEnc": 0,
  "Consumption": 644391,
  "ChecksumVal": #####
}
{
  "Message Type": "SCM",
  "ID": #######6,
  "Type": 8,
  "TamperPhy": 1,
  "TamperEnc": 1,
  "Consumption": 908145,
  "ChecksumVal": #####
}

The message with the ID ending in 5 turned out to be my export, and the message ending with the 6 was the delta between the import and export. Why they send all three is a question for the power company. With the information I had, I could configure RTLAMR. My message type was SCM, and I had the two IDs for the meters I wanted. Entering those values in the config, I exited the listen all mode, and started up the HomeAssistant integration.

2 minutes later I had a reading, and my data was now ready to use.

I spent a bit more time faffing about trying to find an interval that didn't leave RTLAMR always listening, while keeping the data fairly fresh. By default, it sleeps for 5 minutes (300s) after a successful packet read, but through using the LISTEN ALL mode, I discovered my meter sends pretty accurately every 2 minutes. Since 5 does not divide cleanly into 2, I set my sleep interval to just shy of 120 seconds, so I'm getting every packet my radio sends, without wasting much effort.

You could probably do something similar just by watching the MQTT messages, using something like MQTT Explorer, but be aware of one caveat. If your meter doesn't report a change in the value, then no message will be sent to MQTT. I'm not sure if this is my server dropping duplicate messages or a feature built into the RTLAMR system I'm using, but it does reduce network congestion, so I'll take it.

If you leave the default configuration the addon came with, it will automatically create new sensors in your HomeAssistant install for all the meters you've added. If it doesn't see a meter you've told it to watch for, it won't add it till it sees it. This is the current situation of my Gas Meter, which I suspect only reports data when polled.

Configuring HomeAssistant

Without Tariff Data

If you don't care about how much your utility usage costs, or just have a single flat rate, I envy you. You get a much simpler configuration.

Head over to your Energy dashboard in your HomeAssistant installation, and open up the config page. Under the relevant section, in my case Electricity Grid, add your relevant meters. For me, it would be the import meter under the consumption section, and the export meter under the Return to Grid section.

And that's it. You now have usage stats reported accurately. In a few hours, you'll see accurate usage information.

With Tariff Data

This is a somewhat more complicated flow, and involves the setup of multiple Utility Meter helpers, automations, and more. But if you want to get as accurate a picture of your costs, its worth it.

For me, my electric company breaks things down into a few tariff structures for purchasing:

• Summertime
  • First 400 kWh: $0.090279 / kWh
  • All additional power: $0.11721 / kWh
• Wintertime
  • First 400 kWh: $0.079893 / kWh
  • All additional power: $0.103725 / kWh

There's a similar, but simpler, pricing structure for export power:

• Summertime: $0.05636 / kWh
• Wintertime: $0.04745 /kWh

When I was just running the powerwall data, I made use of a template that just tracked time and cumulative usage from a single "utility meter" integration that reset with my billing period, and flip-flopped depending on the data, but this had the flaw that it wasn't accurately tracking the difference between the first 400 kWh and the remaining power usage. As soon as it flipped to the "higher" usage numbers, it could throw off older calculations.

The "right" way to do this is by creating utility meters for each bin of power usage, and using an automation to switch the currently active utility meter.

Utility Meters

For my use case, I set up 5 utility meters, although you could make due with 3. I just like to have the extra day tracking for my own purposes.

I've set my utility meters to reset on the 27th, as that's the end of my billing cycle. I then create two meters, one that has tariffs applied, and one that does not. The one without tariffs is used to just track cumulative usage across the whole period. The meters with tariffs configured will actually show up as multiple different meters in HomeAssistant. Sure, you could sum up the values of the current seasonally active meters to get your 400 kWh threshold, but doing that logic in templates or automations is always a little more brittle than I like.

Once you've got the meters set up and have reloaded your HomeAssistant config, you should add all the tariff'd meters to your Energy Dashboard Config along with their pricings as fixed values.

To switch between the bins, use an automation.

Active Meter Automation

You can do this pretty easily with native HomeAssistant Automations, but I didn't. I prefer to use NodeRed for my automations, and came up with something like this.

The "Set Season Daily" group starts with a timer that triggers every night at midnight, as well as on startup. This triggers a JavaScript function, which sets a NodeRed flow variable "summer" to a boolean true or false, depending on the current date. This JS function also splits output to two branches, depending on summmer. From these two branches, the current bin for the export power tariffs are set via a pair of select option service calls.

The Trigger state block in the Set Import Tariff group outputs depending on if the monthly meter without any tariff data reports greater than 400 kWh. Both branches, the greater than 400 kWh and the less-than, output to a switch statement, which reads the previously set flow variable for "summer time", and then calls a select option service call that sets the active buckets for the two import utility meters.

[{"id":"7030ec85714e521d","type":"tab","label":"Electricity Tariffs","disabled":false,"info":"","env":[]},{"id":"c2be4e7700b3d8e5","type":"group","z":"7030ec85714e521d","name":"Set Season daily","style":{"label":true},"nodes":["163ac22ed1d95d60","51c6c4d86a4f87b8","1f1ac14da602b108"],"x":34,"y":293,"w":618,"h":174},{"id":"1d6ef5cec40e2629","type":"group","z":"7030ec85714e521d","name":"Set Import Tariff","style":{"label":true},"nodes":["02ad35cf4db4cbd1","4bfdf3be32ae39ac","29c18e062e55fe14","ac28a1977f2b43f6","05cce05de54f7613","871107a9446c7041","35a21073487e74fc"],"x":34,"y":19,"w":972,"h":242},{"id":"1f1ac14da602b108","type":"group","z":"7030ec85714e521d","g":"c2be4e7700b3d8e5","name":"Set export tariff","style":{"label":true},"nodes":["0af23044806f9006","03d08d57cb753206"],"x":454,"y":319,"w":172,"h":122},{"id":"163ac22ed1d95d60","type":"eztimer","z":"7030ec85714e521d","g":"c2be4e7700b3d8e5","name":"","debug":false,"autoname":"0:00:00","tag":"eztimer","topic":"","suspended":false,"sendEventsOnSuspend":false,"latLongSource":"haZone","latLongHaZone":"zone.home","lat":"41.1145565060444","lon":"-111.91224648624485","timerType":"2","startupMessage":true,"ontype":"2","ontimesun":"dawn","ontimetod":"0:00:00","onpropertytype":"msg","onproperty":"payload","onvaluetype":"num","onvalue":1,"onoffset":0,"onrandomoffset":0,"onsuppressrepeats":false,"offtype":"1","offtimesun":"dusk","offtimetod":"dusk","offduration":"00:01:00","offpropertytype":"msg","offproperty":"payload","offvaluetype":"num","offvalue":0,"offoffset":0,"offrandomoffset":0,"offsuppressrepeats":false,"resend":false,"resendInterval":"0s","mon":true,"tue":true,"wed":true,"thu":true,"fri":true,"sat":true,"sun":true,"x":120,"y":360,"wires":[["51c6c4d86a4f87b8"]]},{"id":"51c6c4d86a4f87b8","type":"function","z":"7030ec85714e521d","g":"c2be4e7700b3d8e5","name":"function 1","func":"const month = new Date().getMonth();\n\nif (month >= 5 && month <= 9) {\n    flow.set('summer', true);\n    return [{payload: true}, null];\n} else {\n    flow.set('summer', false);\n    return [null, { payload: true }];\n}\n\n","outputs":2,"timeout":0,"noerr":0,"initialize":"// Code added here will be run once\n// whenever the node is started.\nconst month = new Date().getMonth();\n\nif (month >= 5 && month <= 9) {\n    flow.set('summer', true)\n} else {\n    flow.set('summer', false)\n}","finalize":"","libs":[],"x":280,"y":360,"wires":[["0af23044806f9006"],["03d08d57cb753206"]],"outputLabels":["summer","winter"]},{"id":"02ad35cf4db4cbd1","type":"trigger-state","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"","server":"ffea7422.3895a8","version":4,"inputs":0,"outputs":2,"exposeAsEntityConfig":"","entityId":"sensor.electricity_import_month_total","entityIdType":"exact","debugEnabled":false,"constraints":[{"targetType":"this_entity","targetValue":"","propertyType":"current_state","propertyValue":"new_state.state","comparatorType":">=","comparatorValueDatatype":"str","comparatorValue":"400"}],"customOutputs":[],"outputInitially":false,"stateType":"num","enableInput":false,"x":250,"y":120,"wires":[["4bfdf3be32ae39ac"],["29c18e062e55fe14"]],"outputLabels":["gte 400","lt 400"]},{"id":"4bfdf3be32ae39ac","type":"switch","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"is summer","property":"summer","propertyType":"flow","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":710,"y":80,"wires":[["ac28a1977f2b43f6"],["05cce05de54f7613"]]},{"id":"29c18e062e55fe14","type":"switch","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"is summer","property":"summer","propertyType":"flow","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":710,"y":200,"wires":[["871107a9446c7041"],["35a21073487e74fc"]]},{"id":"ac28a1977f2b43f6","type":"api-call-service","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"summer gt 400","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_import_day","select.energy_import_month"],"data":"{\"option\":\"summer_gt_400\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":900,"y":60,"wires":[[]]},{"id":"05cce05de54f7613","type":"api-call-service","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"winter gt 400","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_import_day","select.energy_import_month"],"data":"{\"option\":\"winter_gt_400\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":890,"y":100,"wires":[[]]},{"id":"871107a9446c7041","type":"api-call-service","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"summer lt 400","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_import_day","select.energy_import_month"],"data":"{\"option\":\"summer_lt_400\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":850,"y":180,"wires":[[]]},{"id":"35a21073487e74fc","type":"api-call-service","z":"7030ec85714e521d","g":"1d6ef5cec40e2629","name":"winter lt 400","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_import_day","select.energy_import_month"],"data":"{\"option\":\"winter_lt_400\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":840,"y":220,"wires":[[]]},{"id":"03d08d57cb753206","type":"api-call-service","z":"7030ec85714e521d","g":"1f1ac14da602b108","name":"winter","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_export_day","select.energy_export_month"],"data":"{\"option\":\"winter\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":530,"y":400,"wires":[[]]},{"id":"0af23044806f9006","type":"api-call-service","z":"7030ec85714e521d","g":"1f1ac14da602b108","name":"summer","server":"ffea7422.3895a8","version":5,"debugenabled":false,"domain":"select","service":"select_option","areaId":[],"deviceId":[],"entityId":["select.energy_export_day","select.energy_export_month"],"data":"{\"option\":\"summer\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":540,"y":360,"wires":[[]]},{"id":"ffea7422.3895a8","type":"server","name":"Home Assistant","version":5,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30,"areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]

Closing

I haven't yet got my water or gas meters integrated into this system. I plan on doing that eventually, but for now I'm happy with the electric results. The data won't be the most accurate for the remainder of this billing period, but it should reflect my next billing period fairly accurately, and I plan to check it.

Updates

2024-03-23

Since the article was published, my system picked up some messages from my gas meter, and now has a gas meter chart. I'd misconfigured the decimal place in rtlamr, and so had to dump my old readings and set it up again. I'll eventually set out to figure out a way to track gas tariffs and get estimates in there for now.

I spent some time tonight setting up the digital-alchemy add-on for HomeAssistant. It lets you write automations using typescript, with some nice tooling for VSCode or other typescript language server compatible editors. I ported my tariff switching logic out of NodeRED and into typescript, and find the end result much simpler to reason about. I'll probably wind up doing the same style script for my gas tariff tracking.

Many years ago, I purchased a PetNet smart cat feeder. This one was well reviewed, and the app worked well enough not to be annoying. It let me set schedules, and dispense food in rather small increments, compared to its competition. Things worked fairly well for a few years, but in mid 2020, the company behind the product went out of business, and shut down their servers. The feeder would continue to work for a period, but you couldn't configure any settings. Eventually, it stopped working all-together.

Meaning to fix it "sometime in the future," I put it in my garage, and forgot about it for a few years. Recently, I started mucking about with my HomeAssistant configuration, building up a new dashboard and getting my wife to use it, and started feeling the itch about the broken cat food feeder. Thinking it wouldn't be too hard, I went out and purchased an ESP8266, and popped open the feeder.

The feeder was a mix of simple and complex. Simple, in that all that really needs to happen is the motor turns on at scheduled intervals, for a short period of time. Complex in that they built so much more functionality into the device than was really needed. This thing is covered in sensors; it's got two scales, ostensibly for measuring the weight of the hopper and the weight of food dispensed, a pair of infrared sensors to detect when the hopper is empty, sensors that monitor the motor's rotations and position, and probably others that I haven't figured out the purpose of.

Since the sensors are things I didn't really need to worry about, all I had to do here was trigger the motor for a burst of time, at a fixed interval of times. I also wanted to be able to trigger it remotely from my phone or a similar interface. This is pretty easy to do with ESPhome.

Wiring up the device wasn't too complex. The device came from the factory with a decent built-in power supply, running over USB 2 on a Micro-B port. The motor, sensors, and everything else plugged into the main board via little JST connectors. The ESP8266 devboard I used can be powered by either a 3.3 or a 5 VDC connection. To control the motor, I used a relay, wiring it directly to the incoming power supply and motor. Since the onboard power supply provides 5v, and the motor is 5v rated, I powered the devboard using 5v, using the 3.3v output to power the relay board, and triggering the board via a GPIO pin.

ESPHome makes the software side even easier. Getting the board flashed and talking to my HomeAssistant system was so trivial I was astonished. I just plugged the devboard into my computer, went to the ESPHome website, and, via the powers of WebSerial, flashed it with the ESPhome base firmware and got it set up on my wifi. From there, HomeAssistant "saw" the device and gave me the option to adopt it. This whole process took about 5 minutes. That's faster than the setup and adoption of some purpose-made "smart home" systems!

Changing the device to actually do what I needed wasn't much more complicated. Using ESPHome primitives, I set up a GPIO output pin, and a "Button entity" to trigger this pin for a second and a half. Finally, I set up a timer entity that triggers the button at a few fixed times throughout the day.

Once I'd put the whole device back together, I powered it up and added some catfood to the hopper. Triggering the button from HomeAssistant, I watched happily as catfood came pouring out of the dispenser. Pressing the button a second time resulted in no catfood, and a buzzing sound from the motor. After some trial and error, I eventually swapped to a smaller size of catfood pellet, and then ultimately to a different USB power supply. The original one that came with the feeder said 5v 1A on its nameplate, but after testing with a meter, I was only getting 3.3v and barely 100mA. Now the dispenser is triggering happily and consistently, and our cat no longer pesters us throughout the day for more food.

If you are interested, you can see the configuration I wrote for the cat feeder here

Updates

HackerNews discussion prompted me to make the following clarifications: I used an ESP8266, not an ESP32.

Additionally, this is not the cat's primary food source; he gets two bowls of wet food a day, so the dry food is mostly for supplemental feeding and dental health.

Recently I moved this site off of the older Nuxt/Nuxt-content based static site generator to Tableau, an Elixir based SSG. Between not really ever being fully comfortable with how "magical" Nuxt content was, and some recent changes broke some aspects I was using. This migration has been fairly enjoyable, and I was able to contribute some of my changes back to Tableau.

Previously, on pdx.su…

Previously, I was using Nuxt and Nuxt-Content. This was a fairly enjoyable stack, for the most part. I could write posts in Markdown, add my own custom components to be used within Markdown documents, and publishing was relatively straightforwards.

But it was often a black box. I could tweak some things about it, most of its layout and even the way some components in its output rendered, but things such as some head tags, and particular aspects of inline code, were always off-limits. Nuxt was very nice for quickly getting a site up and running, but I never really felt that I was in complete control over how it built itself. Its output was nice and fast, and the site rendered rather quickly, but given I was using no server-side rendering, and only using pre-generated HTML output, the server-side JS features ultimately proved more of a headache than a utility.

Getting the opengraph header meta tags to render properly for each post was a bit more difficult than it should have been, and I ultimately didn't get all the tags I should have.

But the biggest source of pain for me was code blocks. This is, nominally, a technical blog. I write about code a lot. And so having code blocks that look good is vital. Nuxt-Content uses a neat system where it processes Markdown code blocks, and lets you customize the output renderer by defining your own vue components. This is how I added things like a copy button, and some other nice decorations.

Unfortunately, those custom components don't extend to the actual syntax highlighting. When I initially made this site at the beginning of 2023, it used the Shiki syntax highlighting library. This library is pretty good, supports a decent amount of languages, and has a nice "css-variables" theme that lets you customize the output via CSS variables. I used this feature to enable use of the Base16 tomorrow syntax theme, and have light and dark mode versions.

But recently they moved over to a different fork of that highlighter, called Shikiji. This fork has some real improvements, such as actual light/dark support. However, it doesn't support the css-variables theme. It also doesn't ship with the tomorrow theme variants, and attempting to manually add them caused issues. I opened an issue against Nuxt-content regarding the problem, so hopefully it will be fixed in the future.

Contributing to Tableau

I've been loosely aware of an Elixir based static site generator, called Tableau. Developed by Mitch Hanberg and part of the elixir-tools project, it seemed like a pretty good solution. I know Elixir, it's the primary language I code in these days. Mitch is a pretty great guy as well, and his code is always clear and easy to read. So it seemed like a natural fit for what I was trying to do.

At the time I was looking, Tableau was somewhat "incomplete," from my perspective. It was capable of doing all the nominal things it should do, and was being actively dogfooded, as it was what powered the elixir-tools site. But it was lacking a number of the niceities that other, older SSGs already have. It required a fair amount of "stuff" to be in the frontmatter of every post, that could be abstracted away to either defaults or somewhat "smart" generation at compile time.

The best way to get an open-source project to meet your needs is to directly contribute the code that enables it to do so yourself. And so that is what I did. I forked the repo, and added a variety of the features I wanted, over a few weeks, and opened pull requests for them. Mitch gave feedback where needed, tweaked the code to match the style he likes for Tablea, and ultimately merged in my changes. Not my first open-source contribution, not by a long shot, but it always feels good when that happens.

With the improvements to the post (and pages) systems in place, I contributed a sitemap feature, and got down to business, porting my site over.

Porting my site

With this current incarnation of my site (there are some older ones lost to time, and I don't care to revisit them), I generally kept things pretty simple. Most of my content would be in posts, and posts would largely exist "alone." A reader who was interested could scan all that I'd written, but I wouldn't use annoying antipatterns to try and coerce them into it.

Most of the focus was on the prose, how easy it was to read. There were some clever amenities I wanted to keep; the table of contents on desktop, "smart" timestamps that show the user's local formatting and zone, and some enhanced markdown features MDC offered in Nuxt, but none of those were really blockers that would prevent me from doing what I needed.

Starting from the basics, setting up a new project to use Tableau was rather easy. Set up the Elixir project via mix new, add the dependencies, and write the configuration file. Writing the root template, and then descendent templates of post and page, was rather trivial. If you've ever written code for Rails or Phoenix, you would feel right at home. Compared to some "magic" other SSGs I've tried, it was refreshing to just have to do it yourself.

Temple templates

Tableau lets you use whatever templating language you want; I could have used Slime, an Elixir Slimlang port, that I've used in the past, but Mitch also has his own templating language, called Temple, which I wanted to experiment with. Temple characterizes itself by being nothing but "real" Elixir code. You write temple templates like this:

temple do
  div id: "some_id", class: "foo" do
    "This is temple code"
  end
  if @some_assigns do
    img src: "image.jpg", alt: "an image"
  end
end

That's it. There are no oddities around switching to or from control logic, no funky characters to control inline whitespace, and ultimately no new syntax needed.

And temple also supports components, similar to Phoenix LiveView, Surface, or your choice of JS framework (react, vue, lit, etc.). You just define a component as a function, and call it via the c macro, from within a temple do…end block.

This let me quickly throw together all the "static" parts of the site that I needed.

Table of Contents

For the ToC, I found the best way to generate it was to parse the generated markdown, extract the headings, and store them in an attribute. Then on each Post render, I could just pull the value, loop over it, and render it as needed.

I created a Tableau extension to handle this, that runs after the Markdown documents have been parsed and compiled, and outputs the appropriate data of the headers. Parsing was done using Floki, and was quite easy to do.

The final bit of enhancements around the ToC were pretty easy to port over from my older Vue based site, in Javascript. I register a simple IntersectionObserver, which keeps track of which headers are visible within the viewport, and updates elements within the ToC accordingly.

A last bit of CSS was used to make the targeted header flash a few times when navigated to. Previously I used JS to hook into the Vue router and detect these changes, but doing it in pure CSS feels elegant.

Timestamps and Notes

For the timestamps and note component, I didn't want to toss Vue into the page again, but still wanted them to be somewhat interactive. The timestamp component can only be done with some client-side Javascript, as there is no server, and all pages are static. The markdown notes could have been parsed and rendered on the server side, by a Markdown pre-processor, but it was still a fun exercise to do them as client-side components.

Ultimately, I went with lit components. Lit components provide a nice little library atop HTML custom elements, which have pretty excellent support across the board. Registering a component is simple enough, and using it is even simpler. The API lit provides is nicely similar to both react and vue, so anyone who is familiar with them should be able to pick it up rather quickly. And its heavy use of typescript decorators make it rather simple to write.

For the in-text notes, I didn't bother with adding a noscript equivalent. If you don't have JS enabled, sorry, they just won't render. You won't see anything broken, but they'll just be absent.

For timestamps, this was not an acceptable compromise. So I wrapped the timestamp calls in a Temple component, rendered server side, that outputs the custom element and a noscript wrapped <time> tag. If you have JS turned off, you'll see a date in US formatting. If you have JS on, the timestamps will be rendered according to what your browser says is appropriate for your current location and locale.

Assets

Tableau uses an approach similar to how Phoenix handles custom asset processors: it just lets you call out to them during the development phase, and that's it. It isn't aware of anything special regarding CSS, JS, TypeScript, and doesn't need to be.

Because of this, my asset processing pipeline is fairly boring and regular TypeScript and SCSS, built using esbuild. Tableau starts up the esbuild watcher, which compiles files on changes, emits them to the output directory, which then triggers Tableau's file watchers to send a refresh event to the browser.

For production, I just call the esbuild build script as part of my GitHub Action.

Keeping assets and code separate may feel a bit antiquated, but it's ultimately rather simple, and with modern CSS features such as scoping, nesting, and customizable cascade layers, all the pain points from the past are pretty much absent.

It's all open source

The previous "build" of this site was in a private GitHub repository, mostly because I wasn't the happiest with the quality of the code. But since there are comparably fewer Tableau sites to Nuxt sites, I felt that having this one be open-source would be a nice way for people interested in Tableau to see how some things work.

So you can check out my site, or view the source of a post, at https://github.com/paradox460/pdx.su.

Recently, I wanted to use CalVer with Release Drafter. This is for a project where the SemVer approach would result in a perpetually increasing patch version number, and a practically frozen major and minor version. Unfortunately, Release Drafter has no inbuilt support for CalVer, so we've gotta calculate version numbers ourselves.

function parseVersion(version) {
  if (!version) return;

  const regexp = /^v?(?<calVer>\d+\.\d+)\.(?<incremental>\d+)/i
  const matches = version.match(regexp);
  if (!matches) return;

  const { calVer, incremental } = matches.groups;

  return {
    calVer,
    incremental: parseInt(incremental)
  }
};

function currentCalVer() {
  const date = new Date();
  return `${date.getUTCFullYear()}.${date.getUTCMonth() + 1}`
};

module.exports = async (github, context) => {
  const calVerDate = currentCalVer();

  const latestRelease = (await github.rest.repos.getLatestRelease({ owner: context.repo.owner, repo: context.repo.repo })).data;
  const parsedVersion = parseVersion(latestRelease.tag_name);

  if (!parsedVersion) return `${calVerDate}.0`;

  if (parsedVersion.calVer === calVerDate) return `${calVerDate}.${parsedVersion.incremental + 1}`;
};

This script is meant to be used with Github's actions/script workflow, which allows you to use JavaScript inside an Actions workflow. You'd call it something like this:

name: Release Drafter

jobs:
  update_release_draft:
    permissions:
      contents: write
      pull-requests: read
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: "Generate CalVer"
        uses: actions/github-script@v6
        id: calver
        with:
          result-encoding: string
          script: |
            const genCalVer = require('./.github/workflow-scripts/calver.js');
            const version = await genCalVer(github, context);
            return version;
      - uses: release-drafter/release-drafter@v5
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_TOKEN }}
        with:
          prerelease: ${{ github.event_name != 'pull_request' }}
          publish: false
          version: "${{ steps.calver.outputs.result }}"

Note that the version input is provided to the release-drafter script. This overrides release drafter's internal version calculations, setting it to the output result of our script.

CSS has been undergoing a quiet renaissance lately. Lots of big features which previously required an external tool to use, are now native parts of the language, and its growing more and more all the time. If you haven't used CSS in a long time, for whatever reason, now is the time to take a look again.

Brief history, and how CSS became "not-fun"

Back in the late 90s, we styled our websites using inline attributes. bgcolor, font, and friends ruled the roost. And this was okay. If you wanted a more complex style, you used tables and sliced up images. Or you just used Flash, but thats a whole different beast. When CSS came out, it was pretty cool, but far too simple to really do what we needed. You could set colors and sizes and some positional properties, but that was kind of the limit. You still had to use tables to do complex layouts. You still had to do image slicing for round corners and shadows and other silly things, but it was "better." CSS 2 rolled around and gave us some primitives for actually positioning our stuff. We could move things around the page, float them off to the side, and adjust a lot of how they looked. CSS was finally viable for doing the whole page. We ditched tables. And we quickly ran into the limits of what we could do. Rounded corners still needed image slicing. Complex positioning, i.e. grids (so complex!) required JavaScript. And for many people, this is where CSS stagnated. CSS didn't really stagnate. It slowly marched forwards, growing things like shadows, border-radius, and then later flexbox and grids. But, despite this slow progress, it had gathered a reputation of being brittle and difficult to work with. Tricks like clearfixes arose; bits of knowledge you just had to find and use, and then apply like cheap plaster in a house you intend to flip. Reset styles arrived to make things somewhat consistent. External tools like Sass and PostCSS streamlined the process of authoring CSS, making it so you didn't have to remember all the browser prefixes or how to write all the border radiuses. And eventually, some developers just started throwing away parts of CSS, favoring simpler approaches, while not being sure exactly of what they were giving up, but certain they didn't need it.

The quiet renaissance

As time passed, CSS slowly grew new features. Most of them were novel, but constrained in their utility. Selectors like :is and :where, while useful, only slightly moved the needle in terms of just how much code you had to write. Flexbox and CSS Grids arrived, made things easier for those who used them, but received less fanfare than they deserved; many sites and layout frameworks still use older methods, the ones the developers learned through fire trials.

Variables, or custom properties, made a big splash, particularly because they can do things that couldn't be done otherwise. Preprocessor variables are not context aware, they can't be reset using things like media styles or user preferences, and so they're basically just super-fancy find-and-replace placeholders.

But recently, the pace of new features has accelerated so much, that I envy those who are just learning CSS today, and don't have to learn all the baggage and old ways of doing things. They'll just see all the new tools at their workbenches, and get down to work.

CSS Nesting is the biggest "preprocessor" feature, with every preprocessor worth its salt implementing it, and now it's done in native CSS. There's a bit of a caveat around syntax as some last minute changes landed, but it's not quite like the prefixes of yesteryear; you just have to use an & in more selectors than is ideal, and as browsers mature, this limitation will likely disappear. But you can now easily scope styles to a specific element or selector, and write support for the various psuedos that are common, in a neat, clean, non-repeating way. And you can finally write your media queries inline, where they belong, with properties inside them, rather than the previous redundancy we saw before. Not really exciting if you've already been using a preprocessor, but now you don't need that preprocessor for this.

color-mix takes another bite out of the preprocessor pie. Instead of having to use sass functions like lighten or darken or transparentize, you can now just write them with real css. And since they're done in real css, they're color space aware, and can make use of custom properties too. Wanna make a highlight out of a primary color, which is picked and set externally? Piece of cake

.selector {
  background: color-mix(in srgb, var(--primary-color), white 50%)
}

This mixes your primary color with 50% white, in the sRGB space. You can use other, newer, better color spaces too, such as OKLCH, which is my favorite.

Containment and Style queries landed, which let you make whole sections of your stylesheet that style based on their size, not the window's size. For component based designs, this is incredible. Now you can make components that render one way when they are "small" and another way when they're bigger.

And it's not all big new features either. Lots of things got quiet little improvements. Transform properties were broken out into separate properties, so now you can do translate: 50% instead of transform: translate(50%). display was revisited, allowing now to mix and match display types, so you can more adequately specify both how something is positioned relative to its content (inline vs block) and how content inside it is positioned (flex, grid, table, etc). display also became animatable, so now you no longer have to figure out how to move an element between hidden, shown but not visible, and shown but visible. New trigonometric functions landed, allowing for accurate angular math.

All of these features are available today, as of this reading, with every major browser supporting them in the current version

The renaissance isn't over

And more features are on the horizon, or already landing.

Colors are gaining even more superpowers, via the Relative Colors feature of the CSS Colors level 5 spec. When it lands, you don't even have to use the "new" color-mix feature to do common things, like hue-shift or lighten a color. You can now easily manipulate any value of a color, in any space you like, same as you would any other value.

Want to make a transparent version of a color? Piece of cake:

:root {
  --primary: blue;
  --transparent-blue: hsl(from var(--primary) h s l / 50%);
}

How about making a lighter version of a color?

:root {
  --light-blue: oklch(from blue, calc(l + 25) c h);
}

Instead of having to use cumbersome functions, you can just use calc and the same color functions as you would to define a single color.

Native CSS scoping is landing too. Using it you can now target styles to a specific scoping root, or specifically carve out holes in your styles, without having to get too funky with things like :not. This is a rather complex feature, so I'll link to the Chrome blogpost about it

Closing

I've been using nesting and the color-mix features pretty heavily since I found out about them earlier this year, and they've been great. There's a bit of a learning curve, as with any new tool, but its rather short and gentle. I still keep Sass around, as well as PostCSS, for things that CSS just wont ever be able to do (nor should it), but they're fading into the background, rather than being at the forefront of my mind when writing styles. And for small, simple projects, I don't use them at all. Just pure CSS. I haven't found it lacking.

Sass, Pug, Haml, Slim, Stylus, and their friends all aim to make writing various bits of your frontend easier. And they mostly deliver on this primary promise. But they are all victims to the vagaries of open software development, and seem to have mostly fallen by the wayside. I loved using these through my career, so its with a bit of sadness that I realized I don't want to use them for new projects.

What they set out to do (and usually achieve)

All these indented syntaxes, haml, slim, and pug mainly, aim to reduce the amount of boilerplate needed when writing HTML, and Sass and Stylus aim to do this with CSS. They are typically poised as an alternative to a built-in or already popular template language in their respective web development ecosystems, most of which typically are just HTML or CSS but with a special syntax for escaping to "real" code. With CSS the improvement is negligible, but with HTML its significant, as HTML likes to make you repeat yourself a lot.

Take the following simple HTML document:

<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Document</title>
</head>

<body>
  <header>
    <h1>Header</h1>
  </header>
  <nav>
    <ul>
      <li>Link</li>
      <li>Link</li>
      <li>Link</li>
      <li>Link</li>
      <li>Link</li>
    </ul>
  </nav>
  <article>
    <h2>Article</h2>
    <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quae blanditiis voluptas amet eligendi nemo libero corrupti accusamus minima laboriosam iure, quia modi nulla. Accusamus, excepturi! Voluptate dignissimos repudiandae minima facere.</p>
  </article>
</body>

</html>

Kind of long? You can write it in pug a lot faster:

html(lang="en")
  head
    meta(charset="UTF-8")
    meta(name="viewport" content="width=device-width, initial-scale=1.0")
    title Document
  body
    header
      h1 Header
    nav
      ul
        li Link
        li Link
        li Link
        li Link
        li Link
    article
      h2 Article
      p Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quae blanditiis voluptas amet eligendi nemo libero corrupti accusamus minima laboriosam iure, quia modi nulla. Accusamus, excepturi! Voluptate dignissimos repudiandae minima facere.

18% fewer characters, and it's generally pretty easy to see how everything works. Pug and other indent based languages have small differences, but will largely look the same.

With Sass (specifically, the .sass indented system) and Stylus, the difference is smaller, and mostly comes down to a bit of developer ergonomics. You don't need to remember to write all those messy {}; characters (stylus lets you even omit the :). They both also provide nesting, mixins, functions, and other utilities, but they're kind of irrelevant to the normal, day to day ergonomics.

Where they start to fall short

Technical issues

Indent based syntaxes look very good on the surface, and so many developers dive right into them. Only after using them for a while do you begin to encounter the little flaws that make them less fun to use. And generally, these flaws are fixable, or at least work-aroundable.

Take a recent "flaw" I encountered when writing a style using Stylus. I was trying to make use of the newer syntax for writing hsl colors with an alpha channel. Problem is, Stylus internally has a function called hsl, that exists for legacy reasons (converting HSL to hex). It has a very specific signature expected, one that matches the "old style" HSL syntax.

.foo
  background-color hsl(210 100% 52% / 25%)

This started throwing compiler errors, and my stylesheet stopped working. There are a couple fixes. Obviously, I could rewrite it in the old style and use hsla instead. But stylus also provides an escape hatch to pure CSS, which is what I wound up using instead:

.foo
  background-color @css { hsl(210 100% 52% / 25%) }

Not really great, but I was able to get on with the project.

Another such example is that the indented syntax for Sass, .sass, doesn't support multi-line constructs. At the time sass was first created, this wasn't really much of a problem, but since then it has become one. grid-template-areas rely on newlines to indicate different rows of the grid, and a built-in sass feature, keyword lists, is hobbled by the inability to have multiple lines. There was an issue opened about this in 2011, and it is still open, with no resolution. Dart-sass does support the indent based syntax yes, but it seems relatively limited; bugfixes only appear if they first surface in the .scss format, and also happen to manifest themselves in the indent based format. Stylus, for what its worth, supports multi-line constructs via a \ at the end of lines, indicating a continuation.

Stylus has drifted in and out of maintenance status for the last few years, with the future of the project being uncertain. It works fine now, but as the web continues to move forwards, I worry that more places will need to use the @css escape hatch, and at some point you're writing more CSS than Stylus, and might as well just switch over

The small learning curve

No matter how close these things are to their "native" counterparts, they are not the same, and so there is always a technical learning curve. In many organizations, this curve is too much, however gentle, and so they aren't used at all. I think this limits their exposure, and so they're always sort of a side project, with minimal support. .scss is far more common to find "out there" than .sass, despite the latter existing before the former. Pug is extremely rare in the JS ecosystem, most people either use something like mustache or just JSX. In Elixir land, EEx and variants are more common than Slime, and in ruby the same goes for ERB vs slim/haml.

And that's assuming you even need them. CSS has nesting now. CSS has variables. CSS has all sorts of combinator selectors that can let you write code that's even more terse than you could with preprocessors. JavaScript is gaining newer and better ways to template out HTML, and at the end of the day you can always fall back on template literals.

The inherent flaws in indent-based syntaxes

Indent based syntaxes are very easy to write. But finding out where blocks start and end can be frustrating. Most editors have systems that help you with this, be it indentation indicators, visible whitespace characters, etc. But you still have to rely on them to know where you are. With a delineated block syntax, you just look for the } or the closing tag, and you know that's where it ends.

I don't want to use them on future projects, and that makes me sad

I love the indent based syntaxes. Throughout my career, I've reached for them time and time again. They've saved countless keystrokes, and when written well, look excellent. But I've moved away from them.

The little annoyances start to mount. Having to use escape hatches all over the place just gets tedious. And trying to figure out if it's a problem with the preprocessor or the output code is often an annoying bit of yak shaving that I could do without.

And god help you maintain consistency if you're working on a team. Stylus is amazing for small passion projects. But it does hamper maintainability. Write a codebase as a solo engineer long enough and you'll start to see inconsistencies in how properties are applied. Some lines will have semicolons separating the property from the values, others won't. Stylus doesn't care, and eventually you won't care either. And if you toss other engineers into the mix, then it becomes even more messy.

Getting templates working isn't hard. Typically, its just install a dependency, add a few lines to a config, and start using it. In Vue.js you have to tag your template with a lang tag: <template lang="pug">{lang=html}, but that isn't too bad. But out of the box you don't even have to do that. You can just start writing HTML. And since you're using components, all the messiness of HTML is somewhat soothed and combed down.

I worked on upgrading some stylings on this blog over the past weekend. I moved from hardcoded base font size to respecting the browsers font size, and using those sizes for the few breakpoints I have. Initially I did all my changes in the stylus files that I created with this blog. But I had to use a few escape hatches here and there, mostly around color and the "new" color-mix{lang=css} function in CSS, and it left a sour taste in my mouth. I started moving the files over to .sass, to get away from the need for escape hatches, thinking "Since Sass is more actively maintained, it will have kept up with these standards". No more issues with color-mix{lang=css}, but now I ran smack-dab into the multi-line problem. From there, I just swallowed my opposition, and moved to .scss files and syntax across the whole codebase. Vim-surround, textobj-indent, and a few other tricks made this migration rather easy, and now the codebase is fairly clean.

I have no opposition to indent based syntaxes, and would love to continue using them. But the cost-value proposition is currently out of alignment, and unless things change dramatically, will likely stay out of alignment.

A common pattern in CSS, particularly when using frameworks, is to use a bunch of classes to affect how something looks. Things like btn btn-primary btn-blue are far too common. There is a better way, with support built into CSS too

What's the purpose

Let's dissect the classes being used on our aforementioned button. We've got btn, which is pretty straight forwards, it indicates that it's a button. Then we've got btn-primary, which probably indicates that the button is somehow an important or primary button. Then we've got btn-blue, which means our button has a blue-ish color.

Only one of those is actually meaningful. The rest are presentational aspects. And there's no way to tell which ones can mix and which can't. We can intuit that btn-blue and btn-red don't mix, but maybe they do, and we'd get a purple button.

What we're trying to do is affect how the button is rendered, by changing a few other properties of said button. Lucky for us, HTML already gives us the ability to set attributes, without having to over-rely on the class attribute

Use Data attributes

If we define our button like this

<div class="btn" data-primary data-color="blue">Button</div>

We can then style it like this:

.btn {
  border: 1px solid;
  padding: 5px;

  &[data-color="blue"] {
    background: blue;
    color: white;
  }
  &[data-color="red"] {
    background: red;
    color: white;
  }
  &[data-primary] {
    font-weight: bold;
  }
}

Now there's a very clear separation of concerns. The class indicates what this thing is supposed to be, and the data attributes affect parts of how it looks. The primary appearance is styled by classes still, but secondary aspects are given to us from data attributes.

Read more

• Attribute Selectors
• Data attributes

Testing in Elixir is pretty great. ExUnit, combined with the functional nature of Elixir, makes it very easy to test almost everything in your codebase. However, it is very easy for boilerplate to creep into your tests. Common setup patterns, similar assertions, and more can quickly make your test suite feel cumbersome. But ExUnit files are just Elixir files. So you can write bits of code that will help you out tremendously.

Common Setup

You're testing something in your codebase, and you have to set up a user, maybe an account or organization, and maybe even some content to "work" with. You've got some fixtures or factories ready to go, so you just write the setup code inline in the test. But now you need to write another test. So you extract the setup to a setup/1 block. Pretty good. Now all the other tests in that file or describe group will have those steps run before they do, and any results added to the context.

But now you have another test file, and it's got similar, but slightly different, setup steps. You copy the setup block, change it slightly, and proceed with things. Repeat that over the next several other test files, and now you're in an unfortunate situation. You've got a bunch of similar, but not identical setup steps. Some tests need users of different permission levels, some need different content, others need their own special things.

Extracting Universal steps to a macro

A common approach is to put the most universal setup steps into a custom __using__ macro, and then using it via use. If you're using Phoenix, you already have a macro generated for you, in your test/support/conn_case.ex file, and similar ones for database access and channels. You can just add a setup block to these macros, and it will effectively be included in every test that needs it.

This is great for setup steps that need to occur universally. But that's a surprisingly small number of steps. You might assume that your user setup and auth steps need to be universal, but they don't. How are you going to test if unauthenticated users are prohibited from using parts of your application?

In the past I've seen people split out a separate case, or add options to the __using__ macro, that are passed in from runtime. Things like:

defmodule MyAppWeb.SomeBigTest do
  use MyAppWeb.ConnCase, noauth: true

This works, but it quickly becomes opaque. You have to document all the various opts you can pass to the __using__ macro, and discoverability is minimal. And as your test suite grows, the number of opts you have to support, plus the ways they can be combinant, can get overwhelming.

Setup functions

An alternative that supplements universal setup steps is setup functions. While setup/1 nominally accepts a block, it also accepts a single arity function, which is called with the current context. This lets you define a collection of discrete setup functions, that can be mixed together to handle your test case setup. If you edit your aforementioned using macro to import a module containing these setup functions, you can use them anywhere the macro is used, however you like.

Take for example the following helper module:

defmodule MyApp.Support.Helpers do
  def insert_user(context) do
    # Some code that sets up a user and adds it to the context
  end

  def authorize(context) do
    # Some code that takes a user off the context and "authenticates" them
  end

  def insert_content(context) do
    # Generate some content
  end

  # More functions...
end

As you can see, those functions are pretty small. They implement single, simple concerns, and are pretty straightforwards, from the name alone.

If you added them to your __using__ macros, you can then simply call setup :functionname anywhere you'd put a setup block, and it will be called, and passed in the current context. You can have each setup function take and return things to the context, and build pretty powerful setup pipelines.

defmodule MyAppWeb.SomeBigTest do
  use MyAppWeb.ConnCase

  setup :insert_user
  setup :authorize

  test "test_something", %{current_user: user} do
    # ...
  end
end

Very clean, and for test suites that don't need to have a user or an authed user, you can simply omit them.

Configuration

You've now got a pretty clean setup system. You can pick and choose whole chunks of setup, and everything more or less works. But you're still in a position where you have to write a bunch of different functions for various permutations of how you'd set up your test cases. Maybe you need both a regular user and an admin. Whatever the case, having to have an insert_user/1 and an insert_admin/1 that share much of the same code. Maybe you extracted them to a private function inside your Helpers module. That works, but there's a better way.

You might also want to have slightly different configuration for each test in a suite. They all call the same functions, but you might want each to be subtly different.

There's a nice solution for this, built into ExUnit. The @tag.

Out of the box, without any configuration, you can change any value in the context of your tests using the @tag attribute (and the @describetag for your describe blocks). Since setup is run once per test, you can simply call @tag someattr: somevalue before each test to override the values generated by your setup functions. Have a file-wide setup for inserting and authorizing a user, but want to test what happens when current_user is nil? Trivial, @tag current_user: nil.

But that's just overriding values from the setup with your own values, on a per test basis. Better than nothing, but you can do better still. Remember that the custom functions we wrote for setup receive the context as their parameter. You can use this to accept configuration values from the context and make your setup functions do different things.

Take this example of changing insert_user/1 to handle things like the user being an administrator:

def insert_user(%{admin: true}) do
  # some code that makes a user and sets them up as an administrator
end
def insert_user(context) do
  # code that just creates a normal user
end

Now, without changing the setup calls at the top of the test suite (or describe block), we can make tags have an admin user on the fly:

@tag admin: true
test "an admin can delete a post" do
  # something that tests this
end

test "a regular user cannot delete a post" do
  # similar code, but it would refute that the post was deleted
end

That's it! Very simple, very flexible, very useful.

I've started using the above patterns all over my testing, and now can't imagine working without them.

Custom Assertions

Another common smell I'll see all over test suites is a lot of boilerplate around assertions. You'll see a lot around things like testing HTML matches certain values, or certain elements are present, or in an evented system that an event was received. People coming from other testing frameworks seem to think there's something magic about the native assert, and may ask if there are other libraries, such as how Rspec has the matchers libraries in Ruby.

Assert, and its sibling refute, are just plain Elixir code, same as most everything else in Elixir and ExUnit. You can write your own asserting functions and macros trivially easy.

Checking for presence of an HTML element

A common pattern when testing web apps is to see if the generated output contains a particular element. When working on LiveView apps, this is even more common. Generally, a lot of test suites will start off by just doing simple string matching, such as html =~ "<div class=\"bar\"". This proves to be quite fragile and prone to dumb breakage. So eventually most developers will implement something like this, using the Floki library:

assert html |> Floki.parse_fragment!() |> Floki.find(".bar")

This does the job, but its rather verbose, and having to write it every time you want to check if an element exists is tedious.

Instead, you can turn it into a simple macro:

defmacro assert_html(html, selector) do
  quote do
   assert unquote(html) |> Floki.parse_fragment!() |> Floki.find(unquote(selector))
  end
end

And use it like this:

assert_html html, ".bar"

Handling Event Boilerplate

In evented Elixir, we have the useful assertion assert_receive. This lets you state that a process should get an event within a timeout, and even specify the message to error with in the case no event is received.

But if your events have a particular pattern they follow, i.e. {:event_fired, %{action: FooEvent}}, and you want to implement custom timeouts other than the default 100ms, or custom error messages, it can get pretty verbose pretty quickly.

assert_receive {:event_fired, %{action: FooEvent}}, 1000, "Did not receive event"

Not terrible, but now you have to do that all over your tests, and if you want to change the failure message or timeout, you have to update all the implementation sites.

Macros can simplify this:

defmacro assert_event(event) do
  quote do
    event_name = unquote(event).name

    assert_receive {:event_fired, e}, 1000, "Did not receive event"

    assert event_name == e.name

    e
  end
end

You can then call this like so

event = assert_event FooEvent

Since the macro returns the matched event object, you can use it further in your test suite:

event = assert_event FooEvent
assert "some event data" == event.data

If you want a further optimization, you can add a version of assert_event that accepts and calls a function, giving said function the event. This lets you treat the function as a lambda, and keep any assertions on said event scoped to only that event. In busy test suites, where you're asserting event after event, this can dramatically improve readability.

defmacro assert_event(event) do
  quote do
    event_name = unquote(event).name

    assert_receive {:event_fired, e}, 1000, "Did not receive event"

    assert event_name == e.name

    e
  end
end

defmacro assert_event(event, func) do
  quote do
    event = assert_event(unquote(event))

    unquote(func).(event)
  end
end

And you can use it like so:

assert_event(FooEvent, fn e ->
  assert "some event data == e.data
end)

Very clean!

Generating test suites from a matrix

Often you'll find cases where you need to test that a variety of cases are valid, and a variety of cases are invalid. These test suites might be largely identical, apart from the variable factor. Since tests are just elixir, you can do this:

describe "Post Removal" do
  setup :create_post

  for role <- [:moderator, :admin], own <- [true, false] do
    @tag own_post: own
    test "#{role} can remove #{if own, do: "their own", else: "someone else's"} post" do
      # some code that removes the post and asserts its removal
    end
  end

  @tag own_post: true
  test "users can remove their own post" do
    # some code that removes the post and asserts its removal
  end

  @tag own_post: false
  test "users can't remove other people's posts" do
    # some code that attempts to remove the post and refutes if the removal was successful
  end
end

While this is a contrived example, you can see how we were able to test 4 test cases for the moderator and admin roles with a single test, and then test the more specific user test cases separately.

Remember, all that ExUnit provides is some useful tools around testing. Under the hood, its just Elixir.

Updates

• Post was updated to add a note about why macros were used instead of plain functions

Reddit has recently disabled/removed access to the compact interface, which was useful on mobile and low power devices. As my first professional project ever, I'd like to reflect on it, 13 years the wiser.

Origins

Back in 2009, I had a Blackberry Storm. It wasn't a great device, but it was neat for the time. The screen was "big," it looked pretty, and it could browse the web, albeit poorly. The built-in browser was mostly optimized for WAP style sites, a few steps below the then amazing browsers on the iPhone and Android devices. There was also the Opera browser, which was much better at rendering pages, but used server-side rendering, meaning no javascript support at all. Because of these limitations, browsing reddit on a Blackberry, even a big touchscreen one like the Storm, was almost exclusively a read-only experience. You could choose between the then-WAP styled, read-only reddit mobile site, which looked a lot like HackerNews does now, amusingly, or the "full" desktop experience, which kind of worked without JS, but not well. I was mostly okay with this, as I could blame it on the lack of decent browsers on the Blackberry. Christmas 2009, I received a Motorola Droid, known as a Milestone outside the US. This was a substantial upgrade over the Blackberry. It had a physical keyboard, a better touch screen, ran Android, and, importantly for this post, a Webkit based browser. Suddenly the read-only reddit seemed like a much bigger hindrance than it did before. You could post and browse the full desktop site, but it was suboptimal (seriously, try browsing https://old.reddit.com on your phone). I wasn't impressed by what I saw in terms of mobile interfaces; I can't remember if Reddit is Fun (now RiF for Reddit) was one of the apps I tried, but if it was, 18-year-old me wasn't impressed. And so I set out to make my own mobile interface.

The early steps of the mobile interface

This was right smack-dab in the sekuomorphic interface era, and there was plenty of iPhone style interfaces for "studying" across the internet, mostly in screenshots. Reddit had its own in-house iPhone app, called iReddit, but nothing for Android. Being a web developer primarily, I started out making a simple interface on top of reddit. My javascript skills at the time were quite lacking, so I was handling most of the API stuff with glued together bits of Perl and PHP, rendering it on the server. This wasn't terribly unusual for the time, the era of jQuery, Ext.JS, and Google Web Toolkit. The interface was lightly styled, mostly using webkit based CSS features, lots of background gradients, and so forth, but it looked a lot like the final compact interface looked.

Becoming an official reddit interface

January 2010, I was in the #reddit IRC channel on freenode, and some reddit admins were active one night. I showed keysersosa some screenshots of the reddit interface, and he seemed fairly interested in it. Extended conversations turned into an invitation to make it an official reddit project. I was given some access to a staging instance of reddit, ssh access to said instance, and some instructions on how reddit's internals worked. Chris, aka keysersosa, and the other reddit admins, were tremendously helpful, and put up with a lot of my naïve and immature questions and approaches. Over the next several months, the mobile interface took shape. Towards June, it had become fairly feature-complete, with voting, submitting, commenting, sign-ups, and most other core reddit features. Some interfaces never really got built explicitly for mobile, but they still worked reasonably well, since the mobile layout was fairly flexible to render content. Several people helped with the QA, catching bugs I had no hope to ever figure out.

Come June 9th, we launched it, and got a nice article in TechCrunch about it.

I wrote an old blog post; for whatever reason Archive.org didn't preserve my sites styles, but you can read and see some great images, regardless of CSS. You can see the android 2.0 style pop-overs for post and comment options in that article.

You can view the old reddit comment thread here.

Becoming a contractor/employee for reddit

With the success of reddit, and my need for cash as I went off to University, I asked if I could become gainfully employed by reddit. Surprisingly, they said yes, and I was able to join reddit's team and become an admin. This was incredible for 18-19 year old me, as I was able to get a good jump-start on my career.

Over the next year, I worked on a variety of reddit "things," from a revamp of the advertisers dashboard, several side-banner ads for certain campaigns, a large resolution banner we used for the Rally to Restore Sanity reddit booth, some UX tweaks and changes for the fresh reddit gold, and more.

Improving Reddit Mobile

One thing that some people noticed about that first version is that it could be quite resource intensive to render on first loads, and had weird performance issues. Additionally, certain systems were completely inaccessible from mobile, bouncing you out to the desktop interface.

Most of the slowness was traced to the massive amounts of CSS gradients and effects used. Every post, button, and more, used gradients, shadows, and more. They would often be repainted for each button, which was costly on phones of the time (the Nexus One was considered speedy, with its 1GHz CPU).

This was fixed by rendering a small image of the button styles, and then slicing the image up, 9-patch style, with CSS border images. This gave us buttons that could be any dimensions, with nice looking backgrounds. Other gradients were also flattened into background images, such as those of the header toolbars.

The post options pop-overs were kind of a sticking point as well, with some people complaining about their usability. We replaced them with inline expandos, that spanned the width of the browser, and had much bigger touch targets.

Finally, we added a mail and modmail link to the upper right, replacing the old double-chevron menu, and removed the Home button in the upper left, instead rendering the subreddit logo image and subreddit title.

These features launched, and had a small blog post celebrating them., with a corresponding reddit comment section

Leaving reddit

During 2011, reddit was undergoing a lot of significant changes. They split off from Condé Nast, becoming their own company, and underwent some organizational restructuring. Almost the entire staff, who worked there when I joined, had moved onto other things, and there was an effort to consolidate power and access to the San Francisco office. Finally, my school-work was starting to take more of my time up, and so I was unable to seriously focus on reddit and school at the same time. So it was time to move on. Some features that I had intended to add to the compact version, that never materialized, were mobile moderation, better sharing (this would take nearly a decade for browser-induced share intents to arrive), web-worker/service worker backed notifications, and many more things. Many of them materialized in the "new" reddit interface

The end of compact

For the last couple of months, there have been comments on the /r/compact subreddit about issues rising up with the interface. Commenting, posting, voting and other features would sporadically stop working. A few weeks ago, adding i. as a prefix, or .compact, to reddit URLs, stopped rendering the interface. A workaround was to use https://old.reddit.com/whatever.compact instead, but that was patched last week.

There's been an outpouring of disappointed comments in the compact subreddit, which is touching to see, and several people have started working on userscripts, alternative interfaces, and so forth, to try and resurrect compact.

When people are asked why they like compact, typically they will say something along the lines of how performant it is. Which is not surprising, because it was built targeting a device with a 533MHz CPU and 512Mb of ram. JS was minimal, mostly jQuery style, and the CSS optimizations we made in 2011 still hold up.

Looking back, 13 years later

13 years wiser, with nearly 10 years of full-time software engineering experience, I can't but help look back at where I got started from. I am extremely grateful to the people that helped me along the path, Chris, Mike, David, Eric, Jeremy, and many others. Without them, I surely wouldn't be the software engineer I am today. Without the opportunity to "jump right in," I might not have even become a software engineer. I really appreciated the professionality I was afforded, as someone so green, was awesome. And getting to go to DC for the Rally to Restore Sanity is still one of the high points of my freshman year of college.

The comments I've gotten over the years, and the recent outpouring of support for the interface, a decade later, is incredible. Knowing that people love and continue to use the interface I built, and are going out of their way to resurrect it when it was End-of-Life'd by reddit, is awesome.

So, Thank you old reddit admins, users, and everyone else who helped me along the path.

comments on reddit comments on hackernews

When working on software, I've noticed a tendency in myself and others to overly focus on the MVP. We aim to get the big stuff working as soon as possible, with some vague promise that we'll apply polish later. While this is efficient from a corprorate perspective, I still feel it does a disservice to building quality systems in the long run.

Think about the various systems that you've used. Think about what you like about some of them, and what you don't like about others. Chances are you'll like something rather trivial, when you view it at a distance, and dislike something for, again, a rather trivial reason. These are the little things I'm talking about. Tech debt, warts under the skin, or a messy back office are all problems, but they aren't the problems that often make or break something.

Picking code editors

As a programmer, I'm rather passionate about my text editor. It is the canvas upon which I work, after all. Over the years I've moved between editors a few times. Starting out, I used vim (technically Elvis, as the early 00s didn't see particularly well implemented vim GUIs on Windows). I'd use Notepad++ a few times, but always came back to vim. When I got my first Mac, I used Textmate 2, followed by SublimeText 2, followed by Atom, and now I use VSCode.

Vim is a perfect example of little things mattering. Vim is atrocious when it comes to discoverability. There are no contextual hints, no helpful pop-ups, no type-aheads that guide you. You're tossed in to the deep end, and have to learn piecemeal. But every time you figure something out from the manual, a tutorial, or a coworker, you get a little jolt of endorphins. Vim is excellent at focusing on little things. Textobjects are a perfect example. I'd wager a very large plurality, if not flat majority, of vim users don't use nor care about textobjects. But when you're working, being able to replace the whole word/paragraph/line/region/quote content while your cursor is inside it saves you time, and makes you feel just a little more efficient.

TextMate 2 can be looked at as the beginning of the "editor revolution." Yes, there are very good Mac editors that predate it, such as BBEdit and TextWrangler, but TM2 did so many little things right, that its legacy is felt today. Most modern syntax themes use a format devised by TextMate. The idea of a heavily pluggable editor, while not new (Vim and Emacs have had plugins forever), was popularized by TextMate 2. And TextMate 2 had something else going for it: it was pretty. Working with something that looks nice is a lot better than working with something that's functional, but ugly.

When TextMate 2 got a bit long in the tooth, and some work paradigms shifted (i.e. using a tree view baked into the editor came into vogue), I moved to SublimeText 2. ST2 did a lot of things well, and a lot of those little things were carried over from TextMate 2. But ST2 also introduced its own little things. The keyboard-driven quick command menu is fundamentally no different from Vim's command mode or various Emacs chords, but in practice, the little UI that comes up when you trigger it, the typeahead nature of it, and the implicit discoverability, all tie together to make it a better user interface. Atom and VSCode have also copied this pattern, and it's now emergent in pretty much every technical application out there. Even "creative" tools like Photoshop have implemented similar features.

Atom improved dramatically on how much power third party plugins can have, both over the text, and over various presentational aspects. Things like rich markdown preview, regex railroad graphs, embedded diagrams, and whatnot were the little things atom added that dramatically improved it over sublime text, enough that the performance hit was worth it. VSCode further improved on this by adding performance back while only limiting plugins in certain ways, and by introducing standardized ways to build support for IDE like features, without needing a big heavy IDE.

I also tried Emacs for a bit. I like a lot of the "big things" about emacs: it's modal, it uses lisp for everything, it has features out the wazoo, org mode is hard to beat, etc. I used the Spacemacs distribution, which adds vim like keybindings, Atom/VSCode like command palettes with typeahead discoverability, and some helpful conventions for how everything works. I spent a decent amount of time fiddling with my configs, writing a ton of (very fun) lisp, and just seeing where I ran into limits.

And one of these limits was what made me stop using Emacs. It was an incredibly trivial thing, and most people would say that it's a stupid reason not to use it. But it was a nucleation point for dissatisfaction. That small issue? Smooth scrolling. Emacs is, like vim, generally a terminal oriented program, and as such, scrolling tends to "jump" from line to line. You never have a partial line visible. And that's what I initially noticed. If I was writing a document, there would be a wee bit extra padding at the bottom, where there wasn't enough space to display the next line, but more space than the previous line needed. I tried to ignore it, but it nagged at me. Looking for solutions proved fruitless, the few recommendations some people had didn't work for me. And suddenly, I couldn't not see the issue. Things that were fun before, like editing the configs, took on an air of tedium. I couldn't get it exactly the way I wanted, and so previously where things were just challenges to work around, gleefully, they became chores, annoying issues I needed to fix.

My snowblower

My snow blower isn't tremendously remarkable. It's a recent model Toro two-stage blower, with a 27" scoop. It does its job, well enough, that it's not too annoying to have to blow the driveway after a heavy winter snow. It's reliable, and, provided the snow is dry enough, has never failed me. But it's better than every other kind. It doesn't move snow any better or worse than other brand. Its tires aren't better or worse than any other brand. Its motor isn't easier or harder to start than any other brand. What makes it better is one little thing: the chute controls.

If you've never used a snowblower, you might not know what I'm talking about. A snowblower is a machine, typically gas powered, although there are electric models, that uses a large auger in a scoop to pull snow from in front of it to the middle and back, into a chute. Two stage blowers use a fan (of sorts) to push the snow up and out the chute, with enough force that you can clear a snowbank and blow it into your yard. The chute typically needs to be repositioned a few times during use, so you can aim the jet of snow around obstacles, or as you reorient while clearing a space.

I tend to clear my driveway using a spiral pattern. I aim the chute to the right, and start traveling down my driveway. At the bottom, I turn 180º left, and repeat the process. This creates an ever widening space of cleared snow, pushing the removed snow out and off my driveway. By doing this I never have to reposition the chute, in theory. But in practice, as I get to the outer part of my driveway, I want to aim the chute so as not to spray snow onto my neighbors house. And when I get to a certain stage, it doesn't make sense to continue spiraling, because the other side of the driveway has been completely cleared, so I switch to a more simple back-and-forth method, swapping which side the chute aims at with each reversal. When clearing the area in front of the mailbox, I want to direct the snow around the box, so it doesn't fill up and soak any letters inside when the snow melts. And when doing walks, I typically make one pass down the walk, turn 180º, turn the chute 180º, and walk back. Not too much chute movement, but its consistent when it pops up.

Every blower I've used before has varying degrees of terrible chute controls. Some have fixed angle, which is set via a screw, and can't be changed. That's not really too much of a problem, you just aim the chute down and go about your business. But the direction is the issue. Many have a crank and worm-gear based approach, so every time you want to move the chute 180º, you have to spin a crank far too many times. Do that at the end of a few runs, and you'll start to hate the thing.

Which is where the Toro's approach becomes the distinguishing little thing. Toro uses a control interface that works like a joystick. You grab the stick, and can move it around a hemisphere. Every movement directs the chute in some way, either adjusting the throw angle or the direction. Reversing the chute to throw to the opposite side? Just move the joystick, so it points to the opposite side. Want to throw snow high up in the air to distribute it across your lawn? Pull the stick back. It's so intuitive you don't even think about it, you just do it. It saves maybe 5 minutes a job, nothing significant, but when you're out in the cold and wet, those 5 minutes sure can drag on.

Lutron home automation

The first home automation device I ever purchased was an X10 module and IR remote. I bought it at a neighbors garage sale, for about $5. I was 10 years old. It was extremely cool to ten year old me. I never managed to get any apparatus to connect it to my computer, and after the shine wore off, it went into a drawer with other toys.

I'd followed home automation on and off for years after that. Eventually, in late 2015, while renting, I built up enough of an interest to buy some Z-Wave switches, a USB Z-Wave stick, and got started. Our rental home eventually sprouted automation in all the places that mattered; the bedrooms, hallways, and living room. I never got around to the kitchen or bathrooms.

Z-Wave was good enough, but it had a few problems. First, every software package out there had its flaws. HomeAssistant, at the time, was extremely immature and not very easy to use. OpenHAB had issues with Z-Wave locks. HomeSeer was paid, and felt dated, but "worked." Everything was a compromise, including hardware. Most switches at the time were somewhat limited in what they supported. Most didn't have comprehensive central scene support, many didn't even have proper reporting, and very few had more than a single association. Software at the time basically assumed associations didn't exist, so if you wanted to network a single smart switch to control two loads, you had to create an "automation" in your software of choice, and then hope the server it was on was reliable enough to stay up so you didn't have to think about it.

Furthermore, there was a small variety of Z-Wave devices beyond the standard switches, plug modules, and motion sensors. Want a decent looking remote? There's maybe one, and its got a somewhat modern experience. Its not something that a guest could look at and immediately intuit. Other, closed ecosystems, in this case Lutron, have very stylish remotes available. When my primary load control for my home automation was Z-Wave based, I invested in a Lutron Caseta bridge, a few plug modules, and a hand-full of Pico remotes, because they looked, and worked, better than anything in the Z-Wave system. Z-Wave was still what I put in the wall, the Caseta wall switches at the time were rather "techie" looking, and the better looking Maestro switches weren't an option unless you upgraded to RadioRA2. RA2 select, the Sunnata dimmers, and other modern additions to the Lutron product line, didn't exist at the time.

Aside from looking better, Lutron's plug modules and remotes had a few other things going for them, that weren't possible (at least from my research in 2016) with Z-Wave. You could hold down a button on a Pico remote, and the corresponding loads would change while you were holding it. Release the load, and the lights would stop at the level you released at. Z-Wave required you to work in "steps" and fixed percentages; you'd say set bedroom lights to 15%, and it would undergo a smooth transition to that 15%, but if you weren't sure if you wanted 15% or 20%, you couldn't hold the button down until it "looked right".

When we finally bought a house, I decided that our HA was going to be much more Lutron based than before. Z-Wave had served me quite well, but the little things about how adjusting loads worked, about how the switches were rather ugly (unless you sacrificed features), and how software support was patchwork, had soured me on it. Fundamentally, Z-Wave did everything I wanted, but so did Lutron, and Lutron did it better. I got myself an RA2 inclusive certification, and have slowly been adding lights to each room in the house.

Lutron isn't without its shortcomings. Advanced logic is locked out of the RA2/RA3 product line; you have to buy into a HomeWorks system, which has basically no way to DIY it. The programming software is windows only, and has its own set of odd terms and features. Integrations with third parties, in this case HomeAssistant, work well enough, and let you fake certain logic features that would otherwise be unavailable. But, even in spite of all those things, Lutron scratches the itch of "little things" that Z-Wave, ZigBee, X10, and whatever else never did.

Ok but who really cares?

When I'm talking about the above points, or really anything with similar sentiment, I'll have people express some variant of "Ok, that's cool, but who really cares?" And that is a valid point! If you endlessly focus on the smallest thing, you can be prone to bike-shedding. But if you ignore them completely, you may ship sub-par products. You don't need to dive into a mire of minutiae on every subject. I find periodically stopping, taking a look at what you're doing, and seeing if there's a better way to do it is often worth the small-time detour it takes to implement.

If you have a command line task you have to execute fairly regularly, and are relying on your shell's history to pull it out, why not just copy it into a script, and then just add a few niceties around it? Modern shells have nice tools for argument parsing, so instead of having to edit the command line each time you have to do something slightly different, just encapsulate those in arguments. Future you will thank you for your foresight. Doing this doesn't take much, doesn't add much, but it's a little thing.

If you use a tool every day, you might as well look at the pain points, and try to optimize them away. Someone else may have had the same problem you did, and may have written an extension or tool that solves it. And if not, and you come up with it, someone else may stumble across your solution.

Everyone pays attention to the big things. The features listed on a marketing page, the topics mentioned during keynote speeches. No one ever really considers the tiny little thing, most don't even notice it. But when they're gone, you miss them.

I've been using Fish Shell for nearly a decade now, and over that time I've gradually developed a theme and prompt that I enjoyed. At the start of this year, I rewrote it from the ground up, eliminating many years of accumulated cruft.

The beginning

Around the new year, the Fish team released a new update, 3.6, that brought with it many new features, and some breaking changes. Historically, when a new version of the shell has been released, I've updated my prompt/theme to accommodate the new changes, but rarely made significant changes to how the underlying prompt was built. This time, I had a few days off, so I decided that it was a good time to do some cleaning, removing old features, rewriting new ones to use fish builtins, and adding support to features that Fish has grown since the theme was first created. I still wanted the theme to be roughly the same as before, but faster, leaner, and more predictably stable.

Goals

Looking at my existing theme, there were some things I wanted to keep:

• Command separators
• Right-prompt timestamps
• Git/VCS lines
• A display of the current git hash in the prompt
• Ability to support variable color themes

There were also some things that just didn't ever work right, worked but weren't ever of any use, or just were irrelevant with the ongoing changes to fish:

• Current command execution time
• Current ruby version

Initial rewrite

Years ago, when I wrote the first version of my theme, it was mostly a "port" of a theme I'd previously used on Zsh. The old zsh theme is lost to time, but when I initially ported it to fish, certain features the current fish shell has did not exist. The command separators were created using the unix jot command, with some math to calculate the width needed to print the separator. The first version of the separator didn't have a command status, and was a fixed color, but upon seeing a shell theme from a former coworker, he had short (5 hyphens) separators between commands, colored depending on the exit status of the previous command. I liked that a lot, and so for my theme (at the time, zsh based), I copied it. Later, I added a section, using box drawing characters, to display non-zero exit codes.

In the time since that theme was written and ported to fish, fish has gained the string repeat command. This command takes a string and number, and repeats the string the number of times. Simple. And exactly what I wanted. Fish also gained a new feature, called pipestatus. Pipestatus is similar to the plain ol' status variable, except it's a fish list (an array) of exit codes, one for each pipe. Useful. Implementing the printing of these in a style that fit my theme was fairly easy. Other sub-commands of the string command were used, and gave me a satisfactory output.

The rest of the prompt was pretty easy to make. Fish gives you a lot of useful little primitives for printing a prompt, such as a truncated pwd, utilities for displaying user and host name (I don't use either, I find them useless noise), and a very good VCS prompt. The VCS prompt has the ability to show you the current branch, ref, or sha, but getting it to display both a ref and a sha is still impossible. This is trivial enough to wrap in a small function, and so I did.

The right hand side of the prompt is even more trivial. Fish gives you a function to print rprompt, so I just overrode this function and made it print the date.

Finally, I hard-coded some color support into the theme. In my older version of the theme, I was using one of the base16-shell color scripts, but found they have a non-trivial runtime impact, and don't play nicely with older curses apps. Fish supports expressing colors in hex, and, provided your terminal emulator supports it, will directly print components of its "ui" in the colors you specify. You can set these via variables, most prefixed fish_color. This is all fairly well documented, and if you didn't want to do it in a theme file, you can do it via Fish's web configuration interface.

Evolving the color scheme support

Many modern terminal emulators support use of codes to set colors in the terminal. Not colors over certain areas, but rather, the terminal colors themselves. Things like the background, what the older color values are painted as, and so forth. I primarily use iTerm2, which has its own set of escape codes to set colors, and I wanted my theme to ensure that it was rendering the way I intended, so I added support for these color codes.

While doing this, I hacked in some primitive support for base16-shell themes. This was partially because I have a friend, who has expressed interest in my theme, who is unable to use dark mode due to his astigmatism. By adding a "hook"¹, I added support for grabbing the colors out of a theme file, and using them appropriately in the fish UI.

This wasn't the best approach, because both the base16-shell and my theme would attempt to set terminal colors (at least in iTerm2). So as a hack I tossed in a check for either a theme-specific variable, or the presence of a base16-shell set envar, and if found, disable the theme shell color setting.

Light/Dark mode

This was mostly satisfactory, until I got the bright idea to add automatic day-night theme support. Apple exposes the current color scheme to any application that asks for it via a defaults property, so by running defaults read -g AppleInterfaceStyle you can determine if the user is using a light or dark mode. Thinking that supporting two different color schemes would be trivial, I added a set of light colors, and a check to toggle between the two. This turned out to be a rabbit hole that I spent more time on than the writing of everything up to this point. First, reading the value doesn't always return a value. If the user is on light mode, then you get nothing back. Normalizing this value wasn't hard, but it was another thing I just had to figure out, as documentation was sparse. Second, I felt that, for non-mac users, there should be a way for them to set their color scheme preferences. Finally, for users of a specific color scheme from base16-shell, I added a check that just bypasses all the light/dark logic.

Gradually, other features grew into this functionality. None of it is terribly exciting, mostly just trial and error, so I've summarized it:

• Checking the system color scheme at prompt init time gives you a colored prompt, and never checks again. This means that if you change your scheme, you'll get a shell that is dark or light, and sticks out. I added a configuration to check the system color changes with every prompt (defaulted to off)
• Users may want to manually trigger a color refresh, such as if they've manually loaded colors or changed variables. Exposing internal functions as external ones allows for this, in a convenient way
• Some shells will still fudge color rendering, for things like bold colors. This is true, even if you set the extended color table values (more on that in a moment). There's nothing you can do about it, other than instruct your users.
• Programs using older versions of ncurses will break your 256 and true-color themes. Short of the program fixing its dependency, all you can do is reset the colors on program exit. There's no universal way to do this, so for the few apps I use that do it (tig), I just aliased their commands to a function that calls the program, and then calls the color scheme refresh command on exit.

Extended colors and other terminals

Some programs have support for "extended colors." This lets you specify additional colors over the older 16 colors, which can enhance TUIs. The base16-shell project has long set these colors, but the code I'd added to my prompt didn't. I thought I didn't care about this feature too much, but seeing the colors absent from certain programs was jarring. Fixing it was trivial enough, one just has to find the extra colors that can be set, and set them. I was only targeting iTerm2, and so this was a piece of cake.

But what about other terminals? One of my friends uses Linux, and the theme consistently didn't render as well as I'd have hoped on their terminal. And, from time to time, I'll open a terminal in VSCode, which doesn't support the iTerm2 color codes either, and so my prompt would render funky there as well. Fixing this wasn't the most trivial thing, but it wasn't exciting either. Mostly trial and error, again. Along the way I had a few false starts, with mistakes about how to encode the colors (hex pairs, 0-255 pairs, etc.), how to properly send the escape codes, and so forth. Documentation on this is atrocious. I had to consult several older websites, wikipedia, and then engage in a lot of testing in kitty.app to get the output I wanted.

During this iteration, I also "matured" some features. I genericized some terms (they previously referenced only iTerm2), optimized some code paths, and fixed up the readme (which I wrote using asciidoc). The theme is now "stable", and I've not had to make any changes since. If you do choose to use it, and find something wrong with it, please open an issue, and we can figure it out.

I've been using Markdown for a long time, and have grown accustomed to it. It has various quirks, features, and oddities, but what doesn't. But recently I decided to take a look at Asciidoc, a Markdown "competetor". I found it a great little document toolchain, but it won't replace Markdown.

Both Asciidoc and Markdown allow you to write text based content using a simplified markup syntax. Instead of heavier syntaxes, such as LaTeX, HTML, and friends, or rich-text formats that require WYSIWYG, both allow you to focus on the writing, and add formatting with simple, plain-text friendly characters. And, for most use cases, you won't really see the difference between Markdown and Asciidoc. 90% of the time, it won't make a difference in how you write, save maybe using a different character to format something some way. But the remaining 10% is where empires are built, and in this area Asciidoc is far superior.

Markdown++

Asciidoc will happily accept Markdown, verbatim. You can take a Markdown snippet, paste it into your Asciidoc file, and it will (generally) work. But ad has so much more to offer, and so many little better ways of doing something, that you'll soon wish Markdown worked similarly.

For example, Markdown has a syntax for inserting a break, inside a paragraph. You put two space characters at the end of a line, and the parser will inject a line break. Unfortunately, spaces are a "weird" character. They're not visible, unless you turn on the "show invisibles" equivalent in your editor, and even with that enabled, it can be difficult to see them while scanning a document. Also, many editors will remove trailing spaces, causing your break to disappear. You can typically configure them to not do this to a Markdown document, but its another thing you have to remember, hope the developer of the editor remembered, or set up a plugin to do for you.

You don't want to make newlines significant, like GitHub does on its issues and some other sites have done, because then you lose a great feature of Markdown (and most other markup languages): the ability to format and fit content in your editor, loosely independent of how it would be presented to the user. If you've got your editor set up to hard-wrap at 80 cols, for example, you want those lines to be joined together in the output, as the width of output content is a stylistic concern. Copying and pasting a hard-wrapped snippet of text into GitHub usually requires you to join the lines together, using an editor feature (like J in vim) if you're lucky, and by hand if you're not.

Asciidoc fixes this elegantly. Instead of making trailing spaces indicative of a significant break, they make a trailing + character indicate the break. This solves basically every single problem the Markdown implementation has. You can visually scan for the + at the end of a line, you can have syntax formatting that makes it significant, and, going the other way, you don't need special editor features to show you its presence.

And this style of minor improvements persists almost everywhere else throughout Asciidoc. The formats it chooses for its primitives are better than Markdown, in almost all cases. I'm not a huge fan of how it approaches links (you put the URL before the link's text), but it's not any stranger than HTML.

You get a lot more formatting tools out of the box, including super and subscripts admonitions, easy video and audio embeds, automatic references, tables, and more. And where features are common across both Asciidoc and Markdown, the Asciidoc implementation is typically better, such as how they handle nested and ordered lists. Markdown list depth is usually a game of wrangling with indents and newlines, to get your particular parser to pick up and agree to how they work. Asciidoc uses a repeated list delimiter approach, where you just repeat the delimiter to indicate depth:

* item 1
** sublist
*** sub-sublist
* item 2

Ordered lists are better too. Instead of having to either manually number them yourself, or just use 1. as the indicator, you just use a . character:

. item 1
. item 2
. item 3

You can also adjust where the list starts, and any skips, via attributes.

Other list types exist too, such as definition lists, questions and answers, and checklists.

Asciidoc also has first class support for "admonitions". Admonitions are basically a specifically formatted block of text, designed to draw the readers eye, and call out something that might be relevant to the surrounding text, but is ultimately not part of it.

If you've ever read the O'Reilly programming books, you should be familiar with these little things, they're used liberally.

Attributes and blocks give it superpowers

One of the most powerful features of Asciidoc is the attributes and blocks system. Asciidoc documents are structured in blocks, which are arbitrary length collections of lines. Lines can be text, attributes, directives, or formatting instructions.

You can arbitrarily create a block using some delimiters, and then use directives and attributes to change things about that block. Admonitions, which I mentioned earlier, are just one example of a special block type. There are also attributes that affect lists, code blocks, blockquotes, tables, images, and even just simple ones for adding a CSS class to a paragraph.

There are inline attributes that let you have access to more powerful text formatting, such as attributes for underlining text, inserting a kbd formatting style to indicate keystrokes, and more.

Includes, macros, and references save you time

Includes, macros, and references are all things that can simplify document creation, particularly in the case of technical writing and documentation, where you have lots of repeated content, content that could be better written in its own source file, and cross-links.

Includes are what they sound like, the ability to have all or a portion of another document injected into the current one. If you've ever used latex, and have a "master" document that includes all the other documents in your project, you'll understand how nice it is. You can split up thoughts into chapters or logical sections, with file-system level distinctions, and then arrange them however you want without having to cut and paste large amounts of text.

Macros are great for repeated urls, acronyms, and disclaimers. You can define them once, and just use a shortcut syntax to reference them anywhere.

Finally, the references syntax Asciidoc uses is far superior to the de-facto reference syntax Markdown has adopted. Markdown cross-references are a function of the output, typically being HTML, and require your output tool to generate memorable, but unique, IDs on your headers, so you can link to them like [some reference](#some-reference). There's generally no easy way to know what the reference will be ahead of time, and no way to customize or override it.

In Asciidoc? References are first class, and have special features. All Asciidoc implementations use the same rules for a reference, so its easy enough to predict what it is. References are inserted using a special syntax, compared to links, which makes them visually distinct while editing. And references, by default, insert the contents of the header that defined them at the reference site. You can override it, both at the reference site, or the default at the header. In one document, I had a very long header title, that was referenced frequently. I set its reference tag to a short 3 letter abbreviation, and the injected text to be a slightly longer abbreviation.

Extensibility

Asciidoc is inherently extensible. Since the document structure is very well-defined and described, writing extensions that hook into any part of the processing isn't difficult. You can add your own custom blocks, admonitions, or anything else. You can implement your own handlers for things such as video tags, so you can reference a YouTube video as simply as writing video::4QdWRgNdir4[youtube]

The downsides

After reading all that, you might be wondering if there's anything bad about Asciidoc? I've waxed positive about it for nearly 8000 characters, but in fairness we should discuss some of the things that aren't so good about it.

First off, it's a single implementation.¹ This is both a blessing and a curse. The blessing is that you only ever have to worry about how your document will be parsed once. The curse is that you have to be happy with whatever the main Asciidoc developers decide, or write your own extensions. If your extensions get too far out of sync with the main standard, you kind of run into the problem Markdown faces, where you're basically a language that looks and kind of reads the same, but is ultimately incompatible.

Single implementation also limits its utility for other systems. If you're maintaining a service, like StackOverflow, Reddit, or GitHub, and you want to parse Asciidoc content for your users, doing so can be more complicated than it would be with Markdown. I wouldn't be surprised if there's a Markdown parser in every programming language ever written, but I would be surprised if you can find more than a handful of Asciidoc processors. The Asciidoc website lists 3 official ones, a ruby one, a javascript one (transpiled from ruby), and a java one. There's no C implementation, no rust one, none of that. So if you want to get Asciidoc support in Elixir, you have to either write your own, hope someone else wrote one, or come up with some way to shim one of the official ones into your application (NIF, port, dedicated service).

With Markdown, if you wanted to parse user content, and you were worried about the parsing being a bottleneck, you had a wide variety of options to choose from. You could just send the raw MD over the wire, and let a client-side piece of JS do the formatting (to be fair, you can do this with Asciidoc too). Or you could reach for some speed-optimized Markdown processing toolchain, implement it on your server, and go about your day. Over a decade ago, reddit moved between Markdown parsers a few times, from a python one, to discount, to a variant of sundown. I suspect other sites that parse a large amount of Markdown content, such as StackOverflow or GitHub, have done similar things.

GitHub does support Asciidoc, and their support is very good, but it runs in Asciidoc safe mode, which disables some of the more interesting features such as includes. Additionally, you have to do some special trickery to get admonition icons working right.

Markdown is still king

Markdown has inertia, and that's one hell of a thing. Its almost ubiquitous at this point. There are editor plugins, there are services for it, there are universal conversion tools that convert to and from Markdown. Many languages even have multiple, competing Markdown implementations.

Markdown is (loosely) universal. You can take something written using primitive Markdown (not any specific implementation's features, but the core described by Gruber or Commonmark) and use it on a huge variety of sites and services.

Markdown is fairly extensible, within reason. While the true extensibility of Markdown depends on your processing toolchain, how hard you want to work, and what you're willing to do, it is ultimately still extensible. Asciidoc extensions are more standardized and easier to reason about, but there are simply more Markdown extensions and implementations out there.

Want GitHub-style Markdown? Go for it, GitHub has even published a standard for GFM. Want your own? No problem, Slack and Telegram have both done it.

If you want to build documentation sites, static sites, dynamic sites, use a CMS, use a form, whatever, chances are there's extensive Markdown support for what you want to do.

I looked into using Asciidoc for my blog. I got really excited to do so, but then ran out of steam almost immediately. There's just no real extensive support for it, so anything I was going to do, I'd be blazing my own trail. While those kinds of projects are often really enjoyable and educational, I just wanted to get the blog online, so I deferred.

I'll still likely keep using Asciidoc for certain types of documentation sites, although with limited support in Elixir's HexDoc, I'm not sure how often I'll be able to.

Addendum

In the Hacker News discussion on this post, zh3 pointed out that, on many systems, an installation of Asciidoctor is frequently rather heavy. This, as pointed out by yrro, is because it tries to ship with support for db-LaTeX, which allows for the generation of LaTeX (and therefore PDF) files via Docbook. On systems that use Apt for package management, one can just get Asciidoctor, and none of the latex support, by running

apt install asciidoc asciidoc-dblatex-

You can also just install the bare minimum package by running apt with --no-install-recommends

Djot

Some other commenters, both on reddit and HackerNews, have pointed out the djot markup language as a markdown alternative. Djot was created by John MacFarlane, creator of Pandoc and driving author behind CommonMark. It's safe to say that he understands markdown more than almost anyone else, and Djot arose out of some of his ideas on how to "fix" markdown.

Djot does a lot well, some less-well, and some that I've yet to form an opinion on. I've used it even less than AsciiDoc, so take that into account. I'm mostly writing this after skimming the syntax cheat sheet.

I really like that it attempts to put user-controllable overrides to basic markdown parsing. Things like intra-word emphasis (where you have an emphasized section within a word) are tricky in Markdown, due to how the parser binds left or right delimiters. In djot, if the defaults don't do what you want, you can force a delimiter to be left or right facing, by prepending/appending a {} as appropriate: normal{_emphasis_}normal.

The "powers" granted by the use of {} as meaningful characters allow for some new syntaxes, such as {+ins+} and {-del-}, for their respective HTML tags. Similarly, super and subscript are simpler to use, because the whole block will be wrapped in {}

I like the approach to soft line breaks, which is something AsciiDoc also does better than markdown. I actually like the djot approach more, as its more inline with other tools that programmer-types might encounter. In djot you make a soft line break by ending a line with a \ character.

I'm not sure if I like the idea of smart punctuation by default. I understand, most of the stuff we're going to write in a markup language will be prose, and for a long time a lot of us just ran our MD content through SmartyPants or a similar tool, but I've fought too many battles with the punctuation prettifier on macOS to feel entirely comfortable with this approach. At least, with djot, there's an easy escape hatch via the slash-escape (\"), consistent with other special characters. And djot quotes also respect/listen to {}, similarly to how strong and emphasis characters do, to indicate the direction of a quote.

Attributes and raw sections (both block and inline) are welcome, although I haven't run into much need for output-conditional segments of a document before.

Djot's pipe tables are better than most markdown implementations, as they allow header-less tables to be constructed:

| cell 1 | cell 2 |
| cell 3 | cell 4 |

Djot lets you have multiple headers in a table, which is nice for more complex applications.

Unfortunately, it doesn't improve on the ability to do col/row spans, so you're still stuck doing them by hand in pure HTML.

Finally, heading links get a tiny improvement, in that every single heading defines itself as a reference link. Reference links are a feature of djot (and markdown) where you can put the url somewhere in your document, typically at the bottom, with an identifier, and then only use that identifier at the individual link sites.

This has a [reference link][]

[reference link]: https://www.youtube.com/watch?v=t5JDypdHNnw

This solves the issue of having to figure out how the heading text would be transformed into an anchor tag, although I still would prefer the ability to make explicit reference links, complete with their own text, like AsciiDoc permits

Showers are a part of my daily routine. And I've been putting up with a lousy shower head for too long. But I've finally found a new one, one that I like.

Yep, that's right, I've written an entire blog post about a showerhead. It's a bit of a review, a bit of me having a little nerd-out over a "gadget", and hopefully an interesting article.

The past

A lot of people will agree with me when I say that showers don't feel as satisfying as they used to. Not much has changed, the US national shower maximum flow rate is still 2.5gpm, and has been since 1992¹. Some states have lower maximum standards, and so manufacturers have been targeting lower and lower flow rates.

Shower head manufacturers attempt to juice their showerheads, make them deliver a shower like the ones we used to have, by a variety of tricks. Aerating the water, decreasing the number of holes, spreading them out, nozzle design, and so forth. The problem is that these all fundamentally don't work. They make the streams feel weak, prickly, cold, dispersed, or otherwise unsatisfying.

I've tried many showerheads over the years, both as they come from the factory and with the flow reducer removed (in one case, drilled out). Things would get a little better, but still nowhere near as good as I remember the showers I had as a child at my grandparents house was. They had an old speakman showerhead, and that thing would just absolutely drench you in water.

Some people would point out that you can probably find such a showerhead on Ebay or at a Garage sale. That's true, you can buy an older shower head and experience torrential showers again, but there is something to be said about using a bit less water. At the very least, your water bill stays low.

The problem

In my home, we have 5 bathrooms. The previous owner left behind a variety of showerheads, the most common being simple Waterpik brand sprayer, with "6 spray patterns" proudly silk-screened onto the white plastic side. These are adequate, but ultimately unsatisfying.

In the master bathroom, I replaced it with a Moen magnetic showerhead, something we moved with us from our previous apartment. This shower works, has a wand and fixed unit, and was "good enough" for me to put up with.

I put a Delta equivalent of the Moen shower head in the main bathroom on the upper floor, which was mostly used by guests, and forgot about it. I never really cared for it, but it got the job done for a few years before being replaced by the Moen.

I hadn't given much thought about replacing any of the showerheads, until my father-in-law was visiting for Thanksgiving, and mentioned that one of the waterpik showers, in the kids jack-and-jill bathroom, was far superior to the Delta. He's visited us several times, but this was the first time it came up. Chagrined, I apologized for the lackluster shower, but didn't really plan to do anything about it, as modern showerheads are unilaterally pretty terrible.

I mentioned it on the phone to my parents, who, when visiting us, typically sleep in a bed on the ground floor, not upstairs, and have their own small bathroom adjoining. This bathroom has a waterfall style showerhead. They told me that this showerhead was very unsatisfying as well.

So I've now had two complaints about crappy showers. This nagged at me, and eventually I decided to see if I could fix it.

The fix

Browsing online, I searched wirecutter shower to see if Wirecutter had a new review on showerheads. They'd praised the Moen and Delta showerheads I'd had before, so I wasn't really hoping for much, but lo and behold they'd updated their review. They still push the multi-head units I'd used before, but in the "also great" section, they mentioned a new brand, High Sierra.

The review was generally positive, but the showerhead was critiqued for its lack of features. I've never used anything other than "spray," and so this didn't bother me. Figuring I could return it if I didn't like it, I bought one of the 2.0gpm fixed sprayers.

The future

When the shower arrived, I was initially impressed. Inside its extremely minimal packaging, it sits, with no flair. It's heavier than the other showers I have, which are physically larger, but made of plastic. It's got very simple construction, and everything feels well-made. The ball pivot joint moves freely, the toggle is smooth and easy to articulate, and installation was a breeze, as the mount has two flat sides, so you can get a pair of smooth-jawed pliers around it, to cinch down and prevent leaks.

Turning the water on is surprising. You'd expect a 2gpm shower to have less presence than a 2.5gpm one, but that was not the case. Immediately a thick, wide spray of water came cascading down. Angling the shower up to spray my head, and not my torso (being tall has its own issues), I was very quickly soaked. Toggling the water off, I lathered up with a shampoo bar, rubbed some Dial soap on the important areas, and turned the water back on. Rinsing off was a treat, the water easily carried the soap and grime away.

Taking a moment to just relax and enjoy the shower, I noted that it generates a fantastic amount of steam and vapor. This made the whole shower feel warm, not just the area under the spray.

Stepping out, I decided to purchase these for all 5 of my showers.

For the master, I've installed a combo unit, which has two 2.0gpm units, one fixed and one on a hose. You can turn them on and off independently, so if you want you can have a 4gpm shower.

For the upstairs main, I ordered a handheld unit, as it is a shower tub, and the handhelds are useful for cleaning the tub, a dirty child, and filling mop buckets.

For the other two bathrooms, I've ordered more of the fixed unit that prompted me to write this article.

You seriously wrote an article about showers?

Yup. I've always had a strong sense of "If you're going to do something, do it right." Particularly for things you do every day (hopefully). Programmers endlessly tune their programming environments, changing and configuring editors, shells, terminal emulators, browsers, tooling, and endless other changes, designed to improve their experience. Why not apply this to the rest of your life?

Additionally, I just feel impressed by the company that makes the shower heads. They're a small company, making their showers here in the US, using metal, and not going in for heavy marketing or packaging budgets, or really anything other than the showerhead itself. I want to see them continue to succeed, and so I'm waxing poetic about them.