Writing in Djot

History

This is more of the history as it pertains to this blog. Djot itself has its own interesting history, starting with John MacFarlane’s beyond markdown blogpost, and moving into a proper set of rationales and then a full specification.

A few years ago, I wrote a piece on AsciiDoc and Djot, as alternatives to markdown. I mentioned at the time that I’d looked into using AsciiDoc to write this blog, but deemed that it would be untenable, due to the dearth of converters and simple tools at the time. With regards to AsciiDoc, not much has changed in 2.5 years, so its more or less still a curiosity more than a useful tool.

Djot, on the other hand, has seen a bit more success in adoption. While nowhere near the success of markdown, Djot has parsers available in a number of languages, some blessed officially, others just working nicely and quietly. It reminds me of the state of Markdown in the early 2010s, where people were ready to move beyond markdown.pl, but the field wasn’t saturated yet. Unlike that era however, Djot started off with rigorous standards, and so implementing support has a lot fewer unknowns.

Djot in Elixir

When I first started thinking about writing posts in Djot, I explored the field, to judge feasibility. At the time, there were no native Elixir, Erlang, or any other BEAM language based Djot parsers. There was, of course, the official JavaScript/TypeScript implementation, a few implementations in other languages, and Pandoc support, so there were options.

At first, I explored writing a parser. I looked at using Packrat grammars, which are used to write the slime elixir parser, but ran into some issues with how Djot would be parsed. I’m sure if I’d kept at this path, a proper grammar could have been written, quite easily. But this was still at the toy project stage, and I didn’t want to spend hours thinking about grammars. This also knocks writing a parser using tools like NimbleParsec out of the running, as thats just trading one parsing technique for another.

I toyed with the idea of writing a simple Port around Pandoc, since it has very good support. But ports are always tricky beasts. You have to figure out how to manage their version with the systems you’d install them on, in this case getting it running neatly in a Github Actions workflow, as thats what I use to build this site. Finally, I’d have to choose an output format from Pandoc that Tableau could use. Would I output markdown and have Tableau then parse+render it? Would I directly output HTML and transform it? Or should I use some intermediary format, an AST of some kind.

Using a port ultimately seemed like more complexity than it was worth, and this also disqualified a number of other options, like a shim to run the JavaScript based implementation. So all that really left was either writing it entirely in Elixir, which I discussed above, or using an existing implementation in another language that could be brought into elixir via a NIF.

I’ve done a bit of work with NIFs before, both in closed source codebases as well as minor contributions to tools like MDEx, the markdown engine used by this blog, so I’m a little comfortable with them. I still have a lot I don’t know about them, but I knew enough to get started.

Djot in Tableau

Around October 2024, I had succesfully moved my site from NuxtJS to Tableau, and was writing in Markdown. I messaged the author of Tableau, the excellent mhanberg, asking about feasibility of getting other markup parsing formats working in Tableau. He was open to the idea and started implementing it, but there were a few minor issues with my elixir djot package at the time. I’d specified it with a rather strict version for Rustler, which prevented it from compiling neatly with Tableau, which uses a Rustler based package for markdown parsing (mdex).

The easiest way to get them both working well together was just to relax the version number, which is what was done for Djot v0.1.2. But that still means that, to compile the application, you have to have a full rust install alongside your elixir install.

Plans for the future of Djot in Elixir

Working with the MDEx API has been a joy, and working with the HTML output from jotdown has been a bit of a pain. Since jotdown has an event based system under the hood, I think a project I might explore over the next few months is to implement a Djot AST, similar to MDEx’s AST. It wouldn’t be 1:1, as Djot and Markdown have different structural components, but it would be similar enough to have the same good ergonomics. In particular I am a massive fan of how MDEx implements Access, which leads to one of the most ergonomic tree manipulation experiences I’ve ever had in any language:

# Walk the MDEx AST, finding all MDEX.Image structs, and rewrite their URLs to netlify urls
defp do_netlify_images(pipe) do
  selector = fn
    %MDEx.Image{url: <<"/postimages/", _::binary>>} -> true
    _ -> false
  end

  Pipe.update_nodes(pipe, selector, fn %MDEx.Image{url: original_url} = image ->
    %MDEx.Image{image | url: "/.netlify/images?url=" <> original_url}
  end)
end

I’m also interested in moving Autumn based syntax highlighting over to the Rust side of things, so that it can occur with some native speed, without the serialization and HTML parsing costs. For Elixir Djot to be something usable in a dynamic setting, thats a bit more of a requirement.