Gerbil, You Better Werk

Well, while I’m here I’ll do the werk — and what’s the werk? To ease the pain of living. Can I get an amen?

― RuAllen Ginsberg

By line count, I just wrote the Lispiest program of my life. By that I mean that the number of lines of code that comprise Lispy things—code as data, macros, etc—as opposed to normal blub things, comprise about 70% of the program. This was a fun program to write—with one surprising, notable exception—and all that Lispiness was a big part of it. I’d like to share that fun with you.

The program is a task runner. Like you could use make for, though some people would rather you didn’t. (Of course there are plenty of other tools that already do this, but, as far as I know, none use a Lisp as an implementation or configuration language.)

With this program, you can write a file like the one below. This is the file I use for the Portland Haiku Dispensary.

$ cat tasks
(task css
      "Make the index.css."
      ($ sass --style=compressed --no-source-map styles/main.scss public/assets/css/index.css))
     
(define home-page-path "public/index.html")
     
(task page
      "Make the home page and write it to the file."
      ($ ./db.ss > $home-page-path))
     
(task home
      "Make the home page and write it to stdout."
      (load "db.ss")
      (make-home-page))
          
(task pull-db
      "Pull the database down from the server."
      ($ scp user@pdxhd.info:~/pdxhd.info/db-file db-file))
     
(task push-db
      "Push the database up to the server."
      ($ scp db-file user@pdxhd.info:~/pdxhd.info/db-file))
     
(define poem-pages-path "public/poems")
     
(task poem-pages
      "Makes a page for each poem in the database."
      (load "db.ss")
      (make-poem-pages poem-pages-path))
     
(task (qr-code poem-id)
      "Make the QR code for the given poem."
      (load "db.ss")
      (let* ((poem (query-by-id poems: poem-id))
             (uuid (assoc-val poem uuid))
             (image-file (string-join uuid ".png"))
             (url (string-join "https://pdxhd.info/poems/"
                               uuid
                               ".html")))
        ($ qrencode -s 200 -d 600 -o $image-file $url)))

And run tasks like this:

$ werk css
sass --style=compressed --no-source-map styles/main.scss public/assets/css/index.css
Done.

And get help like this:

$ werk --help
css: Make the index.css.
page: Make the home page and write it to the file.
home: Make the home page and write it to stdout.
pull-db: Pull the database down from the server.
push-db: Push the database up to the server.
poem-pages: Makes a page for each poem in the database.
qr-code poem-id: Make the QR code for the given poem.

If you’re interested, the code is on Sourcehut. If you want to run it, you’ll need Gerbil. If you don’t yet have Gerbil, get it, it’s great.

Tasks

Let’s start with the tasks file. Here’s a task:

(task css
    "Make the index.css."
    ($ sass --style=compressed --no-source-map styles/main.scss public/assets/css/index.css))

It’s so simple. You don’t even need it explained. I’ll explain it anyway. It defines a task named css.

The quoted string is some documentation that describes the task. It’s useful both to a reader of this file and to someone looking for help on the command line, as you saw above.

The next line, ($ sass ..., comprises the body of this task. There’s just one action: a shell command that generates the index.css from the main.scss.

Note that there are no quotes needed around the shell command. It appears just as you’d type it on the command line, or in a Makefile. We’ll explore the $ macro later.

This task:

(task page
      "Make the home page and write it to the file."
      ($ ./db.ss > $home-page-path))

demonstrates how to use a variable ($home-page-path) in a shell command. A variable can be referenced in a shell command by preceding its name with a $. We’ll discuss this later too.

This task:

(task home
    "Make the home page and write it to stdout."
    (load "db.ss")
    (make-home-page))

demonstrates a different way to do the same thing as the page task. Instead of executing the db.ss script via a shell command, this task loads that file and calls the make-home-page function it defines. Which saves you from having to write a main function to execute the script, from handling command line arguments and dispatching to helper functions, etc.

The tasks file is a Scheme program. It gets loaded by werk, which is also a Scheme program, so the entirety of Scheme is available to the task at hand.

For example, this task:

(task (adds a b)
    "Adds a and b and prints the result."
    (println (+ a b)))

takes two arguments, a and b. It’s not a very useful task but it demonstrates (1) how to define a task that takes arguments, and (2) that numeric arguments are converted to numbers before being passed to the task.

This task is more useful:

(task (qr-code poem-id)
      "Make the QR code for the given poem."
      (load "db.ss")
      (let* ((poem (query-by-id poems: poem-id))
             (uuid (assoc-val poem uuid))
             (image-file (string-join uuid ".png"))
             (url (string-join "https://pdxhd.info/poems/"
                               uuid
                               ".html")))
        ($ qrencode -s 200 -d 600 -o $image-file $url)))

It takes an argument, being the ID of the poem to work with. It then loads the db.ss file, queries the database for the ID’d poem, gets its UUID, defines a path for an image file, defines a URL, and shells out to qrencode to write a QR code encoding that URL to that file.

So, aside from all the parentheses, what makes this file so Lispy?

The task macro shows how, in Lisp, code is data, and data is code. The task’s structure defines a data type: a name, optional arguments, an optional docstring, and at least one expression that defines the function body. The macro does multiple things with each of those.

And the $ macro demonstrates some of Scheme’s flexibility with handling symbols. This expression:

($ ./db.ss > $home-page-path)

isn’t valid shell or Scheme: the symbol ./db.ss isn’t bound to anything; the symbol > is bound to the greater-than function, which isn’t what we’re using it for; and the symbol $home-page-path isn’t bound either, but home-page-path is, and its value is what we’re interested in there. But the process of converting these symbols into a valid shell command is easy in Scheme—I’m not sure this would be possible in a language that lacks macros, at least not without quotes and eval.

It also demonstrates, as we’ll see later, how you can use Scheme to modify Scheme’s own reader to change its syntax.

Werk

Let’s go through the $ macro first.

The work is actually split into two macros. Here’s the first one, annotated. Its job is to convert something like ./db.ss > $home-page-path into a shell command, then run it.

;; Define the macro.
(define-syntax $
  (syntax-rules ()
    ;; Specify its pattern.
    ;; As with other Lisp macro systems, Scheme macros receive
    ;; forms after they're read but before they're evaluated,
    ;; and return the forms to evaluate.
    ;; Unlike other systems, they work based on [pattern matching](https://www.scheme.com/tspl4/syntax.html#g135).
    ;; This pattern matches 0 or more forms.
    ;; The `_` is shorthand for the macro's name.
    ((_ forms ...)
     ;; Bind some variables.
     (let* (;; Collect and convert the `forms ...` into a string.
            (command ($/collect forms ... '()))
            ;; Run the command and capture the result.
            (result (shell-command command #t))
            ;; Split the resulting exit code...
            (exit-code (car result))
            ;; ...from the output.
            (output (cdr result)))
       ;; Print the command to stdout, like `make` does.
       (log command)
       ;; If the command ran successfully...
       (if (eq? exit-code 0)
         ;; ...just return true.
         #t
         ;; Else...
         (begin
           ;; ...print an error message...
           (log "Error: " output)
           ;; ...and return false.
           #f))))))

This is a helper macro. Its job is to convert the symbols passed to $ into a string that shell-command can use.

;; It's named $/collect to indicate that it's a helper.
(define-syntax $/collect
  (syntax-rules ()
    ;; This pattern separates the uncollected `forms` from the
    ;; `parts` of the resulting string.
    ((_ form forms ... parts)
     ;; This macro is recursive. It recurs with the `forms ...`,
     ;; and the `form` is inspected and added to the `parts`.
     ($/collect forms ...
                ;; Collect the possibly-quoted string into the `parts`.
                (cons (quote-maybe
                       ;; If the form is a symbol (and not a string, number, etc)...
                       (if (symbol? (quote form))
                           ;; ...convert it to a string, bind that to a variable, and...
                           (let ((sym-str (symbol->string (quote form))))
                             ;; ...if that string starts with a "$"...
                             (if (string-contains sym-str "$" 0 1)
                                 ;; ...that means its a variable, so remove the "$",
                                 ;; convert the truncated string back to a symbol,
                                 ;; evaluate that symbol to get its value, and return
                                 ;; that value's printed (string) representation.
                                 (displayed (eval (string->symbol (substring sym-str
                                                                             1
                                                                             (string-length sym-str)))))
                             ;; Else, just return the symbol's string representation.
                             sym-str))
                           ;; Else, return its printed representation.
                           (displayed form)))
                       parts)))
    ;; This is the base case for the recursion.
    ((_ parts)
     ;; Reverse the list of strings and join the parts with spaces.
     (string-join (reverse parts) " "))))

Here’s an example of how $/collect steps through its arguments to produce a valid shell command:

0. ($/collect ./db.ss > $home-page-path ())
1. ($/collect > $home-page-path ("./db.ss"))
2. ($/collect $home-page-path (">" "./db.ss"))
3. ($/collect ("/path/to/home/page" ">" "./db.ss"))
4. "./db.ss > /path/to/home/page"

The task macro is simpler than $:

(define-syntax task
  (lambda (stx)
    ;; It uses `syntax-case`, not `syntax-rules`, which involves a little
    ;; more code but in some ways provides a simpler mental model: it's a
    ;; procedure that maps a syntax object (`stx`) to a `syntax` object.
    (syntax-case stx ()
      ;; As with `syntax-rules`, it matches the pattern of the input form.
      ((_ (name args ...) form forms ...)
       ;; If the value bound to the `form` pattern variable is a string,
       ;; that becomes the docstring.
       (if (string? (syntax->datum (syntax form)))
         ;; This is the returned syntax object. It...
         (syntax (begin
                   ;; ...defines a function with the name, args, and body forms...
                   (define (name args ...) forms ...)
                   ;; ...and adds a pair to the *tasks* list that associates the
                   ;; `name` with the arguments and docstring.
                   (set! *tasks* (cons (list 'name '((args ...) form))  *tasks*))))
         ;; This case is the same, just without a docstring.
         (syntax (begin
                   (define (name args ...) form forms ...)
                   (set! *tasks* (cons (list 'name '((args ...) ())) *tasks*))))))
      ;; This block is the same as the one above, but for tasks with no arguments.
      ((_ name form forms ...)
       (if (string? (syntax->datum (syntax form)))
         (syntax (begin
                   (define (name) forms ...)
                   (set! *tasks* (cons (list 'name '(() form)) *tasks*))))
         (syntax (begin
                   (define (name) form forms ...)
                   (set! *tasks* (cons (list 'name '(() ())) *tasks*)))))))))

The function described by the task is defined at the top level (so one task can call another), and its name is added to the *tasks* association list, so we can verify that the task named on the command line is valid) before calling it by werk’s main function:

(define (main . args)
  (if (file-exists? tasks-file-path)
    (begin
      (with-reader (#\| read-as-is)
                   (load tasks-file-path))
      (if (need-help? args)
        (print-help)
        (let ((name (string->symbol (car args))))
          (if (assoc name *tasks*)
              (begin
                (apply (eval name) (convert-args (cdr args)))
                (println "Done."))
              (println "Unable to run task: " (car args))))))
    (println "The tasks file doesn't exist.")))

That with-reader touches on the less-than-fun surprise I mentioned at the beginning of this post. I had no idea that the pipe character is syntactically significant in Scheme—it acts as a quoting character for symbols, enabling symbols with spaces, parentheses, all kinds of fun:

> (let ((| this is a long symbol with spaces | 123)
        (| this is another long symbol, oh why god (don't answer, please) | 234))
    (+ | this is a long symbol with spaces | | this is another long symbol, oh why god (don't answer, please) |))
357

Clearly that conflicts with the hoped-for ergonomics of the $ macro, in which you’d want to use the pipe character for piping. So that required this macro:

;; with-reader evaluates the `body` expression/s with a readtable modified
;; per the given `char` and `handler`. This is useful for only temporarily
;; modifying the readtable.
(define-syntax with-reader
  (syntax-rules ()
    ((_ (char handler) body ...)
     (let ((extant-handler (##readtable-char-handler (current-readtable) char)))
       (##readtable-char-handler-set! (current-readtable) char handler)
       (let ((result (begin body ...)))
         (##readtable-char-handler-set! (current-readtable) char extant-handler)
         result)))))

And this helper function:

;; read-as-is reads the given character with the given readtable and
;; returns it as-is. This is useful for removing the syntactic
;; significance of characters, like `|`.
(define (read-as-is readtable char)
  (##read-next-char-expecting readtable char)
  char)

I’m not especially proud of these, and they’re Gerbil/Gambit-specific, but they get the job done. And I’m happy that, even though neither Gerbil nor Gambit really support Common Lisp-style read macros, it’s still possible to hook into the reader’s operations and work some magic.

And, along the way, I discovered (1) that one of Gerbil’s core developers has added a Common Lisp-inspired feature to Gerbil’s repl via read macro, and (2) Chicken Scheme has exactly what I was hoping for:

> (set-read-syntax! #\| (lambda (port) (read-string 1 port) "|"))
> '(a | b)
(a "|" b)

So it goes.

But Why Werk?

Because that’s what happens when you write programs while drinking wine and watching Ru Paul. You’re welcome, America. I’ve given you all and now I’m nothing.