When you get to be smart writing a macro

102 pointsposted 4 months ago
by borjs
(tonsky.me)

27 Comments

zelphirkalt

4 months ago

I don't think this is a good enough reason to write a macro. Macros should be used for doing things one otherwise couldn't do and have sufficient advantage. For example the threading macro used right there in the code is worth writing a macro, because it makes things easier to read and follow and couldn't be done like that using functions, because of the order of evaluation and argument injection that the threading does. Another example are new define forms, which by definition cannot be expressed resulting in the same syntax using functions. Defines are special. Another example for a justifiable macro could be a timing macro, which relies on changing the order of execution, because when you write (time something) then something would be evaluated before time is ever called, if time was a function.

But the #p in the post seems to me to be a dubious choice for writing a macro. Too specific, too easy to use something else, too much confusion, too little gain.

dkersten

3 months ago

I started using Clojure in 2009 and used it heavily for a long time. I even ran my local Clojure meetup for many years. Lately, I've moved away from Clojure however and today I mostly write Typescript, some hobby C++, some Rust. Gleam when I can (which is rare). Anyway, that's just context to say that I've written and enjoyed a lot of Clojure which made me think about macros a lot over the years.

I've come to the conclusion that macros are NEVER the right choice for normal application developers and only RARELY the right choice for library authors.

I would pick function over macros every single time.

Even in libraries, I feel that most uses of macros are unjustified. Even in cases where macros enable something that wouldn't otherwise be possible, I question whether its really the best way. For example, its pretty cool that core.async could be implemented as macros, but I feel that core.async has rather poor developer ergonomics because of it and building it into the language itself would have lead to a much better system with a much better developer experience. I have a number of reasons, but I'll just mention the biggest here: macros cannot see into functions, so core.async requires its async functions to be called directly within a go block (eg you cannot wrap core.async/<! inside a helper function because the macro won't be able to find it to transform it).

Sometimes using macros means that things can be optimised at compile time (eg expanding core.match or specter selectors), but I feel these cases are pretty rare.

I do like many of the macros in clojure.core (threading macro etc) but I see these as a language implementation detail -- they could have been built into the language grammar or compiler and the end user experience would be the exact same.

I do wish Gleam supported some limited form of macros to generate code based on annotating types (kind of like Rust's derive), but I very much agree with Gleam's logic for not including macros, and my experience with Clojure has basically solidified my feelings that macros are very rarely a good idea.

dkersten

3 months ago

> I would pick function over macros every single time.

I should have went into a little more detail, maybe.

The general advice has always been "prefer data over functions, prefer functions over macros", but I don't think "prefer" is strong enough. I would rephrase it as:

"Prefer data over functions. Only use macros if there's absolutely no other option."

That means that macros shouldn't be used to make code more terse, or more convenient, or more "pretty". They should be used when they make code possible that wouldn't otherwise be possible (at least without jumping through a lot of hoops). For all my complaints, core.async is actually a good example of a good use of macros, as far as a library goes. It adds functionality that would be quite difficult to do cleanly without macros. My complaint is just that a macro implementation of something so integral is very much inferior to an implementation that was part of the language itself. I don't think async should be something tacked on to a language as an afterthought.

An example of what I consider a bad use of macros would be something like this:

Imagine you have a system to register event handlers that can then be triggered by name:

    (defhandler my-event (println "my-event triggered"))

    (trigger! my-event)
Many clojure libraries exist with patterns like this, especially from earlier on before the community began to shift more closely to the `data > functions > macros` mentality.

A macro-less version might look something like this:

    (register-event! :my-event (fn [] (println "my-event triggered")))
    ; or maybe: (def my-event (register-event (fn [] ...)))
    ; or maybe even (def registry2 (register-event registry :my-event (fn [] ...)))

    (trigger! :my-event)
Where the expression that the macro makes possible is wrapped in an anonymous function and the naming is explicit. Its not quite as convenient as the macro version, but it avoids magic and therefore surprises, and its more flexible because you can compose it or operate on it like any other function.

jeffrallen

3 months ago

And reader macros are a special kind of satanic evil, making sure that the next guy can't even read the code without reading all the macros first.

Or my favorite example of insanity from $WORK: a file that claims to be declarative, claiming to be "just" EDN, but which through the satanic evil of reader macros, is actually executable code.

Argh.

zelphirkalt

3 months ago

> macros cannot see into functions, so core.async requires its async functions to be called directly within a go block (eg you cannot wrap core.async/<! inside a helper function because the macro won't be able to find it to transform it).

Is that something specific to Clojure macros? How does that macro discovery process work, so that they cannot be called inside a helper function? I might not understand exactly what you mean. This sounds very limiting.

kazinator

3 months ago

No, it's a theme in syntax.

For instance, you can't put the "break" of a "for" loop in a helper function called out of the for loop. It has to be enclosed in the for loop.

We can think of the loop as a macro; Lisp would implement it as such.

When macros transform certain expressions enclosed in the macro call, all the syntax has to be right there, enclosed in the call.

In other words macros are "local syntactic transformations". Why I'm putting that in quotes is that this is the exact phrase used in the paper "Macros that Reach Out and Touch Somewhere" by George Kiczales et al.

Kiczales describes a macro system which peforms global program transformations, allowing macros to act on code that is not enclosed in them; i.e. bring about nonlocal transformations.

"In this paper, we present a new kind of macro, called a data path macro, in which transformations can take place at any point along the dataflow path that includes the macro invocation."

This is pretty exotic. I think nbody has done anything like it, and they never released their code. They left unsolved problems documented in section 5, Future Work.

zelphirkalt

3 months ago

I see. Good to know, that people have thought about that before. It would make macros even more powerful. But I am not quite sure, (a) what misuses it would encourage or at least enable and (b) how often one would need that power or how often it would be better than some other workaround. But it did come up sometimes when I did something with macros, having me think: "But how to do this???" and not finding a way, other than having the macro be in an outer scope/higher up scope, so that it has more info to work with.

I guess such a macro system might also lend itself better to implementing type systems. Which makes me think of typed Racket. But I think they are rather using source code location info from their syntax object to look up surrounding context, or are using some kind of global state to store info for inference. I don't know this for sure, because I have been unable to understand TR. Maybe someone can chime in on that.

dkersten

3 months ago

This isn’t specific to Clojure macros, but Clojure is where I’ve felt it.

Consider a macro `(replace-here replacement expression)` that replaces every instance of `here` with `replacement` in `expression`:

    (replace-here 10
      (* here (+ 5 here))
The macro sees `10` as one argument and it sees `(* here (+ 5 here))` as another argument. It can perform the replacement and produce:

    (* 10 (+ 5 10))
But if we have a function `foo` that contains the placeholder:

    (defn foo [] here)
Then our macro won’t be able to see it:

    (replace-here 10
      (foo))
This does NOT replace the “here” inside foo because the macro just sees `(foo)` but is not able to look inside foo.

We can see this I practice in core.async where the `(go expr)` macro will execute `expr` asynchronously and block (await) when it encounters a `(<! channel)` call.

BUT `<!` must be visible to the go macro, because the macro transforms the code into a state machine that can suspend and resume when the call blocks and unblocks. Than means that this code will await the channel:

    (async/go
      (let [val (async/<! ch)]
        (println “got” val)))
However this code will not:

    (defn foo [ch]
       (async/<! ch))

    (async/go
      (let [val (foo ch)]
        (println “got” val)))
Because in this example the `<!` is inside the function `foo`, but the `go` macro cannot look into the function to find and transform it.

Practically, this means you cannot create helper functions that operate on channels, you must always wrap the channel operation in a go block so that the macro will see it.

In practice it’s not that bad, you learn to structure your code into ways that avoid it. But it’s a limitation nonetheless and one that only exists because core.async is implemented as macros.

As an aside, another problem I have with core.async that exists because it’s a macro-based implementation rather than a first class citizen is that there are occasions where the code transformations mean that an error can occur somewhere that isn’t in your source code. I have had a situation where there was an error and the stack trace was entirely in Clojure’s code without referencing my code in any way. Imagine how difficult it is to debug when you don’t even know what source file of yours contains the code with the error! If it was built into the language rather than as macros, then it could at least retain source location contextual data to be passed to whatever internal code raised the error. But that’s a separate issue from the “see inside” issue.

zelphirkalt

3 months ago

Maybe this seems like a bit strange design. Couldn't `<!` itself be made a macro, so that it doesn't matter where it is, instead of `<!` being something to transform? EDIT: No, it couldn't, because then that `<!` macro wouldn't know enough context, as it typically cannot look into outer scopes.

Another thing that comes to my mind reading this is, that in Guile I wrote a macro for function contracts. This macro references a unique value as the thing that should be replaced. That unique value can be imported from another module and aliased, so that name clashes are avoided. In the case of function contracts, I have a special define form `define-with-contract`, which recognizes the placeholder, where it should insert the result of applying function, to check the output contract of the function. I guess, if the placeholder made sense in other places than the contract definition/specification, then I would face the same problem, of the macro not seeing it in helper functions. Say for example I use the unique placeholder value inside a helper function, which I then use inside the contract definition.

I guess in case of a placeholder value, it couldn't be a macro itself, because then it would lack context about what should be in its place, as the scope would then be too limited.

The basic problem seems to be as you already explained, that macros cannot look inside function bodies hidden behind calls. Sort of like macros can only deal with "one layer", which is the syntax that is passed to the macro, but if that syntax contains calls to functions, then the macro is clueless about what happens inside those functions. I wonder if this limitation is present for all lisps, or if there is some macro system or method of designing them, that circumvents this problem.

But also it seems, that this is only (?) a limitation, when the macro needs to look for something specific like a placeholder. For a timing macro for example, there is no such thing and expressions merely get reordered and maybe wrapped in some `let` or something.

dkersten

3 months ago

Consider this JS/TS code:

    async function foo() {
        before()
        const bar = await take(ch)
        after(bar)
    }
its basically sugar for:

    function foo() {
        before()
        take(ch, function (bar) {
            after(bar)
        })
    }
Where the callback passed to take will be called when take resolves. And other async functions can run in the meantime.

In clojure that might look like this:

    (go
      (before)
      (let [a (<! ch)]
        (after a)))
Which would, hypothetically, convert to something like:

    (do
      (before)
      (<! ch (fn [a]
            (after a))))
If `<!` was the macro, it wouldn't be able to see the (after a) only the `(<! ch)`. `go` is the only part that can see both the code before the await and the code after the await, and it splits it at the await so it can suspend execution until the take has resolved.

EDIT: You edited while I was typing and came to the same realisation!

> I guess in case of a placeholder value, it couldn't be a macro itself, because then it would lack context about what should be in its place, as the scope would then be too limited.

In the example you gave, it could be a macro that transformed into a function call: eg `(foo my-placeholder)` turns into `(foo (read-placeholder :my-placeholder))` and that function read the registered value from a placeholder registry. It wouldn't literally transform the placeholder into the set value at compile time, but it could dynamically inject the value at runtime.

> I wonder if this limitation is present for all lisps, or if there is some macro system or method of designing them, that circumvents this problem.

I would imagine so.

Its theoretically possible that you could see the full AST + environment so that if you see a function call, you can look up the function in the environment and then access its AST. I've never heard of any language (lisp or otherwise) that did this. It could exist, though.

kazinator

4 months ago

  #p x
is one character less than

  (p x)
When code golfing Lisps you can remove all whitespace after a closing paren, but not after a symbol. So in the following fully golfed token sequences, #p loses its one character advantage:

  #p x y
  (p x)y
I bring up code golfing because that's what this is about, presumably.

But what if the argument is a parenthesized expression:

  #p(x)
  (p(x))
#p is back in the game with a 1 char lead.

The thing is, we can make the printing operator take arguments and turn them into an expression. Suppose we make a cousin of p called q, such that:

  (q x) -> (p (x))

  (q x y z) -> (p (x y z))
q no longer loses to #p:

  (q x)
  #p(x)

asQuirreL

4 months ago

I think what's more important than the character count is the fact that you can add #p with two key strokes.

Inserting parentheses requires moving your cursor around or invoking some shortcut in your editor if you use paredit, vim-surround, or a similar plugin. Applies equally for removing the invocation (although paredit makes that part easy).

user

3 months ago

[deleted]

daveliepmann

3 months ago

Isn't this the exact same number of keystrokes? 'Shift-3 p' versus 'Shift-9 p' on my keyboard.

zamadatix

3 months ago

I think GP is saying you don't need to define where a closing 'Shift-3 p' goes, not that the initial character is a single key.

A Lisp dialect is probably a poor choice if that's one's concern though.

eviks

3 months ago

The closing parenthesis is auto inserted, so also the same?

dkersten

3 months ago

paredit, parinfer and whatever other Clojure/lisp editing tools exist make this trivial though. Editor macros also exist to wrap expressions in calls.

kazinator

3 months ago

Good point. The parinfer implementation just perhaps needs some kind of nudge to know that when (p is added in front of an object, the parenthesis goes after just one object. If it creates the matching parenthesis in the wrong place (like end-of-line), then you have to manually mess with parentheses.

theknarf

3 months ago

Loving the dark mode.

elicash

3 months ago

Yes, I love the whimsy of that and of the "hamburger" icon.

waffletower

3 months ago

Spyscope has some similar reader macros. One of my favorites, obviously for the REPL and not for production code, is quite simple -- {eva/l clojure.core/eval} -- I use it to evaluate forms in the scope of the REPL prior to passing them onto threads that don't share the REPL interns.

user

4 months ago

[deleted]

EdwardDiego

4 months ago

Very smart. But also a good example of why macros are brittle.

user

3 months ago

[deleted]

ares623

3 months ago

Macros were the OG prompt engineering.

webdevver

3 months ago

thank goodness for firefox reader mode