A custom doom renderer could produce edge lines directly. Maybe. Worst case you could render each surface a different solid color and the do edge detection / line conversion. Sprites as they point out should get separate handling.

Vector graphics are still very cool, and becoming quite rare to see in the wild.

Not sure I understand the lissajous figures part... isn't that an x-y plot where the x and y waveforms are both sine waves? Couldn't x and y channels be controlled more flexibly here? Or is this part of the challenge?

Lissajous figures? They just reinvented the vector display. That's no more lissajous figure as a pixel-based display is Tetris (in each case you can display that, but many other things too).

Now I'm wondering what I can do with my Vectrex if I replace the CPU/RAM with something modern.

Of course it's no longer a retro console, but cool to keep it in the original case with controller(s).