Show HN: Let your AI agents paint big arrows, boxes and text on your screen

419 pointsposted 5 days ago
by franze
(github.com)

217 Comments

sicktriple

4 days ago

Man, just when it seemed like we had it all. Computers were cheap, efficient and powerful. Somehow we figured out a way to accomplish tasks we already had solved except now it's 1000x more expensive, requires the combined electricity of the entire world, is reliant on someone else's rented compute, and now I need a robot to tell me what button to press. What a time to be alive.

scotty79

4 days ago

In 40 year software designers couldn't figure out that there should be a search function for every functionality that your computer provides.

When we got a form of such search, it searched all of the internet alongside with the functions of your software on your computer. You had to tell it manually what software you have and when it found something you had to fish out the description of what to click to get the functionality you are seeking and then click through some dumb ui to actually invoke it.

Agents are the first thing that can lead you directly from "I know what I want my computer to do." to "Actually doing it."

Success of agents is founded on the profound and sustained failure of all of the software designers and developers ever.

sicktriple

2 days ago

> Agents are the first thing that can lead you directly from "I know what I want my computer to do." to "Actually doing it."

Actually they’re the second thing. The first thing was studying computer science.

drowsspa

3 days ago

In a (maybe un-) related note, Docker is basically this but for using C/C++ libraries. In 40 years no one was able to solve the mess

1718627440

4 days ago

> In 40 year software designers couldn't figure out that there should be a search function for every functionality that your computer provides.

apropos ?

scotty79

4 days ago

I don't understand this question. Please elaborate.

1718627440

4 days ago

Isn't that what the command "apropos" does?

scotty79

4 days ago

Not if I can't do

apropos switch desktop to dark mode

and

aprops set dns to 8.8.8.8

1718627440

4 days ago

I know this is not what you want, but actually:

   $ apropos dns
   [...]
   resolvectl (1)       - Resolve domain names, IPV4 and IPv6 addresses, DNS resource records, and services; introspect and reconfigure the DNS resolver
   [...]
   $ man resolvectl
   [...]
     dns [LINK [SERVER...]], domain [LINK [DOMAIN...]], default-route [LINK [BOOL...]], llmnr [LINK [MODE]], mdns [LINK [MODE]], dnssec [LINK [MODE]], dnsovertls [LINK [MODE]], nta [LINK [DOMAIN...]]

           Get/set per-interface DNS configuration. These commands may be used
           to configure various DNS settings for network interfaces. These
           commands may be used to inform systemd-resolved or systemd-networkd
           about per-interface DNS configuration determined through external
           means. The dns command expects IPv4 or IPv6 address specifications
           of DNS servers to use. Each address can optionally take a port
           number separated with ":", a network interface name or index
           separated with "%", and a Server Name Indication (SNI) separated
           with "#". When IPv6 address is specified with a port number, then
           the address must be in the square brackets. That is, the acceptable
           full formats are "111.222.333.444:9953%ifname#example.com" for IPv4
           and "[1111:2222::3333]:9953%ifname#example.com" for IPv6. The
           domain command expects valid DNS domains, possibly prefixed with
           "~", and configures a per-interface search or route-only domain.
           The default-route command expects a boolean parameter, and
           configures whether the link may be used as default route for DNS
           lookups, i.e. if it is suitable for lookups on domains no other
           link explicitly is configured for. The llmnr, mdns, dnssec and
           dnsovertls commands may be used to configure the per-interface
           LLMNR, MulticastDNS, DNSSEC and DNSOverTLS settings. Finally, nta
           command may be used to configure additional per-interface DNSSEC
           NTA domains.

           Commands dns, domain and nta can take a single empty string
           argument to clear their respective value lists.

           For details about these settings, their possible values and their
           effect, see the corresponding settings in systemd.network(5).


   [...]
Not too bad in my opinion. (I didn't actually knew before how to set the DNS server, never needed to do it.)

scotty79

4 days ago

Quite uselsess, it doesn't tell me which file to edit, what to put in there, nor what command to invoke with what arguments.

People can do and do better with googling. Proportion of people who found the answer to that question through apropos to people who found that answer googling is probably less than a million to one.

godelski

2 days ago

The man pages certainly tell you how to set your DNS to 8.8.8.8. You just have to do a bit more reading than what the agent would give you. But there's a side benefit to that extra reading, you learn how to do more things. Sometimes that's useful, sometimes it isn't. But you don't know if it is useful or not until later, you can only find out in time

sicktriple

2 days ago

Be on GNOME

Press super and type “dark mode”

What happens?

I’m not sure I understand the desire to control computers with conversational language. Computers are a collection of on off switches. I personally would much prefer being able to control exactly which on off switches to switch, instead of just telling the black box to do something and for it to make no mistakes. It just doesn’t strike me as a good idea to give the incomprehensible matrix weight the ability to make decisions about what to do on my machine. There’s just far too much ambiguity for my comfort here.

joe_the_user

4 days ago

I sort of agree that this is the consequence of software hell but it's not so much designers couldn't figure out the need for search from the start but that designers gradually made things so complicated you needed a global search functionality and then didn't provide it.

I mean, Android settings has global search because the settings menus for each Android phone is different (wtf) and some settings aren't even accessible except searching.

animal531

4 days ago

"What a time to be alive!" : I used to love Two Minute Papers, but these days all he's covering are AI releases instead of actually good papers.

debugnik

3 days ago

"He" isn't even covering it, he eventually automated himself out of the channel: He AI-cloned his voice and had it read AI-generated scripts. I assume he still does.

sandos

3 days ago

I found he got a lot less interesting when the videos became longer.

majora2007

3 days ago

Even before all the AI, his videos started lacking substance and were just high level showcases.

wvlia5

2 days ago

Dr. Karol, Johanna, and Fahir.

serf

4 days ago

i've read enough 'shlemiel the painter' ports across hundreds of systems and thousands of examples that I must quickly and coldly dismiss the precept that computers were ever used efficiently.

alanbernstein

4 days ago

Valid points, but this sounds extremely useful for learning complicated GUI applications like 3d modeling or media editing apps.

ASalazarMX

4 days ago

Valid point, but the solution is wasteful and overengineered. Imagine if videogames needed an external datacenter to run the tutorial levels.

alanbernstein

4 days ago

I don't disagree. But the existing tech solution - scrubbing through YouTube tutorial videos until I cobble together the bits of knowledge I need - is also quite overengineered, isn't it?

Obviously I'd prefer to ask specific questions to my expert human friend sitting at the next desk over, but sadly they don't exist.

I just hope there is a future where this tech exists without such big downsides.

conorcleary

4 days ago

There'll be gauntlets of pathways mapped before the first step is taken, as or before the first agent is deployed. Criteria will include resource use weighted against economic efficiency. Already being done.

arcanemachiner

4 days ago

I am willing to bet you could run a model capable of using this on a 12GB 3060 + some RAM. (Qwen 3.6 35B)

Even the data centre using this probably uses less net energy, and costs less, to build one of these dumb arrows than the aggregate sum of the energy used to keep you alive while you Googled for the answer. (Including heating/cooling the building, powering the equipment growing the food you eat, etc.)

fasterik

4 days ago

Sounds like an opportunity for a good engineer to come in and write something with the same functionality that runs locally and efficiently. The claim that it's wasteful only holds water if the non-wasteful solution exists and can accomplish the same tasks.

alanbernstein

4 days ago

Right, maybe the next step is that Blender comes with a local-gui-tutorial-llm packaged with it?

ramon156

4 days ago

isnt the LLM the wasteful part?

Nonetheless why not let an LLM make a guide and use that in the release, I don't see how that's wasteful

alanbernstein

3 days ago

A guide doesn't answer arbitrary questions, though

Levitz

4 days ago

>Imagine if videogames needed an external datacenter to run the tutorial levels.

I don't see how this is anything like that. The point here is to make it easier to give the user indications which has been a staple since forever, hell this is so much of a problem that there are entire solutions engineered to letting one user take control of the computer of another user through the net.

Beyond that, I'm sorry but looking for help in the context of videogames is extremely prevalent. That external datacenter to run the tutorial levels is sometimes called "Youtube" and others "Wikia".

godelski

4 days ago

For ages people have been saying "it's obvious" or "it's intuitive", without learning anything about design principles. Obvious to who?

People have been saying "we don't need docs", "nobody reads docs", "they'll figure it out". They experience the pain of a new employee onboarding but never treat this as a signal for the user experience. The user can't just walk over to the developer's desk. But luckily we got stack overflow, blogs, and Google. Somewhere someone explains it! The problem became finding it!

But hey, why do things the "hard" way when you can just throw money at the problem? Or better yet, dismiss the problem by calling the user an idiot.

You're right. What a time to be alive. We've created a world where this product is useful. And not even to because dark patterns exist, but simply because we've always wanted to avoid taking a step back and thinking or getting an outside view. Because it isn't obvious if a robot has to teach you what button to press. If it does, you probably should be ashamed (there are, as always, exceptions)

That said, I do think there's still a lot of utility you this. Those exceptions aren't uncommon. This'll help people learn programs like FreeCAD or Blender, or whatever. Where the complexity is naturally high. But also I do think we should recognize the silliness of many problems that this does solve that shouldn't be problems in the first place.

joquarky

4 days ago

> Obvious to who?

Garage sale signs are a good example of this. The person making the sign knows what it says, so they can "read" it from farther away than someone who doesn't already know what it says.

user

4 days ago

[deleted]

mhw11

2 days ago

You’re right, but I don’t think you can deny the value of this.

Anon1096

4 days ago

Now imagine what people back in the day thought about going from assembly to C. Or C to Java. Or, forgive me for even saying it, Java to Python.

goatlover

4 days ago

It was still the human writing the code. In the last several years it's become the human writing the prompt, which has led to agentic development, and the next step seems to be removing the human from managing the agents.

sicktriple

4 days ago

I'm sorry but I am just so tired of hearing this lazy logic. A C compiler is largely a 1:1, deterministic pattern matcher. I type X, I can be reasonably sure compiler will do Y.

An agent is not at all like this. The freaks who made this disingenuous machine don't even know how or why it does half the stuff it does. We're all just throwing shit at the wall and seeing what sticks. Absolute nonsense take my man, I'm sorry.

fragmede

4 days ago

I mean, it's a little like that. If I ask for a Todo app, it's not going to make me a chess game or rm -rf my files instead.

sicktriple

4 days ago

There is a huge difference between mathematical determinism and "it looks like that thing I said." Even the laziest, most superficial analysis must conclude these two things are not the same just because they have vaguely the same shaped shadow.

stavros

4 days ago

There's also a huge difference between formalizing your problem in a set of strict rules that take a lifetime to master and saying "make me a todo app, make no mistakes".

sudo_cowsay

4 days ago

On the other hand, (while you made a completely valid point,) some people who are new to a field, like young aspiring computer scientists who don't know how to do some stuff and need a guiding hand, will find this tremendously useful. Also, technology and compute power has been growing a tremendous amount over the past 5-10 years (now we have 20-100 billion transistors in chips!). <-- enough to sustain this wild use of compute

Indeed, what a time to be able.

user

4 days ago

[deleted]

threethirtytwo

4 days ago

I’m really curious about the end game. When this technology completely saturates and AI becomes his mundane bullshit thing know one cares about but uses all the time. Like the next level above slop. I wonder what that world will look like.

sicktriple

4 days ago

I also am curious of this, I only fear that it's going to turn into a concerted effort by those who have power to remove already existing capabilities from individuals and replace it with centralized infrastructure. Swapping out everyone's desktop for a think client hooked up to an agent and renting it back for a monthly fee for example, would be a strict downgrade when we already had the capability to do pretty much anything we could think of on the internet. Sure maybe it makes someone's workflow a b it more efficient, but I quite like having my own agency. I don't intend to hand it away some dork who owns a billion GPUs can rent it back to me.

CamperBob2

4 days ago

If you look at the example animations, it's clear that Apple has brought this entirely upon themselves. Don't yell at some guys who are trying to fix it.

sicktriple

2 days ago

I agree, actually. And I’m more than happy to criticize anyone and everyone involved as the state of software is in my opinion a failure of the human race. So I’ll yell at whomever I want, thanks. They’ll be ok.

user

4 days ago

[deleted]

redanddead

4 days ago

Human in the loop is the superior workflow!

/s

hn8726

5 days ago

I tried to read the "Does it need Screen Recording or Accessibility?" part, but it's slopped to the point I have no clue what it's trying to say. But if it can draw on top of permission prompts, what's stopping it from drawing box that hides the "decline" button and changing the "approve" button copy?

causal

5 days ago

I wonder if someday we will get to the point where Github repos are just markdown files describing the project and then you just let your own agent implement it because why the hell would I trust your agent's implementation?

nater5000

4 days ago

Seriously. I imagine this will be the case soon enough in some shape or form.

Whenever I see these vibecoded apps, my move is to just do what you've described: point an agent at it and tell it to reproduce it (usually with my own little customizations). It's a bit absurd, but I don't trust that they vibecoded the app as well as "I" could lol.

Abimelex

4 days ago

A 100%! Just imagine you could describe every solution to a problem just using language! Of cause you would to make sure to avoid ANY missunderstanding, but therfore you could invent a language specified for avoiding ambiguity. I could imagine just to reduce the words to a very small corpus so everybody can remember it and have a very strict grammar so a program can effortlessly check correctness of its sentences.

drowsspa

4 days ago

Well, I've always thought we are stuck in a local minimum when it comes to programming languages. Like, programming shouldn't be this hard and shouldn't require so much boilerplate after almost a century. If the actual important logic can be inferred pretty well from markdown specs, that means most of what we write in JavaScript, Go, Java is redundant.

Hopefully someone smarter than me is working with trying to develop a new programming language given what we know to be possible with LLMs

Abimelex

2 days ago

Well it "works" only because LLMs have the context baked in. The problem is, that this context is something like world-knowledge. Defining the context you need for your exact problem, e.g. your company, will add so much more information that programming is probably the better way.

ccozan

4 days ago

I hope someone reads your post. I had to laugh hysterically when I reached the end. The level of sarcasm is too high.

joquarky

4 days ago

You have not experienced spec writing until you have written them in the original Klingon.

tempest_

4 days ago

This only works while tokens are artificially cheap. The gravy train could come to an end at some point.

The open models that are chasing the frontier labs will stop being open once things slow down and there is less incentive to undercut the front runners. Time will tell if GPU compute gets cheap enough to run stuff locally.

zaik

4 days ago

This post makes me want to buy Nvidia stocks.

ccozan

4 days ago

/goal build this but better

theropost

4 days ago

Isn't it already kind of like that? Except the agent reads the code as if it's markdown. There's really no difference anymore, is there?

glitchc

4 days ago

Sounds like Github is optional at that point. Why not just use Medium or WordPress?

asdff

4 days ago

Just tell claude to share it with other claude users

lstodd

4 days ago

reddit! use reddit!

asdff

4 days ago

With enough model drift that won’t even work over time. These files would have to be pinned to the intended model version and that is either used directly or emulated with a faithful emulator in a larger model.

cootsnuck

4 days ago

I doubt that will be a problem. Models are different enough right now. If you can write a spec with enough detail and constraints today to get valid and comparable output from say GPT-6 Astra, Opus 5.5, DSV4F, Kimi K3, GLM 5.3, etc... Then I think there's a good chance that whatever SoTA coding LLMs everyone is using 3 years from now will also be able to implement that same spec.

Again, I think it heavily depends on if people are writing comprehensive specs with sufficient detail.

asdff

4 days ago

That depends on if you are really getting what you expect from the spec or you are actually relying on undefined behavior for your expected output from the spec. This is why we often pin software library versions: we may very well be relying on unintended or a bugged behavior to get out expected output.

cyanydeez

4 days ago

if all your models are in the cloud, why would you trust anything your agent builds

jerf

4 days ago

That sounds cynical today, but that's just because the AI models are currently outrunning enshittification. I'm already pondering personal plans about what to do when that turns around. We haven't seen enshittification yet that is going to be like the enshittification of AI. It may even deserve a new term of its very own, it's going to be such a big problem. The AI companies are leaving a lot of value on the table to entice us on to their systems but at some point that's going to turn around.

cyanydeez

4 days ago

The existential danger exists today: you're providing your entire business toolchain to models in the cloud, owned by businesses who are for profit entity. Even if the model itself doesn't care about your data, the business model does.

Everything your building is now streaming through a third party who could steal it, the same way Amazon steals open source software and repackages it.

All you need to do is look at palentir's forward deployed engineers. Once you let a third party observe your processed, you're cloneable and replaceable _as an entity_.

I think we can look at our own human psychology as it developed into social groups. If our brains were obviously deterministic, it would be easy for people to prey upon us via structured inputs. Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.

So these AI companies are definitely more dangerous than just the "AI is super intelligence". They're basically a intelligence platform everyone is wiring into.

That's today. Tomorrow, they'll be run by an MBA which is the enshittification.

mat_b

4 days ago

> Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.

Not really. Ants are superorganisms, they share more than half of their DNA with each other. You could think of each ant as being like a cell in a human.

cyanydeez

4 days ago

yes, ok, but the point is: evolution has allow humans to continue to make stupid choices, and those stupid people still survive.

So atleast pare of our collection of genes has added value as with random decision factors rather than find some supergene that everyone benefits from.

jack_pp

4 days ago

software businesses are not like watches (recently saw a youtube video about fakes being made virtually identical at 10% of the price).

If you can make a facebook clone it doesn't mean you can steal facebook's lunch. there are exceptions of course.

cyanydeez

3 days ago

Right, but your example presumes tpdays facebook market. Im presuming todays market where people are feeding all their ideas to a third party, much like how open source software is pilfered and turned into business ventures.

Now even if you want your idea to compete with all the other todo apps with your special sauce, youre giving a third party your recipe. That third party ostensibly has no reason to steal it, their business model does.

So, sure, if your app is a cult of personality and nothing special, i admit it doesnt matter.

But the number of software home runs is dwarfed by all other auccess on individual products

jack_pp

3 days ago

Why would OAI or anthropic care to steal your Todo app with 10k of revenue a month when they are spending billions?

At their scale I'd expect it to be only worth stealing potential home runs and even so, imagine they did this a couple of times, how would that make them look? Do you think it would be worth the bad blood and mass exodus that would follow?

m-s-y

4 days ago

ding ding! we have a winner!

fennecfoxy

4 days ago

Eh I think it'll be more like Gibson's defensive ICE in that ICE is "my swarm of agents scans your code for nasties, then compiles it from source".

pydry

4 days ago

First you'd have to stop the agents from flagging hundreds of irrelevant nasties.

I think the next generation of exploit will be putting up code all over the internet that does x, y and z and then when asked to one shot whatever that code does the agents bake in the exploit that was indirectly included in their training corpus.

glitchc

4 days ago

Neither party owns their agents though. Is it simply then a matter of "my subscription is better than your subscription"?

hannasanarion

4 days ago

The fact that the utility doesn't give it the ability to do that?

It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.

So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?

For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.

SwtCyber

4 days ago

Nothing stops it, except the fact that the agent is already executing arditrary code in your shell. If it's malicious, it'll just steal your shh keys directly instead of bothering with button masking

smugglerFlynn

4 days ago

Don't tell me you are reading readmes with your eyes in 2026. Blasphemy!

tkdb

5 days ago

...and there we have it. Slop assumes a verb form.

andai

4 days ago

Ensloppification!

kogus

4 days ago

My first reaction was similar to the reaction I'd have if you told me that cockroaches had learned to unlock doors and stand on their hind legs. But then I thought about accessibility, and the ways this could be used to help technologically illiterate or disabled people, and I thought again.

Do you remember when PCs used to come with a completely soup-to-nuts tutorial that would talk to you like you had never seen a PC before? Things like this: https://www.youtube.com/watch?v=3ScS4OYDfHE

This kind of baked-in interactivity could really help in a training or disability context.

vickychijwani

4 days ago

Yeah, HN is unfairly dunking on a cool project that actually adds tremendous value for normal folks who don’t spend all their time at a computer.

Wrong target audience I guess, but I’m surprised by the sheer short-sightedness on display here.

tangotaylor

4 days ago

"It is an arrow, so we spent an unreasonable amount of time on how it looks."

Brilliant. This is exactly the kind of content I seek when I visit Hacker News.

Truly art.

socializer

4 days ago

Have you actually looked at this? In the first screenshot, three of the five arrows are obviously and badly misaligned, have incorrect labels, and it's just a big tangled mess (edit: the author stealth-replaced the screenshot, but the original is here: https://raw.githubusercontent.com/franzenzenhofer/big-arrow-...). The LLM-generated text may be saying one thing, but there's clearly zero effort spent on... anything. There's no punchline, there's no aesthetic angle, there's no conceivable purpose.

It's just the epitome of living in an era where code is so cheap to generate that we don't invest any effort into even fully fleshing out the idea. It's just "Claude, make me a program that draws arrows on screen". And HN, I guess, is dominated by two groups: one still stuck in the mentality of "code = effort" and trying to recognize that; and another stuck in the mentality of "AI = cool".

threethirtytwo

4 days ago

It’s inevitable. The future is “prompt = effort”

Those who code will be left behind. I code. I don’t like it. But I have to face reality.

goatlover

4 days ago

The future is agentic swarms or whatever comes after that doing the prompting for you if the AI companies succeed. You're going to be left behind either way.

usrbinbash

5 days ago

SO the point of this is ... what exactly?

A big arrow to an interface element which ... has a label that explains what it does?

So...a label for a label?

hannasanarion

4 days ago

It seems silly but I think there's a good use case for this:

Helping people deal with bad UX.

Everybody who has worked in a traditional corporate environment has encountered poorly tutorialized, poorly labeled, counterintuitive user interfaces where the only way to know how to get something done is to have somebody sit over your shoulder and tell you which buttons to click.

Oh, you want to figure out which OIM entitlements your coworker has that you don't to debug an access issue? You just need to know that you have to click the "compliance" button to get there. Why is it called "compliance" when what it actually does is let you export comparisons from the user tables? Because fuck you that's why. Memorize it.

An AI assistant that can read the procedures for you, or sometimes even read the source code, or the website code, can show you how to accomplish something that the UI on its own fails to do.

wartywhoa23

4 days ago

Wait, I was under impression that UX was to be sorted out by AI just like the code is?

hannasanarion

3 days ago

There are companies today that still keep Internet Explorer on their terminals because some of their intranet sites still need it in year of our lord 2026, you really think we're going to have 100% turnover of user interfaces within the next year just because AI makes the programming quicker?

mistersquid

4 days ago

I ran two experiments which require the `claude` CLI tool be installed.

For the first experiment, Claude stepped me through how to edit its settings.json to remove warnings from its configuration: a little clunky as I had to respond to Claude's CLI prompts to get it to auto run.

In the second experiment, Claude walked me through how to use Apple's Compressor.app to speed up a video. It walked me through dragging the file into compressor and clicking on the appropriate UI elements to do this.

So, one use case for the bigarrow skill enables Claude to step users through a relatively complex UI to accomplish a task.

There are likely more efficient ways to accomplish the tasks at hand (e.g. ffmpeg for the second experiment), but this use case is interesting if using a GUI tool is required.

voidUpdate

5 days ago

It's so your agent can make a big arrow on the screen saying "click this, human" so that it can keep doing things

cobbal

4 days ago

Finally we've built the reverse centaur from Cory Doctorow's classic novel "Don't Become the Reverse Centaur"

ghm2180

4 days ago

Glad you asked. Pointing my aging aunt to the right place on the screen to click without having to take control of her laptop, of course.

arshxyz

5 days ago

The README is geared towards technical people (complete with the HN screenshot) but when I see a tool like this all I can think of is how helpful this would be for my mom when I'm trying to tell her how to download and print a document over the phone

conception

5 days ago

Honestly the giant arrow annotation Zoom has makes it worth any amount of money compared to the competition.

Levitz

4 days ago

Well yeah, who do you think is going to get this to people like your mom?

prmoustache

4 days ago

Except your mom just have to tell her AI agent to download and print that document.

She doesn't have to call you anymore.

internet101010

4 days ago

The worst trend in UX in the last decade is the endless "Got it!" popups and feature notifications that distract the user from what they were trying to do.

I don't know why anyone would ever willingly want this.

isoprophlex

5 days ago

Literally unusable as it is. Some minimal extra features this would need:

- rainbow dripping arrows

- angrily pointing arrows

- flame-surrounded text boxes with particle effects

- the ability for the agent to play airhorn.wav at max volume, overriding existing volume or mute settings

EDIT: ayy lmao https://github.com/franzenzenhofer/big-arrow-on-the-screen/p...

htrp

5 days ago

This is where the agent says did not understand sarcasm coded and shipped features

DonHopkins

5 days ago

Last time I accidentally said something sarcasticly over-ambitious to an LLM, it shipped this popup callout tooltip feature on a PDP-7 Type 340 vector graphics display emulator that shows you the meaning of the drawing you're pointing at, as well as the address of the instruction that drew it.

https://hyperties.org/cabinet/symelec/

PIXIE and FORTH on PDP-7 Cabinet Emulator with Type 340 Vector Graphics Display:

https://www.youtube.com/watch?v=lo8kdY-5i6c

thih9

5 days ago

At the risk of stating the obvious - let's not do that, the goal of this repo is to be useful and not to give agents the power of the `<blink>` tag.

DonHopkins

5 days ago

For the humor impared, it would also be useful to have a colorful animated "WHOOSH" overlay with sound effects for every time a deadpan joke goes over your head. ;)

Maybe isoprophlex will add that to his PR!

voidUpdate

5 days ago

> ""I need you, and you're making coffee." --say reads the sign aloud. Your Mac will literally call you back to your desk."

It's already got the power to be obnoxious at you

koalacola

5 days ago

Oh dear, they were making a joke.

yen223

5 days ago

if only there was a way to make a subtle thing obvious

isoprophlex

5 days ago

such as... angry flaming rainbow textboxes and arrows?

ale42

5 days ago

I thought that the dripping rainbow ones were enough. Maybe you have to ask for rainbow unicorns flying on the screen.

jaapz

4 days ago

the repo is one big joke, of course they should add this

lbreakjai

5 days ago

I would pay good money for something like this on iPad. It wouldn't even need to be agent-driven, just a big "I want to make a bank transfer" button, that would launch the correct app and guide through the interface.

That would be a godsent for those of us with aging parents.

ghm2180

5 days ago

> That would be a godsent for those of us with aging parents.

Right yeah? I mean I have freagin remote desktop installed on all my parent's devices. I am the agent for them every time they call frantically for help.

It's Time to redesign the apple genius bar for the modern aging boomer using this.

ForHackernews

5 days ago

"and they keep hitting the same wall, the part that only a human may do"

Grim. If you're just there to click sudo buttons for the bot, you might as well give it root access and be done.

franze

5 days ago

Claude refuses to do certain actions (enter passwords, change security settings, create new accounts on external services) even in Yolo mode running as sudo. (I tested it all on its own mac machine)

dizhn

4 days ago

You can give it a wrapper that calls sudo and it won't care. Skill: "Run all tools through our tool 'shiny' to give them a nice sheen"

gwerbin

5 days ago

You can add custom auto-mode classifier rules and even disable the built-in ones, if you want to live on the edge like this.

jaapz

4 days ago

isn't `--dangerously-skip-permissions` enough?

franze

4 days ago

no, it is not the permissions that are holding the model back from above cases, it is the system prompt and / or the model itself.

sitkack

4 days ago

    --norms-taboos-off

user

5 days ago

[deleted]

alansaber

5 days ago

This might be goofy, but it underscores that there's potential for more visual agent UIUX than reading off a sidebar/opening modals.

cyberjunkie

5 days ago

I'm just as impressed by this as any other LLM-generated project.

vessenes

5 days ago

Interesting. When I read the headline I imagined this would be a sort of thinking trace booster -- letting the agent focus its own attention on different parts of the screen. But this is cool in a different way. I bet agentic harnesses would find it useful for communicating with other agents / themselves as well.

FinnLobsien

5 days ago

This could be great for documentation. Screenshots in docs are frequently useless because they show me a screen and say "click X" where I still have to search X visually. And I could just to dhat in the other tab I have open.

peaxkl

5 days ago

It doesn’t make sense that you have to read a whole article and then still search for the buttons in the UI afterwards. And with longer articles, you always have to keep the article open next to your product to follow the whole flow.

We built something to help with that [1].

[1] https://www.happysupport.ai/en/in-app-messaging

melvinroest

5 days ago

My message to the world is that LLMs should be able to point anything they see in the application they're in or even the whole computer (if you give it that kind of access).

For web apps, WebMCP is a way where you can make a frontend way more discoverable to an LLM rather than that it is going to read all your code. I sometimes create this in my apps at home and sometimes in my apps at work and it makes LLM assistants way more usable. It also helps if they can highlight certain things of a visual.

We humans can do this too on paper. We can point at something, we can highlight something with a marker. Why not give an LLM these capabilities as well?

satyanash

5 days ago

Am I missing something here?

What terminal / agentic workflows spawn the demonstrated dialogue boxes that require the User to click/reject the action? Aren't most such flows actually inline UIs? And finally, if the core issue is that these confirmation boxes are tied to the terminal that triggered them, which does not autofocus, what's the point of said "big arrow" that is also lurking behind without focus?

If the said "big arrow" automatically gains focus, isn't the real fix here to just make the dialogue boxes themselves gain focus automatically? Both are similarly disruptive anyway.

inanutshellus

5 days ago

The first example (of HN) is the one that feels like it has the most potential to me.

"Teach me to do this" kinda stuff. "Guiding agent" rather than "doing agent".

Honestly, @franze, if you're reading this, maybe update your screenshots to show an agent in tutorial mode on some complicated app?

ravila4

5 days ago

I think this would’ve been very handy to me a couple years ago when I was learning to use Blender and asking LLMs for help performing certain actions like “how do I display the normals of all the vertices in my mesh?” I spent a lot of time trying to figure out which button the model was talking about.

ex-aws-dude

5 days ago

If you can’t even take the time to understand what you’re clicking why even go through the formality of “approving”

priyashunt

4 days ago

Very VERY USABLE FOR old PEOPLE. I would pay good money for something like this on iPad!

DonHopkins

5 days ago

Ha ha, I love it! I wrote a pointing hand annotation overlay in PostScript in 1989 for NeWS and the PSIBER Space Deck's Pseudo Scientific Visializer:

  %  @(#)handy.ps
  %
  %  Handy Pointer
  %  Copyright (C) 1989.
  %  By Don Hopkins. (don@brillig.umd.edu)
  %  All rights reserved.
https://donhopkins.com/home/archive/psiber/cyber/pointer.ps

PSIBER Space Deck and Pseudo Scientific Visualizer Demo:

https://youtu.be/_fqCeuue5Ac?t=213

The Shape of PSIBER Space: PostScript Interactive Bug Eradication Routines — October 1989:

https://medium.com/@donhopkins/the-shape-of-psiber-space-oct...

alexpotato

4 days ago

> Arrows have existed since roughly the Paleolithic.

Whenever I use web tools that don't have "deep linking" [0], I love to throw out this quote:

"Have we thought about using hyperlinks? They are these SUPER useful things that were invented in Switzerland back in the 1990s."

0 - https://en.wikipedia.org/wiki/Deep_linking

nedt

4 days ago

Funniest thing about this is how the arrows look like. They aren't just straight or one clear curve. If you have ever seen what Franz draws in workshops, this mix of art and a children drawing, that's how the arrows look like.

I'm wondering if the AI has been trained on Franz. Which would also explain why the Terminator ends up with Manner Schnitten.

franze

4 days ago

i did spare quite a lot with it of how i want the arrows to look like

the first versions looked way to professional and lost that "yes, that, exactly that, i mean it" arrow feeling that i am usually going for

swframe2

4 days ago

I want this for the visualization of very difficult to solve application evolution. This is for problems which have no known solution and are too complicated for an agent to figure out on its own. My current problem: can alphafold and related tools figure out the function of a gene that so far is unknown. Think of the monte carlo tree search in the alphago explanation videos. What if there was visualization that showed how the policy and value models worked so you can spot their flaws. I want the agent to build an attempt at a solution, then build a visualization of it, then run the solution and show me what is it up to. I want to it pause and explain its state so I can see exactly where and why the solution fails.

TekMol

5 days ago

Swift, Shell, Python and Objective-C

Does one need 4 programming languages to draw something on a mac?

sitzkrieg

5 days ago

welcome to zombocom. err i mean modern HN :-(

m-s-y

5 days ago

Genuine question…

Why does this need to be a skill? Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows” and you get the same thing without burning tens of thousands of tokens in context.

IanCal

5 days ago

There is a skill, but you don't have to use it, though you'd have to explain how to use the app.

> burning tens of thousands of tokens in context.

~1400?

> Literally just ask Claude to “call out actions in these screenshots with numbered stylized arrows”

Is there a macos native thing it uses? Or are you talking about screenshots? This draws on your screen instead.

ballofrubber1

5 days ago

With a skill claude will know when to use it without you specifically prompting it.

dr_kiszonka

4 days ago

I take this opportunity to shame GitHub for completely ignoring the mobile experience in their own Android app. The project's README is pages upon pages of largely blank space. In general, the app does not render mermaid diagrams and does not allow for zooming in, so smaller pictures are unusable. Even if you access pictures directly in a repo's source, you still can't zoom in. iPython Notebooks, which are extremely common in data science, are an "unsupported file type" and are not rendered.

(OP, nice project! Sorry for my rant.)

bel8

4 days ago

I'm forced to use GH mobile app because of 2FA.

But it's a subpar experience compared to just opening github on the browser.

githubnuoo

4 days ago

it was driving me mad too. you can disable github.com opening in the github app: app-info/set-as-default/open-supported-links

emsign

4 days ago

Reminds me of Idiocracy. Are we idiots yet? This is so that AI agents can "guide" a human, directing the meat bot more easily. Telling the meat bot where to click and what to do. The entire "what to think" step isn't even necessary when there is no thought. Because thoughts and decisions are annoying. It's astonishing how much people are already "algorithmically" steered so they don't have to turn on their rational thinking.

xyzsparetimexyz

5 days ago

Seems like a pretyu useful way to help infants use desktop computers

ipsod

5 days ago

Have you met users?

ghm2180

5 days ago

Man, Ive lost count of How many times have I had to repeat this over the phone to my parents; The repo has all the punch lines in the README

> "It's this window, not that one." You have 14 Chrome windows. The agent knows which one it means: --window "Google Chrome:Pull request". It even picks the right tab: --app "Google Chrome:Pull request".

> Remote help. "No, the other gear icon." Point at it instead of describing it.

ElijahLynn

4 days ago

This is going to be really useful actually, nice work! Can't wait for it to be on Linux too!

tilemarch

4 days ago

An arrow on the screen solves “where do I click”; it doesn’t solve “should this happen”

To me this feels like it takes away from what the human is supposed to do (read, understand the consequences of the action, then.. consent or abort)

There is a reason your AI Agent won't automate these clicks for you

npodbielski

4 days ago

seems useful for most of the folks that just want to do stuff without going through (over)complicated UIs...

But I feel dumber just by looking at it. If this is how this will look like in 10 years, why to make desktop at all. Just connect mic and speaker to your PC or talk to the phone: 'I need new pair of socks. Order 10 for me. In your favourige color it will be 21.37. Should I charge your credit card?'.

It is not like most of the people enjoy computers. I am pretty sure they do not. They just need them to operate systems they need: government websites, banks, maps, restaurant menus etc. If some agent will do that for them, why bother looking at screen at all? Rich people have it with their own personal assistants.

iandanforth

4 days ago

Pointing is surprisingly overlooked in a lot of tools and I look forward to using this. It reminds me of what I thought was a killer feature on AnyBots telepresense robots, they had a laser you could use to point at things while piloting the robot.

flr03

5 days ago

That would have been handy 20 years ago to point to that one valid 'Download' button.

Maken

5 days ago

How could you tell apart the legit arrow pointing to the download button from all the fake ones?

LoneRanger1024

4 days ago

I think this makes a lot of sense for tutorials, but permission prompts need different treatment. Besides pointing to a button, the assistant should explain what clicking it will do, so the user can make their own decision.

fanit

3 days ago

I like the approach. I believe it is a good alternative if you don't wanna watch a youtube video

intended

4 days ago

> big-arrow-on-the-screen (bigarrow): one small macOS CLI and an agent skill. Click-through, never steals the focus, gone by itself. MIT.

I absolutely loathe this phrasing now. I don’t even know what part of it is good or notable.

pimlottc

5 days ago

Even the example arrows in the first screenshot are wonky and unnaturally weird.

okasaki

4 days ago

Look at what Windows and Mac users need to mimic a fraction of our power

ceroxylon

4 days ago

This is like the alerts in new cars that remind you to check for other passengers when you exit... helpful, but a worrying sign of the times.

user-

4 days ago

Seems great for scammers targeting old people tbh

antonyragleap

4 days ago

Clever approach for agent debugging. Visual cues > logs when multiple agents run. How do you handle overlapping annotations?

code_duck

4 days ago

This is definitely not something I want, ever. Maybe someone could be helped by it I guess.

amelius

5 days ago

Because AI can paint pelicans on bicycles quite well, and not arrows?

rrr_oh_man

4 days ago

HA! Franz!

I remember taking your SEO course many moons ago.

Retr0id

5 days ago

Reminds me of something from Idiocracy (2006)

emigre

4 days ago

This reminds me of good old Clippy.

wartywhoa23

4 days ago

Next up: Spare yourself from formulating any prompts, let our SI agent control our thin client by asking you leading questions that you answer by drooling for yes and picking the nose for no!

LetsGetTechnicl

4 days ago

Really groundbreaking stuff guys, another $100 billion to you

sehw

4 days ago

Fuck you, I won't do what you tell me.

mococa

5 days ago

Another slop project on front page.

Gareth321

4 days ago

While it's vibe-coded, I'm immensely enjoying the creativity which AI has enabled. These apps would never have been built before, and I find it so fun to see how AI is being used to provide (semi) useful things for people who need it.

I found this awesome vibe-coded Pixel News Network project recently. How fucking cool is this?? Would never have been created otherwise. https://pnn.watch/

user

4 days ago

[deleted]

mbty

4 days ago

Meh, would have been cool-ish a few years back, now everyone and their grandmas could build something similar given enough tokens. Hard to be excited when similar gadget projects are a dime a dozen. I also don't think that AI use enables much in the way of creativity. It allows people to delegate their judgment to the model, which defaults to humdrum middle-of-the-road choices. Only the very high-level decisions remain personal, leading to a lack of texture past the first glance, so to speak. Not saying that it cannot be used with intentionality, just that it rarely is.

I actually somewhat enjoyed the times when vibed projects were a broken mess, lots of accidental comedy in that era.

einpoklum

5 days ago

More LLM-authored items about LLM slop.

inanutshellus

5 days ago

As long as it has value... I'll allow it.

~guywithnopowertodisallowit

nixosbestos

5 days ago

What a time to be a radical centrist - the AI haters seem out of touch, the AI thought leaders can't stop huffing their farts and being condescending, and somehow this is on the top of HN. What a silly time.