The tilde in your PATH may not be your HOME

182 pointsposted 8 hours ago
by arusekk
(disconnect3d.pl)

112 Comments

amarshall

6 hours ago

Or you can just…not quote the tilde. Folks always seem to reflexively quote “strings” in Bash while not realizing that (almost) everything is a string and most strings are not quoted and it would be odd to do it (e.g. no one is doing `"ls" "-a" "foo"`).

isityettime

3 minutes ago

This feature, or at least the syntactic unit in question has a name here "bare words". You also have these in some contexts in Perl and Ruby, YAML, and probably some other languages, idk.

I've also seen this even with people who seem like generally competent shell users. Idrgi

Cockbrand

5 hours ago

I feel like this is common knowledge and should not be worth mentioning. But then it apparently is not common knowledge, as the article proves.

Have I run into this at some point?

I certainly have.

Have I learned to quote better and only where appropriate from it?

I certainly have.

Bourne compatible shells take a while to learn and require some experience. This won't change, but alternatives exist, with their own caveats.

dylan604

6 hours ago

Unless you're running a shell command from python. That was the first time I saw a command string broken down into "string" arguments for every thing like that.

thwarted

6 hours ago

In that case, you're not running a "shell command" from python, you're passing arguments to exec. A shell command would be a string interpreted by the shell, and you'd use that for shell syntax things like having the shell do variable interpolation or redirections as part of executing the command.

vips7L

6 hours ago

Or just stop using bash. It’s a terrible language to write and has tons of footguns.

yjftsjthsd-h

4 hours ago

For the things shell is good at (running other commands, pipes, and shuffling files), I have yet to find anything even close to as good.

vips7L

3 hours ago

A different shell??? Nushell, fish, powershell, etc.

yjftsjthsd-h

3 hours ago

Oh, you mean bash specifically. Then yeah, that's reasonable enough. Although I agree with the other commenter that sh/bash have the advantage of ubiquity.

lelandbatey

3 hours ago

None of those have the #1 best thing bash is good at: bash is already installed, nushell and fish are not.

Powershell is, as I understand it, not available on Linux but is omnipresent on Windows, so hopefully the windows folks can use it like they would bash.

barrkel

2 hours ago

PowerShell works on Windows because of WMI, to a first approximation.

Because Windows is tied together through APIs and databases and not text files (unlike Linux), everything is text is not as useful as something that can talk objects. And a lot of Windows is object oriented, from the window message dispatching system through COM and down to NT's object manager.

yjftsjthsd-h

3 hours ago

No, powershell runs on Linux, it just sucks because it wants objects and everything on *nix speaks streams of text.

hnlmorg

3 hours ago

I agree that shells are uniquely good for that but there are plenty of better shells than Bash.

Such as Fish, nushell, Elvish, or the project I help maintain, “murex”

bornfreddy

3 hours ago

None that I can rely on being available wherever I see a terminal. It's bash or sh as far as I'm concerned.

xigoi

2 hours ago

Fortunately, 99.9% of the time I’m on my computer, where I can install whatever I want.

ifwinterco

an hour ago

Then you ssh into an instance and have to remember how bash/sh work

gavmor

38 minutes ago

At what point does an alternate shell get baked into the image?

shevy-java

3 hours ago

I did.

Ruby replaced all my shell needs. Almost 25 years ago. I even have a shell written in ruby (it handles both bash-like behaviour as well as ruby code as-is); admittedly it is not quite perfect for everything, but I improve on it steadily. And it works on Windows too, which was one reason I wrote it in the first place (need to have it work via cmd.exe as-is).

Never looked back to shell. It is too awful to use.

CodesInChaos

6 hours ago

Unfortunately half of its badness isn't isn't the shell itself, but the convention of how parameters are passed to processes on Unix systems.

hnlmorg

4 hours ago

That’s not correct because POSIX passes what is ostensibly an array of strings.

Windows, on the other hand, only passes one string. So it’s up to the application to choose how to handle whitespace, quotation marks, and other nuances with parsing parameters.

Variable expansion in Bash is lazy. But there’s no reason why variables cannot be tokenised so that strings with spaces aren’t treated as multiple parameters. And in fact that’s exactly how some other shells work, such as the one I maintain.

formerly_proven

5 hours ago

Array of arguments is vastly superior and more secure than every program/runtime inventing a slightly different way of splitting a command string into an array of arguments. No debate. A real problem is the related birth defect in ssh2.

akoboldfrying

5 hours ago

Well, the only other way I can think of that it could be done is the Windows way, whereby you pass the unparsed command line, spaces and all, as a string to the new process. And while this is arguably the cleaner interface, in practice it has meant even worse quote handling, since how -- or even whether -- double quotes are parsed now depends on the probably undocumented process startup code chosen by the program's compiler vendor.

Want to quote a command line that may already contain double quotes, in order to pass it as an argument to some other program? No, you don't. It isn't right to want that.

ChrisSD

3 hours ago

The other other way would be more structured. Arguments are not just an array, they also contain `--switch` and key/value pairs (e.g. `--key value` or `--key=value` depending on who you ask). One problem with the flat array approach is there's no way to distinguish a literal value starting with `-` from a switch, which means there needs to be some way to workaround that (and users have to remember the workaround; how often do people remember to use `rm -- "$FILENAME"`).

In practice almost every application does still need to do its own parameter parsing; a flat array is not enough.

sysguest

4 hours ago

yeah but... that convention is so much of a security/bug headache

sometimes, its footguns seem worse than javascript...

hope some typescript-like "typed shell" becomes mainstream someday

coldpie

4 hours ago

> hope some typescript-like "typed shell" becomes mainstream someday

The trouble with trying to invent a new, more robust shell language is you basically just end up re-inventing any number of scripting languages (eg Perl, Python, awk, ...), so you might as well use one of those.

hnlmorg

4 hours ago

Most regular programming languages aren’t well suited for shells because they have a verbose syntax due to their readability goals. But with a shell, the vast majority of times you’re typing in stuff that you have no intention of reading back ever again.

I’ve done a fair amount of research here and I actually think we do need a new programming language for the shell (and then I created one).

I wrote a blog about this problem: https://murex.rocks/blog/split_personalities.html#conclusion

coldpie

3 hours ago

Excellent article, thanks for the link. I do agree with the premise. But every time I write some Bash code and think there should be a better way to do this, I ask myself why I don't just learn Perl. I dunno. Feels like inventing another solution would just hit the old "there are now 14 standards" problem where it would solve some problems but introduce others (see: PowerShell).

hnlmorg

3 hours ago

I don’t disagree with you per se. But if the new shell solves enough annoying problems then I think it has merit in existing.

malux85

4 hours ago

The widwit answer: "Dont do X" (and nothing more)

The enlightened answer "You should use A, B or C for these reasons"

Even though bash is installed on many systems, I try to encourage people to use better designed shells that have less of these footguns :

Fish (https://fishshell.com/): No implicit word splitting : spaces in variables won't unexpectedly become separate arguments.

Zsh (I use this: https://ohmyz.sh/): Arrays start at 1 by default, but crucially, unquoted variables don't implicitly split into multiple arguments.

Nushell (https://www.nushell.sh/): Passes structured tables and records between commands : avoids fragile parsing of text with awk/grep.

paulddraper

6 hours ago

People quote both too often and too little.

GENERAL RULE

1. Double-quote dollar sign expressions, and nothing else.

  foo

  "$bar"/foo

  baz:"$(cat example.txt)"

  exec cmd "$@"
2. Single-quote words with a literal special character, and nothing else.

  'Die Hard'

  'ke$ha'
---

I should point out that the author's example is NOT fixed by different quoting though.

  # original
  export PATH="$PATH:~/.local/bin/"

  # without unnecessary quotes
  export PATH="$PATH":~/.local/bin/
Because tilde expansion only happens at the beginning of the word.

gray_-_wolf

an hour ago

> I should point out that the author's example is NOT fixed by different quoting though.

Sure, but I would say you almost always want to add to the start of the PATH, not to the end.

JdeBP

5 hours ago

The Z shell's, C shell's, and others's syntaxes for setting the PATH environment variable via a shell array variable alias also does the tilde expansion.

    path=( $path ~/bin )
It's worth noting, also, that the path and manpath settings in login.conf(5) expand leading tildes in individual search path items.

* https://man.freebsd.org/cgi/man.cgi?query=login.conf&sektion...

So putting the addition of things like ~/bin to PATH in /etc/login_conf and ~/.login_conf instead of shell scripts is another way to address it.

    :path=~/bin /usr/local/bin /usr/pkg/bin /usr/bin /bin:
It's particularly handy when there are multiple login shells in use.

* http://jdebp.uk./FGA/BSDs-for-Linux-users/login-conf.html

cr125rider

6 hours ago

The trick is to quote explicitly and correctly. “ and ‘ are different.

dylan604

6 hours ago

why would you use smart quotes in a terminal like that?

tom_

6 hours ago

They probably fell foul of some browser text box auto-correct.

dylan604

4 hours ago

' "

I don't use WYSIWYG editors anymore, so I have to remind myself to use those when writing raw HTML text. Although, most browsers correct raw ' and " symbols in text now, I still try to use them to have compliant HTML

airstrike

6 hours ago

why would anyone use smart quotes ever

mpyne

5 hours ago

They are typographically the right thing to have been using all along.

Using ' and " to pretend to be ‘/’ or “/” is on par with the typewriter days where people would use the l key to stand in for 1 also. A justifiable approximation when technology limitations prevented using the real deal, but an approximation all the same.

dmd

4 hours ago

In fact, if we had been using left and right quotes from the beginning in shells, most “quoting problems” go away, as they’re all inherently rooted in not being able to know what level of nesting a quote character is at.

airstrike

an hour ago

Can't that be handled by the typesetting/rendering software and displayed appropriately regardless of which specific character was typed?

Dylan16807

5 hours ago

Except l has a specific meaning that isn't 1, while the entire purpose of ' and " is quoting (and apostrophe).

The equivalent of l for 1 is doing font-specific pseudo smart quotes with ` and '

mpyne

3 hours ago

" and ' are themselves conjoined with other uses.

" can be inches or arcseconds from cartography.

' can be feet (of measure) or arcminutes from cartography.

These actually all have slightly different symbols, and the symbols for the left/right quotation marks are actually farthest from this approximation, even if they are the most frequent usage.

It’s always been wrong in some respect to use a single available symbol to emulate three or more different typographical tasks, but quotation marks may actually be the one where the emulation is the most wrong.

opello

3 hours ago

That equivalency only holds if there is a visual distinction in the output between the l and the 1. There often wasn't when this practice was popular.

thaumasiotes

4 minutes ago

As others have commented, for the same reason that we use smart parentheses instead of the more utilitarian |.

Grimeton

2 hours ago

You need to read the manual.

chasil

4 hours ago

I think the appropriate method by POSIX rules would be:

  export PATH="$PATH:"~/.local/bin
I may be wrong. If I'm not, that works in any POSIX-compliant shell.

Edit: this appears to work properly with mksh on my phone:

  :/ $ export PATH="$PATH:"~/.local/bin
  :/ $ print $PATH
  /product/bin:/apex/com.android.runtime/bin:/apex/com.android.art/bin:/system_ext/bin:/system/bin:/system/xbin:/odm/bin:/vendor/bin:/vendor/xbin:~/.local/bin

yencabulator

4 hours ago

That's a literal ~ in your path, the exact problem the blog post talks about.

Your use doesn't count as a "word", per man bash. Tilde is only expanded to home at the start of a typically whitespace-separated word, and your tilde is in the middle of one.

> If a word begins with an unquoted tilde character (‘~’), all of the characters up to the first unquoted slash (…) are considered a tilde-prefix. (…)

> word A sequence of characters considered as a single unit by the shell. Also known as a token.

chasil

2 hours ago

My initial syntax does work in dash (and bash), but all these shells appear to rely on expanding ~ inside a word, that your source asserts is not POSIX (which I do not contest).

Perhaps a succinct and compliant expression could be:

  export PATH="$PATH:$(printf %s ~/.local/bin)"
That comes at the cost of forking a subsell.

Edit: 2.6.1 Tilde Expansion in the POSIX standard says that ~ may be expanded "following any unquoted <colon>".

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

That being so, the most succinct and compliant version is my first variant, with the colon moved outside the quotes:

  export PATH="$PATH":~/.local/bin
Thank you for prompting me to look this up.

SoftTalker

5 hours ago

I never use tilde in scripts, only as a convenience when typing commands interactively.

$HOME otherwise, which still has gotchas but they are the same as any other environment variable.

ndegruchy

4 hours ago

Yeah, in scripts I try to be as explicit as possible. `$HOME` for the home directory, `$XDG_DATA_HOME` for `/home/foo/.local/share/`, etc.

The more specific you are, the less gotchas you're going to fall to.

nlehuen

22 minutes ago

I refuse to let knowledge about Calvinball-level grotesque rules about quoting and escaping in bash encumber my mind.

weinzierl

12 minutes ago

Good decision and avoid the shell whenever you can. Too late for me. I did my fair share of shell programming (and learned if from some real masters) so that my mind is already encumbered. My confidence in being able to write a longer or more complex shell script with 0 errors is still 0, so it gives me nothing.

FeepingCreature

7 hours ago

Kind of seems like you should have noticed this by your ~/.local/bin PATH not working?

thefilmore

2 hours ago

You can just do:

  PATH=~/.local/bin:$PATH
PATH is already exported. Quotes are also not necessary for assignments.

stkdump

2 hours ago

Looks dangerous to put a user writable directory ahead of system directories in PATH

bityard

20 minutes ago

Nah, it's totally normal because you often want to deliberately override or wrap system commands for various good and convenient reasons.

If your concern is that some hacker could put an illicit binary in ~/.local/bin, then your security problems are much deeper than your PATH order.

russellbeattie

an hour ago

While we're all getting pedantic and esoteric, in zsh you can do this:

path+=(~/.local/bin)

You don't even have to use the "export" keyword. Not that I would do this, but since we're all pointing out random shell stuff...

(In zsh the lowercase "path" variable is an array that's automatically tied to the $PATH string, so you can just add to it using parens to signify new array elements, separated by a space.)

alexpotato

5 hours ago

Not necessarily about tildes but about some of the craziness that can happen with bash at large orgs:

At a past job, I was trying to figure out what part of my basrhc was setting a particular environment variable. I assumed that it must be some kind of default installed in my user profile and/or inheriting from /etc/<something>.

I realized pretty quickly that my bashrc was importing some other files. Again, the assumption was that this would be one file deep in the import.

It turned out there were 10+ layers of import starting from an "ur-bashrc" and then layer upon layer of more and more imports to finally get to a user level profile.

It was so convoluted that I was going nuts until I found this Stack Exchange post: https://unix.stackexchange.com/questions/813/how-to-determin...

It turns on "tracing" for bash imports so that you can then narrow down on where the env variable is getting set.

dspillett

5 hours ago

I was going to say “it always seems to work for me” then I saw “… actually works in Bash and Zsh, because …”.

Another Bash-ism I need to be careful not to use when trying to be portable.

It is worth noting that on a lot of systems /bin/sh isn't bash (or zsh) so if you want to rely on Bashisms (or just can't be bothered looking for them) be specific and use “#!/bin/bash” for you hashbang. On Debian and similar it is usually dash for instance.

opello

3 hours ago

For various embedded environments that use busybox it's busybox's brand of ash. I generally enjoy the opportunity to learn when bumping up against these kind of edge cases.

the__alchemist

5 hours ago

I wish Linux distros would ship a "Terminal"/"CLI" program etc that is decoupled from the scripting language. Have a universal Path env var that isn't tied to a specific shell. Lets you execute cd commands, launch python/git/cargo/arbitrary applications etc, and have a good bookmark + autocomplete system. It feels like the conflation of scripting language + CLI application is the root of these complications and subtleties.

If you are using shell scripting (And prefer Bash etc over Python), you would keep using Bash/Fish/Zsh etc. If you are using the CLI to launch applications that don't have a GUI, navigate directories and perform file system operations, then you would use the plain terminal.

dicytea

5 hours ago

$PATH is not tied to any specific shell. And what you're describing is a shell, so I'm not sure how it's any different from existing solutions.

the__alchemist

5 hours ago

It is - this is why adding something to the Path is tricky on Linux. Or I should say, there is a mismatch between the common instruction of how to do it (export) vs what you have to do (e.g. edit bash config)

hnlmorg

3 hours ago

That’s because there isn’t a universal PATH variable.

Env vars are not global. They just have that illusion because a fork() by default will pass your running env vars to the child. Thus trickling that value down.

the__alchemist

3 hours ago

Exactly. It's annoying if I'm just trying to launch an application I installed by typing its name, which is IMO what PATH is for.

hnlmorg

2 hours ago

How are you installing it? Normally package managers will symlink it into a system defined $PATH directory so you shouldn’t need to faff with $PATH yourself.

Not all applications executable match the application name (eg Visual Studio Code is just “code”). So could that have been the issue?

the__alchemist

2 hours ago

Depends on the software. Generally, whatever the official installation instructions are.

> So could that have been the issue?

No. Adding things to the path is a reasonably common operation on all OSes and CLI-based software. Sometimes the installer will handle it for you; sometimes it won't. (This may be out of date, but I believe the Rust install script is an example of one that sets it up; Go is one that doesn't, and its official instructions CAO 6 months ago were to use Export, which doesn't work once you reboot)

arccy

2 hours ago

this is why you don't curl | bash

Lerc

5 hours ago

I always thought that there should be a proc style file system for simlinks to user specific data.

I'm not sure why this can't be done. Security people will have a million reasons, I guess it's possible one of them might be valid.

akoboldfrying

5 hours ago

IIUC, you're describing an extremely restricted (some would say underpowered) shell.

It sounds like you could make it yourself in ~10 lines of Python or bash. I don't see it catching on, though.

ligarota

4 hours ago

Juste create a file ~ which is a symlink to home :)

Big brain time here

panzi

3 hours ago

I expected it to also talk about ~username. See e.g.: echo ~root ~pulse ~sddm

arkt8

3 hours ago

It is a clear example of RTFM! An expansion is no a variable. An expansion not expands under quotes. A variable expansion is not simply an expansion as it is between brackets.

Again RTFM instead of wait for a miracle of your agent.

3eb7988a1663

3 hours ago

Ehh, there are many, many (documented) gotchas about commonly used software that routinely bite people. Yes, it would be great if everyone were an expert in how every aspect of their toolkit works, but that is not practical.

Grimeton

2 hours ago

People only reading the manual after they shot themselves in the foot.

Hilarious!

very_good_man

6 hours ago

Reminds me of early days of Cursor when it decided it would be a good idea to create a directory named "~" in my repo root!

That was a scary mistake to unwind!

Ekaros

4 hours ago

Just maybe, just maybe reserving somethings is not a bad idea... Whatever the fanatics say... There is enough features in shell that maybe banning somethings would be correct design.

ChrisArchitect

an hour ago

the irony of the submitted url having a weird double slash // in it.

quotemstr

4 hours ago

I was expecting this article to be about skew between HOME environment variable and getpwent(3) (usually from /etc/passed) views of the home directory.

They can be different things. Suppose your user is foobar, ordinarily homed at /home/foobar. You can write HOME=/tmp/my-test-home. Then, ~/qux will become/tmp/my-test-home instead of the usual /home/foobar/qux. That's because bash and zsh use HOME to resolve ~.

Almost. If you write ~foobar/qux, you get /home/foobar/qux again because the ~-with-username syntax looks up the getpwent home directory not the environment one.

And of course language runtimes are all schizo about whether the "user home directory" API uses HOME or the getpwent database or whatever to determine the home directory.

It's a mess, TBH. It used to be useful to temporarily bind HOME to something else to do things like create isolated test environments. Now, because of the aforementioned schizo sprinkler of randomness in the environment, you're going to have a bad time if you don't keep HOME synced to getpwent home directory.

duncangh

6 hours ago

I should have read the docs before getting ~/ tattooed on my wrist.

lysium

4 hours ago

If you haven’t quoted it, I think you’re fine.

duncangh

an hour ago

sometimes I feel like my environment isn’t properly configured but that’s just as likely a user error

NotMichaelBay

2 hours ago

Alternative title: There's no place like $HOME

gjvc

6 hours ago

bad example in the article:

  export PATH="$PATH:$HOME/.local/bin/"
better:

  export PATH="$HOME/.local/bin/:$PATH"

Backslasher

6 hours ago

Afaik it's habit to give system paths precedence so a malicious script can't shadow e.g. sudo and steal your password, escalating a local file write into root

toast0

6 hours ago

Otoh, if you don't put your local path first, you can't override system binaries that you want to override.

Also, if something can write into your path, it can probably write to your shell config and/or the environment variables.

stkdump

2 hours ago

Why would you want to override system binaries

toast0

an hour ago

Often, system binaries are ancient

BenjiWiebe

an hour ago

Are you a MacOS user by any chance?

3eb7988a1663

28 minutes ago

All sorts of utilities push themselves to the front of path: uv, mise, asdf, python virtual environments, nix shell, etc.

It is a theoretically nice ideal that fails immediately when you want project specific overrides.

paulddraper

6 hours ago

AFAIK it's habit to allow your scripts to override system ones, so you can customize behavior.

I've always seen home dir, homebrew, etc prepending to PATH.

marcosdumay

4 hours ago

That's important only for the people that add relative names (like '.') to their path.

Most people know better.

gjvc

4 hours ago

by that time it's too late and should have been prevented appearing on the host much earlier

IshKebab

7 hours ago

Bash footgun #238503.

paulddraper

6 hours ago

That's standard POSIX.

Tilde expands when at the beginning of an unquoted word.

Pretty straightforward.

  ~ or ~/ --> $HOME

  ~user --> user's home
---

Bash has a few extra.

  ~+ --> $PWD

  ~- --> $OLDPWD

  ~+N or ~-N --> dirs

Joel_Mckay

6 hours ago

In most use cases, the bash scripts location is more important than $HOME. This is because it is resilient to changes in user in the session, and parent process current working location contexts. =3

scriptPath=$(/usr/bin/realpath "${BASH_SOURCE[0]}")

localPath=$(/usr/bin/dirname "$scriptPath" )

/usr/bin/echo "localPath = '$localPath'"

_ZeD_

7 hours ago

well.. it's in zsh (and probably any other *sh) too

vbernat

6 hours ago

In Zsh, you can set PATH with path=(~/.local/bin $path). There, it works as shell expansion works.

mr_mitm

6 hours ago

So many headaches could be avoided if we only allowed `[A-Za-z0-9._-]` in paths. (Arguably, even `-` can be problematic.) Encoding issues, expansion, parameter separation, ... and I never saw a convincing case in favor of supporting anything else.

jetbalsa

6 hours ago

What about people who do not speak English? or even use Latin letters?

mr_mitm

6 hours ago

The language I grew up with does use non latin letters, I can manage. Then again, it's only four of them, so I get your point ... but I can dream, can't I?

ThunderSizzle

6 hours ago

What Latin letter(s) are you referring to?

The (classical) Latin alphabet can be fully described by the English alphabet.

toast0

6 hours ago

What about people who do not ((speak English?) or (even use Latin letters?))

cowboylowrez

6 hours ago

unicode names spit

jetbalsa

3 hours ago

if I want to name my file <turd emoji>.txt I damn well will

Dylan16807

5 hours ago

If you're trying to avoid headaches then definitely don't allow -. At least not as the first character in a segment.

And allowing any letters from any language is probably worth the hassle.

dylan604

6 hours ago

I always joked about setting a custom keyboard layout to replace the space char with the underscore char specifically for avoiding spaces in file paths.