You don't have a buffer overflow problem if you read it in a memory-safe way i.e. read it in chunks and realloc when you reach the size of your allocation.

What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)

[delayed]

In regular unicode, a grapheme can be made up of an arbitrary number of codepoints (and thus an arbitrary number of bytes), which does cause issues at times.

> this just screams buffer overflow problems

Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.

OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).

> […] why limit yourself to 24-bit numbers?

For compatibility with UTF-16:

    o  Restricted the range of characters to 0000-10FFFF (the UTF-16
       accessible range).

* https://en.wikipedia.org/wiki/UTF-16

The original spec had 31 bits (the UTF-32/UCS-4 range):

* https://datatracker.ietf.org/doc/html/rfc2279

* https://en.wikipedia.org/wiki/UTF-32

> ...24-bit numbers?

Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".

I think that wouldn't change much. They would just make use of more grapheme clusters.

For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.

U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.

> some unicode programming script (running Doom)

I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course allowing a script to be written in a single codepoint is the much more elegant solution.

It also neatly solves the problem of how to write Unicode strings inside scripts inside Unicode strings and also scripts inside Unicode strings inside scripts inside Unicode strings.

(*) in RFC 04-01-2027

[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)

Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.

Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.

One more idea:

- Unicode-as-a-character, meaning: a whole text displayed as if it were one character, made for infinite zooming

We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a problem, UTF-16 will be long dead

Yeah that kind of line is what I see all the time in my chats. Even worse worse is when they put it in code comments.

Important to recognize that characters have individuality, that's why there can only be a limited number of them. Unicode is enumerating a finite set of things, not encoding an infinite set. Aenything without this property - any generic form of encoding - is not characters, it's something else like images. If it's not in any alphabet it shouldn't be in unicode, you should use an escape tag for image data instead. (Emojis probably shouldn't, but they do behave like an alphabet)

There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.

256-bit addresses would be long enough proper cryptographic addressing. (128 is barely not enough due to how they're allocated)

512 would also for cryptographic addresses within cryptographic subnets (or 3 128-bit layers plus some extra bits), and 1024 would allow for up to 7 layers. It's not a completely silly idea.

Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it, which isn't adequate in cryptography. Individual addresses can also be created with 112 bits of entropy, which is barely adequate. That's why I say 128 bits total isn't enough for cryptography plus subnetting.

True, but not all non-UTF8 bytestrings contain 0xFF bytes, so it’s not very useful in practice.

It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.

As are FE, FD, FC, FB, FA, F9, F8, F7, F6 and F5.

timecube.com is lost, zombo.com has been redesigned. If this is not proof of a dead internet, I don’t know what is.

> Does anyone have a good example of when it‘s useful?

It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).

Rely on it? Not that I can remember.

However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.

I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.

Only if they've already both undergone normalization to NFC or NFD.

> This is a fun standalone project / proposal

But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.

No, WTF-8 is its own encoding: https://wtf-8.codeberg.page/