Debian 13, Postgres, and the US time zones

213 pointsposted 13 hours ago
by move-on-by
(rachelbythebay.com)

88 Comments

JdeBP

7 hours ago

This does not trace things back far enough. The root is where IANA has long since segregated out a set of timezone file names into a "backward" collection:

* https://data.iana.org/time-zones/tzdb/backward

If one traces references, one finds this connected bug on Launchpad. Amusingly to anyone who has ever seen these sweeping timezone database changes over the years, Launchpad marked it as "This bug affects 1 person.".

* https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/200807...

The rules for the "backward" file are here:

* https://data.iana.org/time-zones/theory.html#naming

All of the US/* timezone names, such as US/Pacific here, have been backwards compatibility measures in place for the whole of the 21st century and some of the late 20th. The Olson database in the 1980s (mod.sources v08i085, comp.sources.unix v14i030) used these names. But the naming scheme changed somewhen in the 1990s to a continent/city and ocean/city form and backwards compatibility with the old names has been preserved ever since by the "backward" file.

* https://groups.google.com/g/comp.unix.solaris/c/ON_MPZxVdv0/...

Debian has moved into a non-depended-upon package a backwards compatibility measure that is as old as Debian itself.

AnssiH

7 hours ago

> Launchpad marked it as "This bug affects 1 person.".

That just means no one has clicked "affects me too" button yet (after logging in).

ForHackernews

6 hours ago

It's good Debian is making this change now to be prepared when the legacy United States are downed at the end of 2052.

tempodox

2 hours ago

I can’t tell whether you’re joking. Seems like a distinct possibility now.

roelschroeven

6 hours ago

> The worst part about this is that it didn't get so much as a mention in the Debian 13 release notes. I read through that document before going for it and never encountered it. Indeed, even now, you won't find "tzdata" or "zone" in it.

I had another issue during my upgrade to Debian 13 that also wasn't mentioned in the release notes. I filed a bug report, but I was told that the issue was not important enough to put in the release notes, and I should have instead more closely read NEWS.Debian of the package. So I don't really know anymore what the "Issues to be aware of" chapter in the release notes is for, because apparently it's only a small selection of issues you need to be aware of.

styanax

2 hours ago

Sharing undocumented gotcha: nginx default config (/etc/nginx/nginx.conf) now has `server_tokens off` set with a comment that it's "common practice" (agreed). This is not in upstream's git version of the file[1], I therefore presume it's a Debian maintainer change.

The operational upgrade failure is if you have any existing drop-in config that sets `server_tokens off` already a hard fail Nginx error about duplicated keywords will occur, causing the apt process to exit with failure code(s) during the standard upgrade process.

[1] https://github.com/nginx/nginx/blob/master/conf/nginx.conf

msk-lywenn

3 hours ago

What was your issue ?

roelschroeven

2 hours ago

The version of pdns-recursor in trixie uses a different configuration file syntax then older versions.

(There is a tool to convert the old configuration syntax to the new, but it requires a working installation. There is a command line option to enable support for the old format, which if nothing else helps to be able to run the conversion tool, but there's no information how to enable that command line option in the way Debian starts pdns-recursor.)

yellers

7 hours ago

Hang on a second, "(...) in 2023. US/* was moved to tzdata-legacy (...)"

US/* was moved to 'backward' (the file for backward compatibility) in the tz database in 1993(!) and as such was essentially marked as deprecated long enough. https://data.iana.org/time-zones/tzdb/backward

You're telling me you didn't notice ? It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.

mynegation

3 hours ago

Not only did I not notice, I have never known that country prefixes were a thing, having to deal with tzdata since 1999. I wonder if that timezone was typed in manually? I doubt Postgres 15 file contained it to begin with.

zahlman

6 hours ago

> You're telling me you didn't notice ? It was...

In a large fraction of cases in the FOSS world, it comes across that the developers really do want to communicate this sort of thing, but there's no clarity on where or how they should do so. See for example various deprecations in Python packaging tools (and standards).

JdeBP

6 hours ago

In this case, they did communicate it, and the aforegiven Vogon reference is a mischaracterization. The naming convention is in the current IANA doco and Eggert copy.

* https://data.iana.org/time-zones/tz-link.html#tzdb

* https://web.cs.ucla.edu/~eggert/tz/tz-link.htm#tzdb

Paul Eggert explained the continent/ocean plus largest city naming convention on a WWW page almost a quarter of a century ago. The WWW page was so well publicized that you can find its URL baked into at least four of the O'Reilly animal-cover books from the early 2000s.

* https://web.archive.org/web/20011023074744/http://www.twinsu...

It was explained on Usenet and on mailing lists prior to that.

bbarnett

5 hours ago

You know, the tzdata people are quite haughty. They claim to store all that change, accurately, and yet here we are.

An example of this falsehood? Well, in the 70s my father convinced most of my hometown, at least the portions between Main St and Wharf, that DST was absurd.

For almost an entire year, this was observed.

Do you think they kept record in tzdata? I tried to convince them, but no! I still have some dateplanners my father had printed up, and even a picture of the sign that was out on the road (to alert visitors!).

But no!

Do not trust the tzdata people. As you can see, they are not so accurate.

kccqzy

19 minutes ago

You are just not providing sufficient proof. Think Wikipedia style proof. The fact that DST is or isn't observed should be published in a real book. A random dude printing some date planners or putting up a sign to be photographed isn't enough proof.

macintux

4 hours ago

Once upon a time I stumbled upon a small book of maps showing the time zones in Indiana each year in the mid-20th century. It was fascinating to see how various towns and cities would jump back and forth between Eastern and Central year to year, with little regard for the surrounding rural areas.

I frequently kick myself for losing track of that book.

rollcat

6 hours ago

Then you get the reverse. I just upgraded to macOS Sonoma (yes I'm always one major version behind with Apple stuff...), and I was annoyed as heck when I had to click through "Look what's new in Calendar!", "Look what's new in Reminders!", "Look what's new in StripClubs!"... I need to use my software right now, I will not read this. Then I will forget it ever popped up, and will not read it in the future either.

thesuitonym

an hour ago

There's a vast difference between Apple showing you all it's useless new AI integrations and developers telling system administrators what they need to know.

skydhash

32 minutes ago

And I think Debian sent those notes also via its mail system (exim + mail). I don’t usually care for my workstation, but I do check it for my servers.

cpburns2009

an hour ago

Why are time-zones even prefixed by continent? Country-prefixed time-zones make more sense because they're defined politically.

umanwizard

5 minutes ago

In addition to what others have said, there are several examples where people disagree about which country a city is “rightfully” in.

Nobody can really find fault with Asia/Jerusalem, whereas either Israel/Jerusalem or Palestine/Jerusalem would be controversial.

eurg

an hour ago

Cities may find themselves in other countries easier than on other continents.

hnuser123456

an hour ago

I disagree. Country borders can move. I have not heard of a city moving between continents however.

op00to

2 hours ago

I feel like I’ve been using America/New_York for a decade at least. Am I misremembering?

db48x

an hour ago

No, that’s been the standard name for a quarter century.

evdubs

10 hours ago

I also ran into this using:

- Debian 13

- Interactive Brokers' Trader Workstation

- Racket's `gregor` date and time library

IBKR still sends the old, deprecated US/* timezones. As noted in the article, the solution is to `apt install tzdata-legacy`.

acdha

4 hours ago

I ran into a similar bug in Git Lab where the v18 upgrade didn’t replace now-unsupported time zones so things like scheduled pipelines simply stopped running and were labeled inactive in the UI without explanation.

https://gitlab.com/gitlab-org/gitlab/-/issues/556779#note_26...

The fix is simply to change them to the non-backwards linked names but it caused some confusing API errors since data which would no longer pass validation was already in the database and didn’t look wrong.

reactordev

3 hours ago

Ooofff. The two difficult things in software engineering, naming, and timestamps.

This hit me in the early 2000s and now everything I do is in UTC. All dates, timestamps, everything, UTC. If you want to look at a local window in time, convert the window to a utc start and end date and search. When viewing, use a js function to translate the utc date to a local one to print. The mental gymnasium of local to utc to local again…

dotancohen

2 hours ago

  > The two difficult things in software engineering, naming, and timestamps.
And off-by-one errors. The two most difficult things in software engineering: naming things, timestamps, and off-by-one errors.

nja

an hour ago

I'm still not totally sure why these names were deprecated in the first place...

I mean, the folks who run the tz db definitely know what they're doing, it just never 100% clicked with my thinking.

I always prefer `US/Eastern` over `America/New_York` -- it seems more "canonical" to me. New York is _currently_ the anchor city for ET, but will it always be? The place I live (Boston) is currently on ET, but in the future it might be on Atlantic Time. If there was an `America/Boston`, I would use that to be safe, but since there isn't, it just seems better to be to be specific that I mean "Eastern Time" and not "whatever the time is in NYC"... At least then if Boston switches to a different tz, I could intentionally switch to "Atlantic Time" -- doesn't that make more sense? Versus I guess what I'd have to do, which is switch to `America/Puerto_Rico`? (I had to actually search that one, too bad there's no `US/Atlantic`...)

skydhash

25 minutes ago

If New-York shifts its timezone, then it will still be New-York and America. And the new tzdata will reflect that shift so your data will always be correct (as in reflecting the time it is in new york). For local time, you will switch to a new city and the date will still be correct.

poemxo

8 hours ago

I don't get how this is a snag, it's right there in the log.

skylurk

5 hours ago

Agreed.

> At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

I get this, but now you get to laugh and dust yourself off. If it had silently started doing the wrong thing, that would be a worthy complaint.

burnt-resistor

6 hours ago

Always run production systems in the Etc/UTC timezone. This eliminates an entire class of problems while only creating minor inconveniences.

bityard

2 hours ago

I work for a company with servers, employees, and customers in basically every time zone around the world. And yet every server, internal tool, and workflow uses Pacific time. UTC is used precisely nowhere. Setting aside the issues of DST, I imagine it's convenient for the employees and managers in HQ but absolute madness for everyone else.

a012

an hour ago

Are you at Google? Because they also keep using US/Pacific timezone in every incidents where it affects everyone over the globe

zappb

an hour ago

That description could match nearly any tech company in the Bay Area or Seattle and everything along the coast.

macintux

4 hours ago

I work with a development team who manages data integration and migration for massive datasets, and they have sensibly set a standard that every date/time value they store in their databases will be UTC.

But they explicitly or implicitly have also decided not to store the timezone in the strings, so every single value is technically ambiguous. Absolutely drives me crazy.

Update: since there have been questions, these are strings, not native datetime values.

shawnz

2 hours ago

Postgres always stores date/time values in UTC (if they are timezone-aware)

macintux

2 hours ago

These are strings. And not Postgres.

ta1243

3 hours ago

> they have sensibly set a standard that every date/time value they store in their databases will be UTC.

Not sensible at all for future date/times.

Tepix

6 hours ago

For servers, i agree 100%. For desktop systems, not so much.

kawsper

6 hours ago

How else could you tell what time it is on the servers?

raverbashing

5 hours ago

That's easy, if you're Walmart just use the TZ of your corporate HQ

(That's what I heard anyway)

kstrauser

3 hours ago

My non-Walmart startup was expanding from California to other regions many years ago, and there was some debate about whether we’d use UTC or America/Los_Angeles globally. I was part of Team Over My Dead Body.

madcaptenor

an hour ago

I work for a different company headquartered in the central time zone and we also do this. On some systems.

ThePowerOfFuet

3 hours ago

Google's internal timestamps are in Pacific Time, too.

bornfreddy

6 hours ago

It also makes sense. Timezone is user specific - if you have users from all over the globe, they will need different settings, so this should be set in frontend.

bayindirh

3 hours ago

You can also run in UTC+$YOUR_OFFSET timezone if you don't use DST.

On the other hand, UTCx timezones are not silver bullets if your fleet is a part of a multi-continent federation.

ggm

12 hours ago

Is there a problem with ISO3166 denoted information in general or is there a specific US issue here? I would think ISO code denoted tzdata was a public good in some sense.

mananaysiempre

8 hours ago

This has nothing to do with ISO 3166 at all. The tzdata database is the work of a single person, Arthur Olson, not a committee (or rather it was, from its birth in the 1980s until 2011 when a company decided to sue him for no reason).

And for most of its life it’s been an explicit policy that timezones are named by continent and representative population center, not by country, to avoid entangling it in territorial disputes and improve naming stability for historical data. The US/* (and Canada/*, etc.) names are deprecated and have been since 1995 (?), but apparently people were still using them because the deprecation wasn’t really apparent unless one was especially into reading release notes.

masklinn

9 hours ago

All the legacy time zones were moved out of the default zoneinfo install. It’s not a us-specific issue but the legacy US/ timezones remain in widespread use, and they stop working on Debian 13 ootb (possibly Ubuntu noble as well?).

pixelesque

8 hours ago

Does anyone know why they are still in widespread use?

Config defaults somewhere still using them? Man page examples? Tutorials using them? Or just force of habit?

zahlman

6 hours ago

> Does anyone know why they are still in widespread use?

Because of a lack of things compelling people to change them until it causes a breakage. And then when it does cause a breakage, most people would rather move heaven and earth to complain, research workarounds, etc. rather than just change it. (Institutional structures can also make "just" changing it far harder than that should be.)

rtpg

7 hours ago

Some of this is surely just muscle memory or intertia as well. I remember random config values from when I was trying out linux boxes back in high school that I replicated into files that just don't get touched for decades afterwards.

When was the last time you rebuilt your company's postgres config from scratch?

Symbiote

4 hours ago

> When was the last time you rebuilt your company's postgres config from scratch?

Last year, when we upgrade to version 17.

I looked at the example/template configuration, diffed it with our configuration from PG15, and for every change decided whether to keep our version or the new setting.

I didn't use it, but Debian/APT has had a tool to do this sort of comparison for any software upgrade for as long as I can remember.

Do other people just copy the old config and shout "YOLO!"?

magicalhippo

3 hours ago

Still typing "nano -w filename" each and every time since back around y2k when I was working on Linux for the first time I was told that bad things could happen if I didn't...

acdha

4 hours ago

There’s definitely inertia but I think it’s also that the US/ names match official usage: nobody, not even residents, says the time zone is New York because the official name is Eastern time.

cpburns2009

2 hours ago

Exactly this. I still don't know if there's a technical difference between America/New_York, America/Detroit, or America/Indianapolis.

fredoralive

6 hours ago

I suspect some of it will be because the legacy form is a bit more intuitive than the standard form. You don’t really use continents and cities as a reference to time zones normally, countries and local subdivisions makes more sense, but as other people note, it brings up POLITICS.

MartijnBraam

5 hours ago

You don't use them normally in the US, I've been referring to europe/amsterdam or europe/paris all my life in Linux installers and various equipment. I've never ever encountered netherlands/amsterdam or something like that.

Symbiote

2 hours ago

From the list of deprecated zones, we could have been using "GB", "Poland", "Portugal", "CET", but that's about it. "Netherlands" didn't exist.

Given the old names were deprecated in 1993, it's hardly surprising that I never before discovered "GB".

Macha

5 hours ago

I wonder how much of an influence it is that US/Eastern is easier to type than America/New_York

pdw

8 hours ago

Are they in widespread use? They were deprecated in 1995.

ahoef

8 hours ago

I don't see conflict in those statements.

loloquwowndueo

3 hours ago

Imagine my outrage when America/Montreal was deprecated years ago. “It’s the same as America/Toronto, just use that” they said :)

ta1243

3 hours ago

Have Montreal and Toronto ever been in different timezones?

loloquwowndueo

3 hours ago

No they have not. But why not nix Toronto instead? Everybody knows Montreal is better, it should get dibs at naming the time zone.

I’m just joking all the way, by the way :)

ta1243

an hour ago

Toronto's bigger. Same reason Boston and Washington are "New York" and Seatle/SanFrancisco are Los_Angeles

(Detroit gets its own zone because it was significantly different to new york before 1970, normally it's only difference post 1970 which merit a new zone)

1oooqooq

4 hours ago

love the takes on this one blog, but "I'm copying an old conf file for over 20yrs and now got this issue" is kinda weak.

American exceptionalism time zones aren't used since the 90s. even the cpus from that time are already dropped from kernel support. heck even the text encoding is gone.

kstrauser

3 hours ago

That’s unkind. I want to do the right thing in such cases, but I’m also learning about this today for the very first time. I’ve never, not once, heard that US/Pacific was a bad idea until this post. If not for this, I still wouldn’t know. I thought it and America/Los_Angeles were semantically identically and just kind of symlinks to PST8PDT or whatever.

If anything, the city TZ always felt off, like I was opting in to that specific city’s strange legal decisions or something.

Symbiote

2 hours ago

For general user interfaces, it's for the user interface to show "US Pacific" or "UK" rather than America/Los_Angeles and Europe/London.

The timezone selector in KDE shows "Los Angeles | America/United States of America | Pacific" and "London | Europe/United Kingdom", for example.

raziel2p

4 hours ago

> At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

You're running a database system and you just casually comment out the configuration setting the timezone?

In what way did everything "seem fine"? SELECT 1 returned something? No further investigation required??

donalhunt

8 hours ago

What sort of monster are you by not defaulting to UTC on systems? (⊙＿⊙') /s

pfexec

6 hours ago

Debian is known to have made similar monstrously stupid decisions.

For example, they patch OpenSSH source code in a way that makes defaults behave differently than upstream. In the name of backwards compatibility of course.

I assume this will continue until it doesn't anymore, and the only notification you shall receive from the ivory tower is a cryptic one-liner buried in a changelog somewhere.

traceroute66

5 hours ago

> For example, they patch OpenSSH source code in a way that...

Isn't it the same thing with the RedHat downstreams ? (Not necessarily OpenSSH but other packages)

IIRC RedHat do all sorts of things to keep their gov / corp customers happy, also usually in the name of backwards compatability, all of which then end up in the downstreams.

burnt-resistor

6 hours ago

Russ Allbery left over bureaucracy and systemd. It sounds like it's chocked full of people who want power and an excuse to patch downstream to create a cottage industry of quirks, busywork, and codependency.

I prefer real choice and light patches that try to upstream as much as possible, or workaround upstream obstinacy rather than create incompatible idiosyncrasies. One area that isn't well represented in barely a/no distro is init freedom neither married to nor completely divorced from the sprawling octopus.

actionfromafar

6 hours ago

I don't think you can be partly married to Cuthulhu. (Systemd.)

styanax

5 hours ago

[ eldritch noises intensify ]

As an actual answer, it's not too bad on Debian; we only really use/need: systemd (system and user), -logind, -journald, -udevd. All in all, not too many tentacles but there are a few...