Re dogfooding. I worked at Oracle back in the 90s, first in the team building the company's HR systems (specifically their payroll product), and then in the team building the company's CASE (computer-aided software engineering) products.

The CASE products were intended to automate away coding, and Oracle's HR products, with their gazillions of lines of code, and with the developers located in the same building and with a common senior management team, were the best possible opportunity for dogfooding.

However while I was in the HR product team we never tried in earnest to use CASE (except for the ER modelling tools). There was no real enthusiasm from the CASE team to support us and the tools as they were then fell far short of what was required.

Later, when I joined the CASE team, I learned that their narrative was that the company's ERP products (like HR) were so complex that they were not realistic targets for CASE (* cough cough bullshit * - and perhaps the sort of attitude that doomed the CASE products in the long term).

My learning was that dogfooding is an awesome strategy, but sometimes much harder to embed in a development team than one might think.

Ironically the _lack_ of dogfooding GCP products at google is often quoted as one of the reasons AWS beat GCP to defining the Cloud market. Amazon builds AWS on AWS as much as possible, Google has only somewhat recently pushed for this

https://en.wikipedia.org/wiki/Tea_lady

Definitely. I remember a class trip to Intel's Jones Farm campus back in the early 90s and they definitely had company cafes. Everything was free.

Only half? I'd be willing to say that there should be some 19th century examples and we can argue if the food arrangements in the Valley of the Kings count.

I worked at a few tech companies before 2000 that had a company cafe but they all required payment. It was cheaper and closer but not free, and not nearly the same level of quality. Charlie's et al. were all free, and I remember even seeing a TV news spot about the free food at Google specifically, in the months before I started there. I think I'd be okay crediting Google with the free lunch (and the nod towards TANSTAAFL that I suspect it was).

But dogfooding, yeah that had been around for a while. Originally from Alpo, iirc. The first tech company to adopt the term as well as the practice was Microsoft in 1988.

A/B existed before Google but 2000s era A/B testing and user research were unparalleled until Facebook also started putting serious capital into it. Nowadays it's considered table stakes but it was revolutionized in the beginning of the millennium. Maybe not entirely by Google but substantially so, and driven heavily by their product launch review process.

(Author here) Yeah, good feedback, I think I overstated this in the article.

It would have better to say Google popularized these practices rather than pioneered them. Or at least, that these practices became much more widespread among tech companies after Google's IPO in 2004 than beforehand.

I think it's also safe to say that Google's culture was strikingly different from other tech companies of its era, as has been well documented in a few books.

Food cafeterias at the office go back to Henry Ford. Creating a distinction based on how fancy it is, is just the modern day “I invented it!”.

I worked out of the SGI campus before it got sold to Google, and I remember the on-site cafe there was amazing. I don't know how much Google changed about it, I've never been.

As a vendor, I (well, my company) had to pay for meals at SGI, I have no idea if the employees got free meals.

Didn't you hear, every silicon valley company personally invented everything they ever did. Chamath invented Data Science at Facebook!

You should know that people at google are the smartest people on the planet. Several of them assured me so.

When I worked at Electronic Arts in 2003 we had a lovely restaurant on the premises.

(Author here.) I agree with this critique in theory, but not in practice. The stakes don't need to be high to encourage strong participation. Corporate prediction markets have already scaled quite far, and those who have studied them don't find evidence of manipulation.

Can't let perfect be the enemy of the good!

If you want a fuller critique of prediction markets in the corporate setting, see the Dec 2021 article linked near the end [1].

[1] https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/...

> This is why we have problems with our current predictions markets, e.g. the stock market (insider trading, etc.)

A unit of stock represents a legal claim on a company's assets even in the absence of a market for that stock.

In many situations, we do want people to influence the value of their stock. Any company that grants stock is doing so because they expect employees to work harder. There are many cases in which employees with stock grants might make short-sighted decisions for quick profits, but on the whole stock grants make employees into better workers.

> A senior executive saw Prophit give a very low probability that the company would complete the hire of a new senior executive on time (filling the position had been a quarterly objective for the past six quarters). “The betting on this goal was extremely harsh. I am shocked and outraged by the lack of brown-nosing at this company,” the executive said to laughter in a company-wide meeting. But the market was the nudge the execs needed. They subsequently “made some hard decisions” to complete the hire on time.

In this case, the predictions market would've been useless had the executive team not used it in their decision-making.

A pure predictions market would be completely isolated from the event they're trying to predict. But I would say that's not an ideal predictions market.

Prediction markets (like the trump vs harris bet on Robinhood) can also be used as a hedge.

E.g. if before the election you think that a certain candidate winning would cause the markets to react in a certain direction, you could "bet" on the other candidate so that if your portfolio value goes down, you earn the proceeds from the bet to recoup some of your portfolio losses. Or if the "good for stock market" candidate wins, you loose the money you bet but the gains in your portfolio balances it out.

In that case, you're not really betting on who you think will win. You're just betting as a hedge just in case that person wins.

You make a very good argument against prediction markets in which the stake has actual material value, but what about markets where the only stake is your reputation? My understanding of the Google platforms is that there were prize rewards for top predictors but largely the rewards were social, which, to me, implies they were not taken particularly seriously. An ideal platform might be one where there is no material incentive but users are instead interested in genuine hypothesizing about future events. Obviously there must be a genuine stake involved, but as we see with Manifold this can be done with 'play money'.

Would be very interested to see further discussion about this.

My experience from predicting on Metaculus, and following this space for a few years, is that it's hard to operationalize the thing you want to predict precisely. Regularly, things go completely sideways in ways that the author did not foresee, and the market ends up hinging on some technicality, rather than the spirit of the question it tried to predict.

Some examples:

    - A question about asset prices specifies FTX as resolution source, but then FTX stopped existing.
    - There was a question about wether submarine cables in the Red Sea would be destroyed by a hostile act before a certain date. The cables got damaged, but it seemed to be a (suspicious) accident, with very limited independent media coverage.
    - There is a question about wether YouTube would be blocked in a certain country before 2025. It got throttled, to the point where it is unusable in practice, but not technically blocked.
    - There is a question trying to forecast LLM progress. How to quantify that? It chose "What is the state of the art score on the Penn Treebank at a certain date?", which was a standard benchmark at the time. But as LLMs evolved, new benchmarks got developed, and although current LLMs probably score much better on the Penn Treebank than a few years ago, nobody reports Penn Treebank score any more. The question ended up being about the popularity of the benchmark rather than LLM progress.

> the two are not and cannot be isolated, except for some degenerate cases like trying to guess the output of a true random number generator

You’re describing endogeneity. It’s unlikely prediction markets affect natural disaster odds.

There are some ways to mitigate this. You can make the prize pool big enough that people will be motivated, but small enough that people won't try to sway the outcome. Also, prediction markets can be very useful for making predictions about competitors, because it would be rather difficult to influence what happens at a competing company.

What do "any real sense" and "scale" mean here? I was watching the Betfair odds during the US election, and that market alone had about $500 million dollars of bets pass through it. Is that not real? Is half a billion dollars not sufficiently scaled up?

A "good" prediction market is when it functions as an accountability market. The insider trading problem isn't an issue if the outcome is for a greater good. The problem is that money is the easiest way to ensure anonymous predictors are serious about their predictions and getting richer is not a greater good. If you were to require politicians to participate in a reputational prediction market, the aligned incentives might make it a positive thing.

You could let ordinary people piggy back participate too, and then use the results to filter through internet/media noise but that starts to smell too social score-ish.

I think there are cases where the outcome of the event can be bound to the prediction. Also we should be looking for cases where:

> Every prediction with a stake is an incentive to alter the outcome of an event

...is a feature not a bug.

For instance, imagine if PR's on core infrastructure (like xz-utils, for instance) were first reviewed by a set of trusted maintainers, and--supposing they pass that gate--were later put into a sort of limbo where people can bet on whether they'll be merged. The maintainers then make a policy around betting that the PR will be merged. They're in charge of whether it actually gets merged or not, so most of the time this is a pretty good bet and they just recycle that money by winning and then re-betting it on the next PR.

Of course third parties can also bet in the same way--these would be stakeholders which are not maintainers, but who wish to "sweeten the pot" and encourage people to spend enough time with the pending code that they might find a reason to "bet against the house". No prior coordination between the maintainers and the stakeholders is necessary.

Suppose I notice some malicious code in one of these commits which is in the betting phase. (Maybe I do my own testing on pre-release versions as a stakeholder, or maybe I'm a bug bounty hunter.) I can bet that the PR won't be merged. The monetary value of that bet makes it clear that even though I'm a stranger, I'm not a spammer. That "buys" me the attention of the maintainers, and if I'm right about the code being malicious, the maintainers will decide not to merge the commit: I'll have successfully altered the course of events (bad commit doesn't get merged) such that I get paid for having been right.

How does a prediction market for say, hurricanes or earthquakes effect the outcome of those events?

You could as easily say that journalism is an attempt to influence events. And it is, in a way! I think most journalists would be happy to point to situations where their reporting was read by influential people and maybe even changed outcomes. Pulitzer prizes are awarded for that sort of thing.

The question is whether this is improper influence? Is it too influential, for bad reasons?

I think a more reasonable critique of prediction markets is that it's guesswork laundering. We are given a number, but we don't know why that number was chosen. How can we tell whether it's justified?

When markets move, there is a whole industry of people coming up with explanations of why it might have happened - more guesswork!

A lucky guess can be helpful if it can be verified, but knowledge should be about more than making guesses. Sharing evidence is important.

lack of isolation isnt necessarily a insurmountable problem, as the market can still forecast, even if it is sometimes a self fulfilling prophecy. Similarly, this distortion does not necessarily overshadow the utility - e.g. the stock market may have insider trading, but is still an incredibly useful financial tool.

They key thing to remember is that these markets can have utility beyond zero-sum betting aspect. For example, the stock market isnt just gamblers betting against each other, but is also a tool for auctioning corporate ownership and raising corporate funds.

I've spent some time looking at group decision making for specific situations and there are clearly some examples where they work very well--and I think you hit on one of the factors. The individual guesses don't affect the outcome and aren't being used to hedge anything. I think there are some other factors as well but I'm not sure I've ever seen a particularly persuasive framework for when prediction markets can work and when they don't.

Great comment, I was fairly horrified to open Robinhood and be faced with a prediction market for the US election. I somehow missed completely that this has been legalized: https://amp.cnn.com/cnn/2024/10/02/business/appeals-court-al...

A market can still be well calibrated even if it has a strong causal influence on the topics it’s predicting.

the same dynamic is true of insurance, credit default swaps, and lots of other financial products. this causes trouble from time to time but can be managed if the products serve a useful purpose.

> See also the famous example of how a predictions market for when public figures will die is just an assassination market with extra steps.

Why not just write it so that non-natural causes of death don't pay out? More generally, make it so you can't wager on illegal events / outcomes, or ones where a crime materially affected the outcome. Anyway, if someone bets big on a public figure being assassinated, and then that public figure gets assassinated, it would seem like a good place to start investigating would be to look at the people who made a lot of money from that bet.

In other words, money is speech.

[dead]

You should make your criticism explicit; just making a joke like this isn't particularly helpful.

To make explicit what I'm assuming is your point, this statement from the article can't be correct because A. POWER is by IBM, not Intel and B. Apple had already launched such a computer back in 1994, before Google existed.

Maybe x86 was meant instead?

(Author here.) This is generally checked via calibration charts, e.g. bucketing markets at various points in time into 0-5%, 5-10%, etc; then counting how often the underlying events actually happen. The more they match, the more it's reasonable to interpret the market prices as probabilities.

Google published [1] one such calibration chart on its current prediction market in late 2021. Also, the 2009 paper in the article [2] on Google's first prediction market published one too.

[1] https://cloud.google.com/blog/topics/solutions-how-tos/desig... [2] https://static.googleusercontent.com/media/services.google.c...

Even if you assume perfectly spherical rational traders with equal fixed budgets, prediction market prices aren't guaranteed to converge to the average belief: https://quantian.substack.com/p/market-prices-are-not-probab...

You cannot really do postprocessing to the market price to get the average belief back out, because the bounds aren't very tight: a market price of 50% could correspond to an average belief anywhere between 29% and 71%.

(Author here).

> To rise above the simple wisdom of the crowds, you would want to identify the subset of market participants that are constantly beating the market (because they have a more accurate mental model of the world).

Identifying, and then working with, the top traders at Google (including one card-carrying superforecaster) was a great joy.

And yes, they're sitting on some great data, on what the employee crowd tends to get right and wrong, who individually is good at forecasting what. Though one complication is that being a great trader is not the same as being a great forecaster.

Markets are what you described. Participants that regularly beat the market are rewarded with more money (and confidence) which lets them bet larger size and have more impact on the market.

Uninformed bets should wash out as noise, and informed bettors should reverse uninformed moves so long as they are profitable.

Many sports gambling companies do this, weighing the bets of "sharps" (people who are more accurate than the average) heavier than other bets. A good example of this was Mayweather vs McGregor where a lot of sharps were betting on Mayweather whereas the public was betting more on McGregor. Even with about 80% of people betting on McGregor, the house still had Mayweather as a favorite.

The problem is that if a prediction market exposed the fact that 80% of their most accurate betters are betting for YES, then wouldn't that skew the rest of the market? If I'm on there and I think it's about 50/50 chances, but I see that the super-betters are mostly saying YES, I'm probably going to vote YES too.

There is a conflict of interest between the information prerogative, which requires facilitating the law of large numbers and factors stability, and the profitability of gambling, which favours encouraging larger bets and volatility.

(Author here) It's complicated :-). I will say: it's hard to define "gambling" in the corporate setting.

Is it gambling to do an extra project in hope of getting a "Spot Bonus" (~$250-$1000) in recognition? That was a very standardized process at Google.

Is it gambling to file a patent application, which if approved, would lead to a $1000 bonus and a trophy you'd often see on the desks of Google Brain researchers?

Is it gambling to decline auto-sale of RSUs, and have your compensation determined more by movements in GOOG than in your cash salary?

They don't use real money. It's no different than having a poker night with Monopoly money and the winner gets a prize.

It's less of a gambling platform than a way for people to crowdsource information in an unconventional way.

> as of August 2024, the team continues to refine its approach to make Gleangen a useful source of information for Google senior management.

And it is apparently now officially a way to get information to Google senior management.

Metaculus and Manifold Markets are both "purer" alternatives