You didn't expect the application you developed called "Skynet" that has an eye of sauron as it's logo to be used by the military? Sorry, I don't believe you.

Yep. Early in my career I developed a simple driver for a CAN interface to be used in a medical device, which was my assigned project.

The same code was used in another project of this same company. The project was a missile. It felt weird.

There should be a distinction between helping the military and helping a police state.

Eg. Flock cameras don’t help the military but do help the police state.

I don't think the author is suggesting we shouldn't build useful things. He's saying to the people who built the military system that they shouldn't be doing that.

It's the last step that's the problem, not the people who built all the things they stand on. Otherwise we'd be suggesting that no one build a database engine or a digital camera or a computer ever again in case they're used to build evil tech.

clearly people have a responsibility to sit down and think long and hard and try to rule out as much as possible the negatives of any tech or ideas they share with other people. Or at least give it a cursory thought or find someone with that mindset to tamp those issues down. Absolutely before even considering a funding step or presentation to anyone outside of a very small trusted group. Ideas take fire way too quickly now not to be cautious.

You didn't expect your telemetry system to be used as a telemetry system?

I don't think the article is saying don't build a payload-agnostic telemetry engine. The specific examples cited are that it's probably legitimate to have a vehicle registration database (with the caveat that even that could be abused), but not a database of sexual orientations or religious affiliations.

Keeping it within game dev, I will say that telemetry and bug report tooling looks really cool. Big fan of it grabbing build info directly from the session that the user was playing. Feeling kinda inspired to build something similar for the studio I work at now.

Thanks for sharing!

I think there's a distinction between creating generic info-gathering technology that helps them gather their own data from their own users, and you gathering data that the government can then ask (force) you to hand over.

Do you actually object to countries having standing armies with ranged area weaponry?

This isn't what the article is talking about at all. It never says don't build military technology. It says don't build things like a database of who in the country is and isn't gay and what religion everyone practices, for fear a future bigoted administration might use it to target those people. If your government is turning howitzers on you, that's less a police state and more a civil war.

I don't know what you do against that, but it's a different situation. There is precedent in the US with the Army being used to round up and forcibly displace native populations, but that really seems to be the kind of thing that requires a political solution, not prevention of any possibility. If we didn't have an Army at all, then secession would have succeeded, there would still be chattel slavery, and one of the hostile European colonial powers likely would have simply conquered the west itself and killed all the natives anyway.

>Harder than it looks.

But maybe easier than it sounds.

All technology can be weaponized. It can also be de-weaponized.

One of my favorite late-night thought experiments, is "what would it take to turn an AK47 or an AR15 into a water pump?" .. the possibilities are endless. "How could a tank be turned into a bicycle factory?", another great one worth the effort to think about.

The only difference between peaceful technology and a weapon is the intention of the user. The very real spiritual intention.

De-weaponization starts with the human being. Those who think weapons are necessary must always take lesser priority than those who invent technology to make the lives of their fellow human beings better. There are so many examples of the tool-makers winning harder and at greater scale than the bomb-manufacturer. We wouldn't be here now if the balance was the other way - our world is however threatened by having this scale weighed heavily, by those who think murder is the solution to anything.

It's a spirit thing. Not spiritu-al, a spirit thing. What is the spirit of the designer? If they want to help humanity, it will be difficult technology to weaponize (although not impossible). It will be easy to de-weaponize, also.

We need to focus more on the spirit of help, and less on hurt.

Imagine if those F35 mounting pylons were used to drop field-portable water-purification systems directly where they are needed most.

Imagine if they could be used to drop a copy of archive.org, anywhere in the world, in any language.

Imagine if those ships were being used to bring power to the villages that need it most. Instead of glide bombs, we have glide libraries. Instead of cluster munitions, we instead drop cluster medical kits!

Oh, the glory to be had by those who would do this - the allies in the poor and weak of the world to be found! Imagine this happening, changing the intention from hate and fear, to respect and love.

All it takes is the spirit, a bit of imagination, and any and all technology can be de-weaponized.

It is the warrior narcissist who stands in the way of this. Disarm them too - smash their mirror: Drop books, not bombs!

I feel really bad for you.

It might actually be wise [1] to create a new, slightly less permissioned license for open source wherein the open is defined as anything that isn’t in a kill or surveillance chain or, more broadly, for military use…

[1] for the community

On the other hand, a few heuristics will get you pretty far:

Don't build technology that interacts with people in ways they did't consent to. Don't build skinner boxes. Let the user decide who gets whatever data they generate while using the thing. Let the user control how it will work, insofar as it doesn't contradict the other heuristics.

These are political principles, but they're pretty basic. You don't have to keep up to date with the news, just check in with your morals once in a while.

How do you make sure a police state doesn't happen? Block traffic? Tweet? Cut off ties with friends and family that don't share your political beliefs?

Of course you can avoid politics. Just be a good person. Vote for who you think is the best candidate at the time. And move on with your life.

Ironically the people most involved in politics don't have their own house in order. They're often mentally ill, over educated, underemployed and without meaningful ties in their life (e.g. family, real life community, etc).

Politics in the broadest sense is just any decision-making process where a single decision affects multiple people and those people don't all agree on which decision should be made.

>you cannot ignore politics and just "focus on technology."

Stallman was right again

https://www.azquotes.com/quote/755239

We are well, well past the point where it's possible to avoid a police state.

An conscientious objection is definitely a right you have as a professional, but it's a neutral action, you refuse to do something, and you let someone else do it. If you really are against it, you actively do something about it, like presenting a law or making a criminal complaint.

That said, they are both compatible, you can refuse to do something, and make efforts so that nobody does it.

But I think the the latter is the actual core security model of our society, and refusals aren't really effective to the point that I wouldn't worry much about it and just focus on compartimentalizing and executing on my duties. Isn't it theoretically possible that upper management divides duties so that engineers cannot accurately judge the ethical correctness of the task and the overall mission? As an individual you don't have the right tools to make such an evaluation. You can be pushed to investigate further, but the moment you do so, you cease your role as an engineer and put your citizen hat on.

I personally wouldn't object on most cases, my threshold would be very high, a database of people's sexuality is not even close to even being concerning to me, that's like a Wikipedia infobox property. But even if it were a feature that did concern me, like a citizen-ID to real time coordinates system, I wanna be at the helm of it, either way (if I get the privilege). If something is wrong, I'd be in a privileged position to do something about it internally, judicially or through congress, there's also whistleblowing if it comes to that, but that's not in me.

Actually, you cannot avoid sex. Everything is about sex, even politics. If you want the freedom to develop technology without worrying it will be used by a police state, you cannot ignore sex and just "focus on technology"

There's a deep failed idea here. It's the idea that technology is an alternative to politics, period. I think that whole idea has failed.

You can't avoid building technologies that could be used by a police state, because literally everything can be used for warfare or aggression. The first person who sharpened a stick or a rock created a weapon. Everything is a weapon. Everything is dual-use.

If you want to prevent a police state or any other undesirable political outcome, you have to confront the politics directly. "Change the tools and you change the rules," an old saying from 1990s cyber-culture, is an idea that I think has failed. Instead what happens is that the rules determine what's done with the tools. Those 90s cyber-hippie and cyber-libertarian types vastly underestimated the power of the political to the point of making a category error.

Cryptocurrency is a great case in point. It started as an attempt to create a technology that would act as an escape hatch from corrupt casino capitalism and state-backed financialization. Instead it became an insane absurdist parody of those things. The same types of corrupt individuals who profited off "too big to fail" and the rest of it simply took cryptocurrency and used it to implement the same type of system but on a block chain. The rules (and rulers) changed the tools, not the other way around.

Few things are more important in modern life than access to your money. That's why I would never use an online bank. I use a local credit union but "local" with actual branch offices you can walk into and talk to a human are the important thing.

Every business school and MBA program has ethics course requirements. Does it really make a difference? Maybe it does, and things would be even worse than they are otherwise. But I'm not sure.

I'm curious why you think an ethics course would actually change minds. In my experience in undergrad, we had a CS ethics course, and it went in in-one-ear-and-out-the-other for 99% of students, probably myself included. I know it was a thorough course but I promptly forgot all of it, and so did everyone else.

Ethical stance is not something you can teach in one course. If you don't have this disposition by the time you're in college, I think you're likely not learning it.

A course is nothing. I've long felt that Software Engineering should follow suit of many other engineering disciplines and push responsibility down to engineers. Like how when a bridge falls, an engineer has to answer for it.

I don't know exactly how that looks, and yes there are many issues with it, but we're living in the age of cowboy coding.

Don't ethics courses make people less ethical because they teach people how to come up with ethical justifications for straightforwardly bad things?

Especially if it has concrete historical examples of technology intended for good and how it was later used for evil, rather than something more generally-philosophical.

You really think taking an ethics COURSE will do anything?

How are you supposed to 'learn' ethics?

Never understood the point of those. My university had over 4 for the cyber students and over half the grads went on to work for the MIC contributing to blowing up brown children in the middle east anyway.

Clearly, our problem must have been that they didn't have 5 mandatory time-waster ethics courses. That surely would have solved the matter. Add in a few extra "racism is bad" and "don't be a meanie" courses to round out the collection, just for good measure.

At least I had 6 humanities courses telling me why communism is good

Edit: To make this comment have a little substance, what I'm getting at is, universities are already way too biased and one-sided on humanities issues to be teaching ethics. Students notice it too. I've gotten As on some insanely bad papers just cause I knew what they wanted to hear.

Whose ethics?

We had an ethics course, it was a requirement for graduation if I remember correctly.

It was entirely performative. You can teach a curriculum of ethics, but that won't make a person empathetic. These are courses for credit hours, not firmware updates.

And a person with empathy can always find or be given reasons to not apply his empathy to a person or a group of people.

The problem is not a lack of ethics courses. Many CS programs have those ethics courses.

The problem is all the techbros who take those ethics courses and laugh at them as easy and pointless and don't care about them.

There's a lot of very unserious people right now.

The article says the same. I think the point is to be aware and try to build things in a way to minimize the chance for abuse. There's nuance and the author says as much.

It's because we have open primaries in the US - registering as a Democrat or Republican is only important as far as voting in primaries. In other countries the selection of candidates is internal to the political parties themselves. There are pros and cons for both systems but the US system is fairly unique globally aside from a few exceptions in South America.

It's weirder still because you often don't actually have to register with a party, even to vote in the primaries like other commenters have mentioned; it depends on which state and which party. [0]

For example, I just voted in the Minnesota primaries. Ours are open primaries, so you don't need to register with any party and the ballot has every party on it. Though, it spoils your ballot to vote in multiple parties' primaries. So you gotta choose, but you do that privately.

[0] https://ballotpedia.org/Primary_election_types_by_state

Because by and large they're all trapped in an "us vs them" team sport where both sides believe everybody on the opposing team is literally Satan.

It seems quite weird. Our parties are generally quite weird because were are not supposed to exist in the first place and so most of the rules are sort of ad-hoc.

I don’t belong to one, but it is sort of annoying to not be able to vote in the “primary elections” (where the parties pick their candidates). My state has a strong partisan lean so this is the only election that actually matters here. Oh well.

Then again, EU citizens would find it questionable how in the UK recruiters are asking about your ethnicity, disability status etc.

Let's slip in your pet bugaboo. Workforce size very little to do with state surveillance. In fact, mandating that it shrink can result in blowback, just like we have seen with how the US provides a great antagonist for China to evolve its own capabilities faster than if we had done nothing.

How does the number of state employees de facto result in a police state?

I agree with you on our obscene debt and our obsession with "conflicts" around the world, but I'd argue we're seeing the mass reduction of state employees and it's just making everything worse (eg massive deregulation and the destruction of checks and balances).

It won’t work for any country. They all want more control, and also create a class of people depending on government money to be on their side. Only people losing out are the powerless ordinary citizens and taxpayers

And a rise in tech companies building products and features that dont benefit the users, knowing we cant really escape at this point.

Thanks, didn't notice it before you pointed it out (and I already made my comment), actually turned into quite an interesting retrospective for me

> I prefer speed cameras, red-light cameras, and so on because automated enforcement removes discretionary power from enforcement agents.

There can still be corruption with automated enforcement. If the punishment is fees and the fees go towards the budget of those in charge of the automation parameters, then you'll see manipulation of parameters to increase fees paid. Shorter yellow lights, shorter distance between a speed limit sign and the speed camera, requiring more effort to contest the automation when it's wrong than to just pay the fee, etc.

If you don't man the concentration camps, somebody else will. So you might as well do it and do a better job and get paid well.

A significant number of people in positions of power in the AI industry get their moral framework from science fiction books and a particular Harry Potter fanfiction. It’s hard to predict what ideas and philosophies will become important.

People have incentives to become more powerful - through wealth, power, leverage, or in other ways. So unfortunately, authoritarianism tends to have gravity going for it.

The Nazis relied heavily on IBM technology for national census data (to identify jews), military logistics, ghetto statistics, train traffic management, and concentration camp capacity

https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

I fully agree with your base perspective, that the main focus should be on preventing authoritarians from coming to power in the first place.

I also agree with your final point, that the government doesn't need tech to lean on a person.

We disagree in the middle, on how tech that makes evil orders of magnitude easier might be a contributing factor and might enable the government to lean on many more people than it would otherwise.

The water flows whichever way there's a gradient of height.

Don't create gradients you don't have dams for.

>The Nazis didn't need Flock cameras to kill millions.

No, but they were extremely happy to hop over to France and realize that we kept lists that indirectly listed jews. Actions have consequences that we may not be able to predict.

The impossibility of drawing a line! So difficult! So nebulous! I will work for a concentration camp

You will get downvoted to oblivion for even suggesting it here but it's so obviously a take that maps on cleanly to reality. Massive power centralization in government makes everything dual use in the service of a police state. The easiest thing to do is resist massive centralization of power in a government.

That has been the motto of our governors.

This is the most correct response.

/me inserts the "You're proposing a software solution to a social problem" copypasta form

Seriously, if the coordination problem you've identified isn't solved by the instantaneous global communications tech that we've had for between thirty and a hundred years (depending on how you count), it's a people problem that's not going to be solved by more tech. Notice how everything except maybe steps six and three in your plan require people to agree on the nature of the problem, whether it is a problem that even should be solved, and how many resources to commit to solving it. Getting people to agree to solve problems that need to be solved -even in the cases where solving those problems makes them personally worse off- is the blocker. Your use of shiny new software tools does nothing to remove that blocker.

> because laws tend to need be broken to be changed

Nope.

> many countries like Singapore, etc. prove this easily

Economic freedom is a necessary but not sufficient condition for political freedom.

well, be glad it's company survival and not personal survival yet. We had that article about the men with guns in egypt recently too

> There will always be someone willing to build that thing

… with sufficient time and money. As opposed to the minimal time and money required if the baddies are just pulling an existing solution off the shelf.

I hope you are joking, or haven't thought this through! If you sincerely believe and act on what you're saying here, you should be shunned from polite society.

Well, there are quite a few frames to think about. What about the unintended consequences of building something for a worldview at the time? Then the next generation of stewards take over, do not have the same understanding of the problem the system solved and different ideas on how to use the system. That's definitely something worth writing about. Stewardship in software is a real and common aspect of the greater system we all live in, and has unknown consequences with each generation that takes over. When you step in as a gen-1 builder, you should probably consider what it is you are about to build.

Just because someone is willing to build it doesn't mean it has to be you.

We may all play some part in societies that have some degree of monstrous outcomes, but I feel like staying on the "tragic civilian casualty" side versus the "legitimate target" side of a freedom fighter's bombing campaign is probably the way to go.

It's not about someone willing or not, it's about you, an ostensibly grown adult who should think in terms of your own responsibilities instead of others'.

“Someone else will build it anyway, might as well be me.” I don’t think embarrassment lies at the feet of OP.

What's embarrassing about it?

Speaking of embarrassing, your argument is an argument against having any professional ethics whatsoever.

It's embarassing for an adult to think like this.

Joe's gonna die anyway. I might as well kill him before someone else does.

It's embarassing for an adult to think like this.

Terrible argument. You can say ”why not work as a concentration camp guard? There will always be someone willing to do it so you might as well cash in”

Smart phones are opt-in technology. Flock is not.

Very flawed argument, anyone can go for a walk without their phone on them and have a conversation, nobody would ask them why they don't have their phone on them.