As someone who spent most of a career in process automation, I've decided doing it well is mostly about state limitation.

Exceptions or edge cases add additional states.

To fight the explosion of state count (and the intermediate states those generate), you have a couple powerful tools:

   1. Identifying and routing out divergent items (aka ensuring items get more similar as they progress through automation)
   2. Reunifying divergent paths, instead of building branches

If you want to go back and automate a class of work that's being routed out, write a new automation flow explicitly targeting it. Don't try and kludge into into some giant spaghetti monolith that can handle everything.

PS: This also has the side effect of simplifying and concluding discussions about "What should we do in this circumstance?" with other stakeholders. Which for more complex multi-type cases can be never-ending.

PPS: And for god's sake, never target automation of 100% of incoming workload. Ever. Iteratively approach it, but accept reaching it may be impossible.

Not directly relevant to the post, but seems like a good place to share.

My team and I once took on a very tricky automation project. At the time we had a complex software deployment done about once per month that involved a team of about a dozen people showing up at 4am to do it while traffic was low.

The deployment involved many manual steps and coordination of everybody involved. The person leading each deployment followed the documented list of steps and got each person to do their bit at the right time; people to run database migrations, people to install RPMs on particular servers, people to test and verify functionality. Mistakes and missed steps were not uncommon.

The very first thing we did was take the documentation and write a Jenkins job to post each step into a Slack channel specifically for coordinating the deployments. Someone clicked "go" and each step was posted as a message in that channel with a 'done' button to be clicked when that step was done. Clicking the button caused the next step to be posted.

The next release we did used that instead of one person reading the steps out of confluence. Everyone involved in the release could always see what step we were at, and when it was their turn to do their bit. This helped ensure no steps were ever missed too.

Over the following months we chipped away at that job a bit at a time. We'd pick a step in the process and automate just that step, starting with the low-hanging fruit first. The Slack message for that step went from "click to confirm you've done it" to "click to do it", with the result posted once it was done; followed by the next step to perform.

It was a long process, but it allowed the rest of the business (and us!) to gradually gain confidence in the automation, and lowered the risk of the project dramatically. Once several steps had been automated and battle-tested we removed the 'click to do' bits in between and the whole release became a couple of clicks followed by the odd bit of manual QA.

The Do Nothing Scripting [1] is an approach that I feel recognizes the difficulty of automation by starting out with a script that just tells you what to do like a checklist. Once you've used it to fully document all the exceptions, etc. you can then start building the automation.

An approach like this seems to give more weight to the fact that just figuring out a way to document exactly what needs to be done is often the hardest part and if you get that right before you start writing automation code, it might make make the automation much more efficient.

[1] https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...

This reminds me of Seeing Like a State[0] (I know, it's not new... but I only got around to reading it earlier this year.)

Automating a process makes it more standardized and legible, but takes away a lot of the nuance and of the resilience that human institutions tend to bake in. Do it too little, you get chaos; do it too much, you're destroying the very thing you were trying to nurture.

It's certainly deformation professionnelle, but the parallel with software is eerily relevant.

--- [0] https://en.wikipedia.org/wiki/Seeing_Like_a_State

I don't set out to do automation but the automation eventually finds me.

I start making a tool for myself to track a tedious or error prone process. Even after you've gotten the wiki to have the correct steps in the correct order, you can still run into human error. I use it and work on it until I stop burning my fingers. Then I share it with a few people who complain about the same thing, and see how they burn their fingers.

Then I roll it out to a wider team, wait for more problems, then suggest it for general use. Usually by this point we are seeing fewer problems with the tools than with humans doing things manually, so we escalate to teasing people to use the tool so they don't end up with an Incident. Then it becomes mandatory and anyone who still refuses to use it is now risking PIP-level trouble if they don't join the party.

People who try to skip steps make not only a lot of problems for themselves but also for anyone else who suggests automation tools. Running toward the finish line makes everything worse.

Related - I highly recommend reading Lisanne Bainbridge's paper "Ironies of Automation" which points out why we need to keep human factors in mind when designing automation.

http://www.complexcognition.co.uk/2021/06/ironies-of-automat...

This becomes even harder if your execution environment is non-deterministic by definition.

Imagine test automation on physical devices (ex-founder of a mobile app testing SaaS company here) where system0level events that can't be programmatically detected or dismissed can delay, interrupt or alter the outcome of your tests.

Dealing with exceptions to make execution outcomes more stable seem like an obvious choice, but what if it's not feasible due to the complexity of the environment? We found that accepting variance and extrapolating test results from multiple executions in the same (or identical) environment was the only way forward.

It doesn’t have to be so hard, though. One thing you can do is learn to just say “no”. Such as no to cash on delivery.

Being flexible is good, but it comes at a cost. When that cost is too high, don’t do it. Realize that the customer who wants that workflow probably isn’t going to be your make-or-break moment. And in fact, they might have flexibility of their own. For example, you don’t accept Amex but they have a backup credit card or cash. It might be annoying to them if they are fussy, but it’s normal enough that the consequences are minimal. And yes, you may occasionally get a customer who doesn’t have that flexibility, but you shouldn’t be pinning your business on rare events. Figure out what’s most common among your target customers and support a few simple workflows. Say no to everything else until you have the resources to do it properly.

Another thing you can do is have a hybrid low tech/high tech solution. Automate structured inputs with software. Write down the rest as notes in a logbook. Over time you will probably see patterns in the logbook that you can automate.

Lastly, remember what Morpheus says in The Matrix, “Some [rules] can be bent. Others can be broken.” For example, you could simply pay the bill on behalf of the customer who wants cash on delivery. Now you assume some personal risk but the computer system doesn’t have to support their workflow. Is it worth it? Maybe, maybe not, but it’s a choice you can make.

Much of the pain can be reduced by considering if you should bother automating the thing in the first place.

Some kinds of automation are absolutely essential. Aligning wafer stages during photolithography, shipping logistics, high frequency trading, etc. The business absolutely wouldn't work without it.

The other kinds of automation are more questionable. Developing E2E CI/CD automation for an internal tool that is redeployed once every quarter might be a more difficult sell to management. For these cases, even if the rate of manual process invocation is somewhat high (i.e. frustrating for some employees), the customer can't see it. They won't pay one additional cent for this.

There is also this entire meta time wasting game where the mere debate about "should we automate" takes up so much resources you could have manually invoked the thing more times than it would ever be feasibly invoked automatically during its projected lifetime. Alternatively, you could have just let your developers build the damn thing to get it out of their systems.

We don't automate things because it's easy. We automate things because we thought it would be easy

> My favorite example of the latter is how the arrival of IBM computing in the 60s and 70s totally changed the definition of accounting, inventory control, and business operations. Every process that was "computerized" ultimately looked nothing at all like what was going on under those green eyeshades in accounting.

How did you arrive at that conclusion? Most of those applications were written by the businesses themselves. IBM provided you a programming environment not a set of predefined business software. Which is why COBOL jobs still exist.

What changed with the mainframe was that instead of having a bunch of disparate and disconnected processes you now had a centralized set of dependent processes and a database that provided a single source of truth.

Businesses were going to want this capability regardless of how limited humans with green eyeshades were previously.

> Much of the early internet (and still most bank and insurance) look like HTML front ends to mainframe 3270 screens.

Well, precisely, those are custom applications. If they weren't we wouldn't have this issue. You talk about automation but you seem to have not noticed that mainframes have CICS, why it exists, and why businesses still use it.

The "old school" ways are actually still exceptionally powerful.

> Then one day the business owner (my father) said “hey this customer is going to come by and pick up the order so don’t print a shipping label.” Yikes had to figure that out. Then one day another customer said “hey can we pay cash on delivery?” C.O.D. used to be a thing. Then I had to add that.

What happens if you just say no? I have a feeling a lot of complexity stems from the fact that these exceptions are easy to handle when you’re doing things manually, so it’s hard to say no. But if you think about the business case I would not be surprised if the correct answer is no more often than not.

This is demonstrably correct, to me, but I'm curious what makes this unique to software?

Hardware automation has had obvious boons to places it works. And often we will force the places we want it to be to change to accommodate the automation. Agriculture is an easy example of what I mean here. Take a look at some fields that are optimized for giant machinery. Very impressive and can manage a lot. My favorite recently was seeing the "kickout" effort to prevent bad fruit from entering a hopper.

To that end, is the main issue with software automation that it thinks it can be a total solution and often tries to start by not imposing changes on what is being automated?

Always reconsider the process itself before you automate it. Human based processes are different than computer based ones. Trying to directly emulate human actions is what can make automation fragile.

That is something that comes from experience - automate happy flow so you can handle all normal cases super fast - handle edge cases manually.

It always was the biggest issue I have seen “sorry cannot help you with that because our system says so”, which should never be the case. There should always be way around the system. Then of course way around the system needs approval from some higher up anyways.

One of the best pieces on here recently (and didn’t realize it was by Sinofsky!)

Anyway, one other insight I would add is that the issues tend to come up at the interface between the systems. Your automation has to get some input and send some output and both those are pain points. That’s why we sometimes prefer imperfect monolith software.

Automation is about saving one's time and time is something that no amount of money can directly buy more of. One can invest their time and money however to indirectly "buy" one more time through reduction of mundane tasks that many humans have come to accept as normal. Software only solutions aside my own personal and professional experiences have involved building several automation sensor networks and one of the first I built was over 20 years ago that was within my own house. In 2004 my master electrician father and I wired my entire house with both high voltage, 220v/110v, as well as low voltage, 24v/12v, in my plan to build a smart home. My automation planning choices then were focused on exactly this, saving me sub seconds of time here, there, and everywhere. As a former electrician I was acutely aware of light switch placement in a room and the time it takes to turn it on and off each and every time but I asked myself then if my creative awareness could successfully design out the need to turn on a light switch as I enter a room? I did just that using the automation sensor network I built that interacts with source code I wrote which manipulates other devices based on logic of sensor states and any other sensors or inputs I deemed inclusive for the logic's objective. In the last 20 years I have rarely turned on light switches as they come on by "magic" when anyone enters a room and this "magic" is actually real world human time saved for that individual entering said room. I encourage those intrigued by these words to do some of your own math on the time you spend where you reside turning on and off light switches. There will be those here that see it foolish to automate anything, any longshore men present here? There will be those here also that see some benefits to it but having lived in it for 20 years I will never live without it again.

Stay Healthy!

This is an excellent and insightful article, but it feels like it was speech-to-text and the author didn't take the time to clean it up before posting it. It's a little distracting.

https://xkcd.com/1205/

and

https://xkcd.com/974/

Ou est mon scrollbar?

It's important to systemize (get it working manually) before automating.

Premature automation can create more complexity in the process as edge cases are usually added in, or needing routing.

If you're building process management and automation from scratch as a way to learn, it's also a red flag of technical debt waiting.

The thing about software is that it can only ever define its own process. "Software as spec". The goal of automation is to find processes whose architecture and dynamics are already amenable to being represented in software.

Public reaction to automation has been mixed for obvious reasons, but also because when software is applied to structure- and/or determinism-resistant systems it fails to capture some essential components that ends up degrading the services that system can provide.