DevSecOps - Pull the Andon Cord

American car manufacturers found themselves caught off guard when they started to lose market share in the 80s. Americans were losing to Japanese manufacturers who could offer something they couldn’t: both price and quality. How did the Japanese do it? The Americans wanted to figure it out so they sent analysts to Japan where something curious happened, the Japanese welcomed them with open arms. It seems the Japanese were excited to share what they had learned.

As the Americans studied the Japanese manufacturing line they happened upon something that was very counterintuitive, they happened upon workers pulling a cord to stop the line and they called it pulling the andon cord. When the line stopped a supervisor would run up and do the most unexpected thing, they thanked the worker for stopping the line. The American analysts couldn’t understand what was going on. There was no berating, no chastising, it was a bizarro world where the supervisor wanted to impact productivity.

Things got even more confusing when the American analysts researched why the worker was pulling the cord. It seems the worker was pulling the cord when they found a quality issue OR when they had learned something that should be shared. The Americans were dumbfounded at the practice considering their approach was to test every unit at the end of the line for quality failures, they’d never think to stop the line. And this learning business? Isn’t that what training is for? Surely this was a red herring. Surely these Japanese competitors were trying to pull one over on the Americans.

The Americans interviewed the supervisors and asked them more about this andon cord nonsense; they found it wasn’t a red herring, they really believed in it. They believed in it so much that they tracked how many times an employee pulled the cord, and were very concerned if that number went down. The Americans were appalled. They wanted the line to stop more? How could this be? Here’s a competitor competing on price and quality, yet they’re stopping the production line hundreds if not thousands of times a day? It didn’t add up.

The Americans were falling victim to the cargo cult mentality, they were focusing on the how and not the why. For Toyota the why came long before Toyota was known for making cars, the why came when they made looms, maybe even sooner. Here’s what Sakichi Toyoda had to say when design plans for their flagship loom product were stolen.

Certainly the thieves may be able to follow the design plans and produce a loom. But we are modifying and improving our looms every day. So by the time the thieves have produced a loom from the plans they stole, we will have already advanced beyond that point. And because they do not have the expertise gained from the failures it took to produce the original, they will waste a great deal more time than us as they move to improve their loom. We need not be concerned about what happened. We need only to continue, as always, making our improvements.

The “why” of andon cords was about learning and improving towards the pursuit of eradicating inefficiency and eliminating waste. That pursuit would lead towards practices the Americans didn’t understand like Kanban and Andon Cords.

Toyoda Automatic Loom Works would be sold off in 1932 and the Toyota family would move on to manufacturing cars, they’d become Toyota as we know them today. Taiichi Ohno, who started as a line supervisor, would eventually be promoted to executive and became the public spokesperson about how Toyota worked, about Toyota Production Systems.

Much like Deming, Ohno was a gold mine of ideas like his “Ten Precepts”.

  1. You are a cost. First reduce waste.
  2. First say, “I can do it.” And try before everything.
  3. The workplace is a teacher. You can find answers only in the workplace.
  4. Do anything immediately. Starting something right now is the only way to win.
  5. Once you start something, persevere with it. Do not give up until you finish it.
  6. Explain difficult things in an easy-to-understand manner. Repeat things that are easy to understand.
  7. Waste is hidden. Do not hide it. Make problems visible.
  8. Valueless motions are equal to shortening one’s life.
  9. Re-improve what was improved for further improvement.
  10. Wisdom is given equally to everybody. The point is whether one can exercise it.

Alright, so what’s it all got to do with security? As a security practitioner I know other practitioners consider security something unique, maybe even special. I hear there’s a shortage of “cybersecurity experts”. Is there a shortage of quality experts? Why is security special? Isn’t it just one of the quality ilities?

Security is rarely a first order need for a customer, it’s a second order need, it’s a quality need. When you start to see security isn’t special you can learn from the past and apply what works for solving quality problems. Andon cord is just a concept, a concept that you can empower and encourage everyone to call out quality/security issues, take what they’re calling out, what they’re learning, and feed back into the organization. We need smart cows in our herd, but they’re only beneficial if the herd gets smarter.

So how do you do it? Odds are your organization already has something in place in the form of an incident management process or a defect tracker. Build on what’s in place for quality control today, figure out what works, and improve every day. Build a process that doesn’t blame humans for mistakes, build a process that understands “Every system is perfectly designed to get the results it gets”, and institute a continuous improvement process that moves you towards a desirable target condition.