As financial services grow more complex — think information delivery to tellers or brokers, call centers, web sites and mobile phones while checking available balances, confirming identity and tapping silos of data to build a better customer experience — firms are automating quality assurance (QA) to keep up.
Quality Assurance (QA) is important in software too. (Photo by: Universal Images Group via Getty ... [+]
“It’s hard to scale up manual exploratory testing,” said Antony Edwards, COO of Eggplant, which provides automated testing tools. “It’s hard to find good testers and they can’t keep up with DevOps. We need to find a way to combine the strength of exploratory testing with automation — a way to combine manual testing, which is smart but slow and expensive, and automated which is fast but stupid.”
Firms need customer-centered testing, he added. “The goal is a great user experience at speed.”
Pyramid Solutions, an Eggplant partner, did a survey of executives — 70% internal IT and 30% ISVs, to see where they were in automating QA. One surprise is that people with QA in their title didn’t necessarily do QA.
“For many IT professionals—even in multi-billion dollar global organizations—QA seems to be seen as a stepping stone to another position, or a place people go to retire,” wrote Carl Johanson, QA practice director at Pyramid. “To truly improve QA automation, organizations need to fill their teams with technically skilled professionals armed with deep business knowledge.”
Organizations have high hopes for automated QA but results fall short, he added. “At every layer, organizations want to achieve 80%+ automation in 2 years, but have only achieved about 30% automation to date. One of the largest gaps is on automating testing of the UI layer, which is surprising since this is where most companies begin their automation journey."
Companies that are successful in automating QA tend to have QA professionals with 20 years of experience or more and they have a professional QA testing function, rather than internal user testing teams. Johanson thinks companies need QA separate from development.
”Developers believe everything they write is gold and they have always been reluctant to put a lot of effort into testing.”
As software has become more complex, companies need experienced QA people who can see the entire app, not just the piece a developer knows well. They also need to know the business.
“QA can validate down to the database level not just the UI. It will be difficult for developers to become generalists on the entire end-to-end solution when the software has gotten very, very complex.”
Johanson said the survey was to learn what mainstream companies are doing, not companies like Google which may deploy new code multiple times a day. QA sessions that share knowledge about where companies are in the journey to QA automation are popular at conferences, he added.
“People like to see their company isn’t as far behind as they thought they were.”
When he started QA in the 80s, the leading uses were in nuclear or defense. Then in the 90s financial services and telecom came along. The stakes were high — a missing semicolon in some AT&T code shut down the nationwide network, he said.
“Finserv tends to be a little bit ahead; they always have been. In his book, “Thingalytics”, Eggplant CEO John Bates cites cases where errors in code were costly. In 2102 Knight Capital turned on a new trading code without first removing the old code. The code launched $440 million of trades in 45 minutes, driving the firm to the brink of insolvency. In June 2010 a Deutsche Bank trading algorithm in Japan took out a $183 billion stock sell position; the bank managed to unwind most of it and then shut down its Japan based proprietary trading group. Better testing could have avoided the massive sale, Bates said.
In the same year a currency-trading algorithm at Rabobank placed a $3 billion FX order. Better testing, real-time risk rules and trade monitoring could have prevented that, he added.
