How can you use test metrics to improve your testing team's efficiency and effectiveness?

No ciclo de vida de um produto, desde sua concepção, são estipuladas métricas que se tornam indicadores de eficácia e/ou aproximam os marcos do projeto. Ainda mais importante que a própria implantação de tais métricas, se dá a atenção durante a sua criação. Para isso, podemos utilizar de diversas ferramentas. Mais comumente, metas S.M.A.R.T. (específicas, mensuráveis, alcançável, relevante e temporal).

Test matrices should also include estimated and actual efforts, which will give a clear idea, that how much efforts you should plan for the same test case in the next round.

Test metrics should include stress testing. On one project I managed, the stress testing highlighted a significant transaction processing problem that would have been a disaster in full production. It could only have been discovered in very high-volume transaction mode and would have been impossible to duplicate in a test environment otherwise.

Test matrices are really dynamic and depends upon the type of project one is working on. It could be both qualitative and quantitative. For example if you are working on a product development project, regression testing and feedback loop is most commonly used Whereas, if it is a process change you might want to assess success through surveys and sentiment testing This is probably a vast topic and can be better understood in context of a unique project.

Em termos de métricas de testes, tive a oportunidade de presenciar alguma dificuldades para solucionar. Em novos produtos, a engenharia de desenvolvimento acaba por optar por tolerâncias mais apertadas. Por outro lado a engenharia de processo, em muitos casos, alerta sobre a capacidade do processo atual de atingir os níveis adequados (Cp/Cpk). Nestes casos é importante termos a Engenharia Simultânea para que o novo projeto seja adequado, atendendo as métricas de desempenho (produto) e capacidade (processo).

As métricas de teste são importantes para poder ter uma visão mais acurada do processo. O monitoramento das métricas permitem avaliar o quanto o seu processo ou produto está próximo do que é esperado e portanto essas métricas precisam ser definidas em pontos considerados chave ou pontos críticos do processo para que o monitoramento seja eficaz e eficiente.

When talking about metric it all starts with proper definition of the metric. It needs to be clear, attainable and you must be able to influence with your actions taken. Not having any of the things mentioned above is one of the key reasons that brings frustrations and make a metrics a headache instead of something that will help you improve your process or product. If you’re able to set clear metrics and goals, it will drive your team to a continuous improvement culture which will always turn into greater customer satisfaction and return for your company.

Without a proper monitoring, the testing process can drift and become a wasteful process. To keep an eye on test activity anticipates future situations. For instance here are common issues: *Tests are taking too long: the chosen process is not scalable and is slowing down release process. Slow steps should be identified and updated asap. *Tests are focused on a minor part of the software or are duplicated : current test coverage should explicit and homogeneous. Even post production “defects” metrics can help to understand and identify testing process failures

As métricas de teste são importantes por várias razões: 1) Atender aos requisitos do cliente (desempenho, qualidade e custo) 2) Definição de investimentos necessários 3) Lead time de desenvolvimento 4) Recursos necessários pela organização

It is said that if you can't measure it, you can't manage it. Testing or not, if you don't quantify the outcome, it is impossible to know the effectiveness of a process. Testing is just one such process. Depending on the metric, process can be tuned. For example, if you measure number of bugs escaped to the field, better test can be designed (but not time taken for testing.)

The test metric will also depend on the product or nature of the operation. For software in general,  We need to perform static analysis (SAST). with code quality (bugs,coverage, code smells, Cyclomatic complexity)  security vulnerabilities  third-party library vulnerabilities. Black box test cases:  functionality DAST: Dynamic Analysis Scanning results against common vulnerabilities. Proxy attacks: number of attacks, successful attacks and apart of all this Good Test Database.

As métricas são definidas em função do produto, por exemplo: 1) Na engenharia de produto as métricas são as definições de tolerâncias, testes funcionais, testes de durabilidade (confiabilidade), etc. 2) Na engenharia de processo são as métricas de capabilidade de processos como os índices de Pp/Ppk, Cp/Cpk, Cm/Cmk, tempo ciclo, takt time, etc. 3) Para a qualidade são os índices de taxa de defeitos, PPM, etc.

My suggestion is to listen to your customer's voice. Are they pained about unfriendly interface? Number of bugs? Instability? Too in/frequent updates? Start with 5-10 parameters that you can sell to other stakeholders. Too less and you are missing, too many and you are likely to become bureaucratic. Start achieving results and let every stakeholder enjoy and take the notice of the success till these become a habit. Then add a few more metrics. Think of an open cast mine - they keep removing layer by layer, including that of the desired mineral. Every layer requires different machinery, blasting approach, disposal/collection mechanism.

A melhor ferramenta que consegui ver aplicada em encontrar as métricas adequadas e que se adaptem a cada projeto e/ou regra de negócio é o uso de metas S.M.A.R.T.

Para as definições de métricas, geralmente adoto uma abordagem feitas por multinacionais japonesas, por exemplo, para características de segurança um Cpk mínimo de 1,5. Para outras características Cp/Cpk de 1,33. Porém tudo vai depender dos requisitos específicos de cada cliente! Estes tipos de cálculos já estão embutidos em diversos softwares como Minitab, SPSS, RProject, etc. E também já apresentam gráficos apropriados para apresentação imediata dos resultados.

There is no alternative to a good, solid test database. Most of your tests metrics would arise from the well-recorded history. Generally a time-line and results record against history are good to begin with. Even in the absence of good voice of the customer, running coverage and result data across time and releases gives a decent idea of where the shoe pinches. Is it worse around holidays? Is it worse when tested in a location or a person? Is it worse when too many release run in parallel? Is it worse when automated? Per release time based plots, per feature time based plots, percent of coverage per release per severity etc. are some such examples. With a decent understanding of graphs and statistics, you can interpret and present the data.

1. Typically bottlenecks manifest with Pareto behavior. Discuss those with your team. They know a fault equipment, too much load, too lax specification, sloppy coders, ... 2. Also, look out for unreasonable expectations. I have literally seen a team of 3 serving 5 products and quality suffering only when 6th product was introduced. 3. Look out for process optimization. Why should not that automated test suit trigger as soon as the build is ready? Why should manpower be spent in updating test results that passed? 4. Look out for those patterns. Which is the most unstable part of the software? Is this release touching that part? Do I need to move more senior hands in testing that part to de-risk the release?

Incorporate customer feedback related to product quality into the metrics. This provides a real-world assessment of the testing effectiveness. In my experience, aligning testing metrics with customer satisfaction has provided a comprehensive view of the product quality & areas for improvement. Assess the effectiveness of test case prioritisation. This ensures that the most critical tests are executed first. Measure test flakiness, which indicates how often a test yields different results under the same conditions. Reducing test flakiness is key to reliable testing. My approach has involved regularly reviewing and updating flaky tests to ensure consistency.

Watch out the black sheep. While most testers are good at heart and able at hands, some aren't. In one case, I had an apparent superstar, finding bugs at a blazing rate. Only when I dived into the details, I found that all such heroism was in testing code written by a single coder. The coder on the other side of the wall was also being celebrated for fixing the highest number of bugs. That was a collusion! The coder used to keep bugs in the code for the tester to find and then fix them. We decided to separate this top 'performing' pair. The performance degraded enough for both of them to be on PIP and later, out of the job.

The metrics will be able to say many things from your current product but it will may falsely give the impression that the product was good, for example when the percentage are high. But , it shall be take into consideration that maybe it has been tested only what have been realized by the author ( for example SW Engineer) or even the understanding from a SW tester. And it would be not enough. A robustness will be achieved when the residual risk will be measurable lower enough to conclude that the overall product has been fully covered.

Acredito que um foco importante que precisamos ter em nossas organizações são os bancos de dados sobre as estatísticas de testes. Principalmente para termos uma referência para a melhoria de nossos produtos e processos! Tornar o conhecimento implícito adquiridos durante os testes e transformá-los em explícitos para que qualquer profissional que pesquisar neste banco de dados tenha uma referência de projetos anteriores, o famoso "Lessons Learned".

Para implementar efetivamente métricas de teste, é crucial estabelecer indicadores-chave alinhados aos objetivos da equipe. Utilizar ferramentas automatizadas pode simplificar a coleta e análise de dados, permitindo uma avaliação contínua. Além disso, promover uma cultura de feedback e colaboração dentro da equipe é essencial para interpretar os resultados e implementar melhorias significativas. A busca constante pela adaptação e refinamento das métricas contribui para um ciclo de testes mais eficaz e eficiente.