My team was technically strong, so they have decided that the testing strategy would include only automatic tests. The scope of the project was initially closed. So, the risk that tests would have to be reimplemented was low, because of the change requests. This was a good start.

To make the tests implementation faster, automatic tests were immediately developed for every component. By doing so, whether it was an API, screen, event consumer, bpm process, or other, the backlog item was instantly closed. How wonderful!

The team implemented more than 5000 automatic tests, correcting some issues. And we delivered the project to the client for the User Acceptance Testing. We were confident that this would be a success!

A few days’ time, we started getting not so good news… in fact, bad news. Some bugs were found, there were several unmet requirements, problems in processes that included more than one component, and some status machine had issues. Finally, some solutions met the requested requirements, but some requirements no longer made sense to the customer.

We were stunned; we had implemented automatic tests for all the components. Why had this strategy failed causing so many problems?

To answer that question, we needed to analyse the methodology used to manage the project scope and its deliveries. And the test strategy.

So, we went through all the requirements with the client. It looked like everything was perfect, but because we hadn’t done this in the first place, we caused ourselves some problems:

• Some requirements no longer made sense for the client when we implemented them. If we had reviewed them with the client, we would probably conclude that other requirements would make more sense, or the same ones but with some differences.
• How could some requirements not be being met, despite having implemented a solution for those requirements? Because in the beginning of the project we did not understand the real needs and pains of the client (we thought we had!). If the topics had been revisited during the project, we could have had a better understanding of their needs and more knowledge on their business. Continuous communication with the client allows us to understand their real pains.
• We developed the project using sprints, but we only delivered all features to the client in the end of project. Hence, we did not receive any feedback during all the development phase, and because of that we didn’t adjust the features to the client’s real requirements.

Automate didn’t kill manual tests.  

So, about the test strategy, we made a huge mistake: to believe that having almost 100% coverage of Automatic Unit Tests (functional tests) would result in almost zero bugs. We were so naive!

When we automate tests we need to ensure that the test is correctly implemented and really tests the requirement. When this is done by the person that develops the feature, then tests are influenced by this person’s knowledge about the implementation, and not by the requirements.

For example, if the developer misunderstood a business rule or implemented it wrongly, the test that he’ll implement will be according to his understanding, so it will be wrong and it won’t validate the business rule when executed.

If another person implements the test, they will look for the business rule specification and do the test without being influenced by its implementation.

If the test is not correctly implemented, this is not an issue because it will fail when executed. Someone will analyse the feature (and the test) and will fix the component with problems.

On the other hand, unit tests are not enough. Integration tests and system tests are essential to ensure that the system works holistically. Even so, testing each component singly with a 100% coverage, does not mean that the system has no bugs.

In fact, there are still bugs that can exist, like errors on the design or specification, incorrect assumptions about the meaning, units, or boundaries of the data being passed between, failures in message interpretation between systems, or other.

To mitigate this, every time an interaction between two components, systems, packages or microservices is developed, a set of integration tests must be executed. These tests are strong candidates to automation. 

Even when all integrations are working, the system testing must be executed because it “focuses on the behaviour and capabilities of a whole system or product”, validating that the system is complete and all processes work as expected.

For example, in a state machine implementation, sometimes some non-final statuses are forgotten and the transition event is not implemented. Other example is in event-driven systems, when sometimes an event is produced but no service consumes it and the process remains unfinished.

These kind of problems are usually found by executing system testing. These are the most important tests for any business, in my opinion. As ISTQB syllabus teaches us, testing includes checking whether the system meets specified requirements, but not only. It also involves validating whether the system will meet the client and its business needs in their operational environment(s). We need to understand if one process or functionality makes sense and works end-to-end.

What knowledge base to use? 

Automatic testing is a strong testing tool, but it cannot replace all manual tests. We need to evaluate the whole system characteristics and environment to understand what tests we should execute manually, and what tests we should automate.

To decide the test strategy it’s important to understand the advantages and disadvantages of each approach.

Manual testing is a knowledge science activity that includes human judgment and capacity to analyse if something makes sense. If we do not execute manual testing, we lose the advantage of experience-based test techniques, and the human capacity to analyse if the solution makes sense for the business and meets the requirements.

Manual testing give us a chance to find extra bugs that automated tests would never find, because it allows us to follow that gut feeling that “something smells bad” and explore themes that may not have been tested or required.

Even when we decide to implement automated testing, we should test the software manually before running the automated tests. First, to be sure that automation is possible, and then to ensure that this automation is correct.

Automated testing is an exact science that ensures accurate results. It is software testing other software.

Testing a new feature manually could be fun, but testing the same features time and time again to prevent regression issues can be demotivating, leading to frustration and wasting time. That is why regression tests automation is a very important mechanism to save money and spare the test team.

And how about performance and load tests? It is insane to execute those tests manually! You could try but you’ll take a lifetime, and probably the tests won´t be accurate and the test coverage will be smaller. The greater the software and the more stable the feature scope, the greater value of testing automation.

When a part of the testing process is automated, you have many advantages:

• The productivity increases because the test execution is faster.
• Your confidence grows since you’re more reliable and error proofing.
• Your team is more efficient because the tests after implementation are repeatable without human intervention, and the team can use their energy on non-automated tasks.

Check the following table to see when each test is more suitable:

Bet your money on the right testing.

So, the one-million-dollar answer to the question on how 5000 automated tests were made and the project failed is simple: the test strategy was wrong!

We need to find the right balance between automated and manual testing, because each one has strengths and weaknesses. Their value only exists when they are applied in the right environment.

As an agile company, Polarising implements automated tests to save teams, time, and above all, to deliver high-quality software. However, to ensure that the solutions bring value and are the right ones to your business requirements, manual tests are also performed.

Márcia Catarino
Business Analyst

Links
https://www.testim.io/blog/test-automation-vs-manual-testing/
https://www.guru99.com/difference-automated-vs-manual-testing.html
https://www.testingcompany.com.br/blog/teste-manual-de-software/
http://www.tecnisys.com.br/noticias/2019/teste-manual-vs-teste-automatizado
https://www.softwaretestingmaterial.com/automation-testing-vs-manual-testing/

We already read about the benefits of remote work, they are undeniable; at Polarising is not new to anyone, as we have the flexibility to do it when we feel it will improve our performance.

But if there has been a general underestimated idea of this context, now we know how equally important both office and remote work are to that work-life balance companies and employees are always advocating.

So, Polarising decided to re-open the office in that assumption: just because things cannot be as they were, it does not necessarily mean we have to exclude any of the options!

República’s office is open since June 1st with limited capacity, and for that we had to come up with a solution that allows anyone to see if there’s place available to go and take care of business or just have a relaxing day outside home.

The challenge.

HR, LABS and Communication teams were already working together to prepare everything, and there were some ideas and possibilities on the table. Finally, I was contacted and asked to develop a registration app. There was a catch, thou… I had two days to do it!

This app had to be a quick and effective solution that allowed each one of our employees to book their day/ place and parking space in the office building.

A few months ago, I had decided to learn a little about Microsoft power apps, an Office 365 tool in which we all have a Polarising account. The concept is to develop no-code applications for PC, Tablet or Mobile. The data sources of these applications can be simple excel files, sharepoint or database repositories.

Another advantage is that it’s very easy to publish applications that can be used, for example, by integrating or embedding them in a website or in Microsoft Teams.

The requirements.

This app had to have some specific requirements, of course:

• an app that any and only Polarising employees can access
• a feature that allows to schedule the days at the office in each of the 3 floors and manage the number of people registered on each floor
• when seats run out on a floor, no more bookings can be made for that floor on that day
• it is possible for each employee to make only one reservation per day and to unbook a reservation
• a feature that allows to book and unbook a specific parking space and only one space reservation per day
• it is possible to check Booking and Parking Space details of all registered employees at any time
• booking history must be saved.

The solution.

After comprehending what was needed, I went to work! My biggest challenges were fields like Data and Calendar, as well as using the logged user information in to the App, so I could make the reservation on it behalf, two concepts that I hadn’t yet used working with power apps; and it wasn’t as easy as I thought!

Step by step and overcoming each challenge, I came up with a first draft version of the app; I was ready to run some tests, so I asked the team to do it covering all sorts of possible scenarios, and most important, to suggest improvements if necessary.

The tests went so smoothly and mission accomplished: Polarising has an “express-app” ready to welcome all on June 1st, according to all safety procedures!

Working as a team was fundamental to have this application ready and functional on time.

It’s great to be back!

Márcia Catarino
Business Analyst

Fortunately for me, I’m a Business Analyst and I’m part of the SCRUM team within my company. And from where I stand, a Business Analyst role is important regardless the methodology in use, especially in any software development project.

A SCRUM team is multi-disciplinary; it must include all the necessary skills to deliver each sprint. A Business Analyst can take on a role within the development team or play the role of Product Owner, depending on the needs of each team. I’ve been in both situations.

According to Scrum.org “A Scrum Product Owner is responsible for maximizing the value of the product resulting from the work of the Development Team.” This means his main goal is to deliver value and to do so it’s important to understand the customer’s business, its processes and needs, establish the main objectives and prioritize. Finally, using the time and budget available to deliver the maximum value with the help of the development team.

I can’t help but think that it’s the same definition of a Business Analyst and that on SCRUM the Business Analyst is the Product Owner!

After understand the customer needs, a Product Owner (Business Analyst in SCRUM) is responsible for:

• Feed the product backlog which is the artifact that represents the understandable object; it is what represents all the necessary changes to maximize the value to delivery.
• Identify which backlog items will bring the most ROI possible to the customer’s business, prioritizing the backlog items.
• Follow up with the customer on every interaction, so he can know if what’s being delivered is really bringing value and help him understand how to increase processes performance.
• Ensure that each item of the sprint backlog includes all the necessary information, so the development team can address all customer’s needs.
• Know how to say NO to the customer and the development team when needed, be realistic and make decisions.

You also hear that SCRUM has no functional specifications because the team must be agile. But does this mean that an agile team can’t grab a piece of paper and pencil and draw a context diagram, a process or an algorithm, to better understand what to implement? Is it required that the solution is in everyone’s mind to be an agile team? I’m sure it will depend on the needs of each project but for me it works this way:

• What documentation will bring value to the customer?
• In what way the development team will have better understanding of the job in hands?
• Documenting in detail will bring further value?

Only when answering these questions, I can decide on how to detail each backlog item. The specification of a functionality can be a user story, a flowchart, a state machine, an entity diagram, a use case, a picture of a paper drawing and so many other things. The Product Owner is responsible to analyse and decide what make sense and each situation.

When is clear that the Product Owner cannot deliver all specifications needed he could decides to include a Business or Functional Analyst in the development team. They can assist in product specifications tasks and even in the quality assurance process (specs, test specs and test execution). This means that whomever takes this role becomes part of the development team, having tasks assigned in each sprint. The detailed specification of a feature must be made in previous sprints prior to the feature implementation or even in the same sprint.

Same applies to the test process; it’s the Product Owner’s job to prioritize each task in order to maximize the delivered value, as well as the deployment team productivity.

By using SCRUM we’re able to iterate, delivering smaller chunks and getting feedback along the way on how customer needs should be addressed.

If we compare traditional approaches with agile methodologies, what are the main differences for a Business Analyst? According to Dave Saboe, the way to change the mindset of an Agile Business Analyst is:

• understanding that conversations matter – conversations are more important than documentation.
• expect change – In a traditional environment, you want to manage change. In an Agile environment, you expect change.
• Focus on customer value – Understand what the customer wants, his needs and exploring various options, being able to prioritize work based on that value.
• Do just enough just in time – Instead of getting a big design up front before we move along, we work in small increments doing just enough just in time.
• Have a team focus – I need to stretch my skills, step outside my role, and see what kind of support the team needs.  On an Agile team, we succeed or we fail as a team.
• Have a bias towards action – We do just enough analysis just in time and then take action.

Value… I’m always talking about value: SCRUM it’s about Value! Business Analysis it’s about Value!

Márcia Catarino
Polarising Business Analyst

• Learn about the customers business, analyze the current state/environment to discover the reason why a change is needed (what he really needs and not what he thinks he needs).
• Understand what could bring value to the business and define the way to meet that business need.
• Work with the technical team to build business cases with possible solutions.
• Support the definition of a possible architecture for the system.
• Elicitate requirements with the right stakeholders and confirm them before implementation.
• Define and model requirements that are possible to be developed.
• Evaluate the impact and implications of the proposed requirements and design changes before asking the development team to implement them.
• Support the development team to specify the technical solution, helping them to organize the documentation.
• Specify a solution with the right level of detail, so both the development team and the customer can understand the solution.
• Specify a solution in which each feature makes sense individually and integrated with the system overall.
• Be able to follow and to give support to the development team.
• Define the right tests to quickly find the defects that most influence the delivery value.
• Report issues with sufficient detail to reproduce the problem, consult the database, system logs, source code or even debug, helping the development team to understand the problem.

If organizations had someone with this profile in each project they could deliver more value to the customer with less development effort – are they aware of this? Well, Polarising recognizes that competent Analysts and testers are essential for projects to succeed. We make sure that the project delivers the maximum value t the same time we help customers to reduce  their development costs. At Polarising it is possible to build and develop these competencies in-house because we acknowledge their relevance to our customers. So, everyone here knows… Analysts really can make difference!