As you might know, I am the tester guy at
peerTransfer. I find myself emdebbed into the developers team. This time I want to tell a story about a deploy, a nice one, a big one.
We use to create short stories about what needs to be done, the usual as a Biker riding my bike, I want my bike to go faster so I can reach my destination in less time.
We create a
github issue with such a story, and to give visibility to all the company we use a Kanban
board to write this stories down. Then we start a conversation about details as do we mean faster on straight line or around corners, are we still going to stop on the traffic lights, drive by night when there is less traffic... details that help us having context about what is the problem and how we might solve it.
This time, we needed a deep refactor of the operations queues. Our company backend has basically three steps, we collect money from our users, we move it from account to account and we pay to our schools. This is our business, this is what we do.
The refactor was about taking out logic out from the daily operations and create a more complex setup, so to automatize the daily operations, looking for decrease the effort we do when performing such operations. As a biker, I want this bike to go faster.
It soon came out that this story could not be split on several releases, as we needed to do core changes on the site, whenever we would deploy we needed to do it all at once, or at least a big deal at the first time. As a biker, I need to use my bike on my daily commute.
So we created a attack team, this is, a team with people from developer, operations and product teams, this commando hold the needed meetings to define what exactly we were about to deploy, and then, the three devs defined a list of tasks that needed to be done and started coding.
Whenever each one of the developers found some trouble, they paired with another to solve the issue and if any doubt came out, a chat room with the rest of the attack team helped to clarify how things where supposed to work. We built a new engine in the garage, without pulling nothing out from the bike.
At some point, the development branch was ready to be deployed so we reserved a testing environment to use it for testing. We deployed the issue and we checked that the happy path was working as expected. Using a beer can as fuel tank, we started the engine to check how well it was doing.
Time to test. We decided to split effort, so while I was updating the automated test suite with the new features, the ops team member was testing that it was working as expected, for doing this, he created a set of tests with examples of usage and checked that all the results where the expected ones.
We found some bugs that where solved and deployed in no time, and we also found some improvements that would be nice to have on next iterations. After all, this is the first one and for a limited amount of time it is going to be okay to have some rudimentary controls... as long as we build them later. Somehow new stories are quite a valid result of a testing session.
Then we met again. I explained what I did automate, he explained what he had tested, what was working. We came out with new questions, we found that another test case would be nice to automate, and then we had a conversation about how the feature could break. we designed new tests to learn about what would happen is things where badly configured or what could go wrong. We found out that the pass to production would be a tricky question. We needed to prepare the bike before we changed engine. We decided the steps to take.
Our new tests found new bugs, so while we automated the last test, the bugs where solved, deployed and tested.
At that point the build was green and we had the definitive +1 from the operations team.
It was time to deploy. We pushed the button and waited the time to run the scripts, we deployed to staging environment... And the deploy failed.
Well, the deploy went well, but we needed to run a rake task to set things up and this was failing due to some conflicts on the last merge.
The unit tests where all green but the integration tests failed because of the failure on the deploy. Then we looked for the cause, fixed it and deployed again to staging.
This time we had success on the deploy. But the time frame to deploy to production was over, and as next day was Friday, and we don't deploy on Friday.
Uh, well... at least we have a rule that says that we don't deploy on Fridays.
This rule, is a agreement the team made at some point in the past. They all agreed that it was better not to deploy on Fridays, to avoid trouble during the weekend. But time has passed since, and now we have a more automatized deploy process, with better tests and better monitoring. So now we are more confident about the deploying process. More confident about jumping our own rules.
So we deployed on Friday morning, what the hell, this is why we test, we check and we monitor!
And the deploy went fine! Operators started using this feature and now they need less time to perform the same actiona, we found a couple of minor bugs once in production, and none critical enough to justify the time we should have needed to catch them before release.
We also sent an email to all the company explaining the new feature, because we like to communicate when we manage to deploy a big story like this one. We like to tell that our bike runs smooth and faster now!
As a tester, I took a look to the requirements, I automated some smoke tests, I helped designing and performing tests, I looked for the deploy process asking when and how we should deploy without causing damage. All that is testing, all that has a result in the quality of the work that we deliver.
We did a nice work, we did deliver a nice feature called #482, it's time to celebrate!
Credits:
I found the pic of the Vespa here
The idea of the bike analogy is from this book
The other pics are from our peertransfer office.
The team I work with is a great team!
