Sometimes it takes a chupito

Sometimes a bug goes to production, and it happened to us last week. When this happens, it use to be a chain of events, of situations, that happens at the same time and allow this particular bug to reach production. As there is always a human interaction, there is also somebody to blame. Here is how we handle blame inside our team.

I was testing a new feature in my testing environment and I noticed that a button of one of the queues was not working at all, I just clicked and nothing happened. No action, no error, no console message... nothing.

My capybara tests where also having trouble with the button, they could not find it and where failing. As this button was not related to the feature I was testing I decided to look at it later.

Once I got some time to look at it, I checked that there were three features merged on this environment, so it should be one of them containing the bug. I did a full deploy of the first story, this is, the master branch and the first feature deployed on the environment, and noticed that the button was not working.

I went to the developers and explained what I had just done. 'No way, this feature and the failing button are not related' he said.

So I deployed the second story and the button was still not working, so the bug was not on any feature, it was on the master branch and we did deploy one day ago to prod.

The second thing I thought was "How did this happen at all? " and just then, a operator came to us to tell that a button was not working on a queue.

So, there was a bug in production, because a javascript code.
And we don't do unit tests for all our javascript code, we should, but we don't.
Because this was a small release, I did not run all my integration tests, I just ran the smoke tests because we wanted to deploy fast and those should be enough. If I should have ran the complete suite I would have caught this bug.

So, as shit happens from time to time, we got a protocol for fixing things.

First, both developer and tester take a chupito, not too strong, 'cos we still need to fix things, but as a shared act of responsibility acceptance, it is our debt to the product and the team, and we pay our debts. (Skol!)
By doing this, we talk with the team about what just happened, how is the fix and how to avoid this happening again.

Then the dev fixed the JS thing and I changed the smoke suite, so a bug like this won't get his way out to prod next time.

For a middle term solution, it is a need we have to create unit tests in JS, and out from integration tests. We also saw this talk from +Amy Phillips and she gave us some light about the way to follow. Not that we already had an idea, but her talk has helped us putting priorities in place. To fasten the deploy process and decouple the testing and the deploying process are going to be our next goals.

Back to the blame, the chupito thing is our way to celebrate our failure, to avoid blame wars and pinpointing anybody because a bug made it to production. We just isolate, celebrate, fix and deploy every bug we find. Some make the way faster, others take some time, but we don't have a separate bug count from our pending features. As +Antony Marcano pointed out, a bug tracking system is nothing but a hidden backlog.

The Deploy

As you might know, I am the tester guy at peerTransfer. I find myself emdebbed into the developers team. This time I want to tell a story about a deploy, a nice one, a big one.

We use to create short stories about what needs to be done, the usual as a Biker riding my bike, I want my bike to go faster so I can reach my destination in less time.

We create a github issue with such a story, and to give visibility to all the company we use a Kanban board to write this stories down. Then we start a conversation about details as do we mean faster on straight line or around corners, are we still going to stop on the traffic lights, drive by night when there is less traffic... details that help us having context about what is the problem and how we might solve it.

This time, we needed a deep refactor of the operations queues. Our company backend has basically three steps, we collect money from our users, we move it from account to account and we pay to our schools. This is our business, this is what we do.

The refactor was about taking out logic out from the daily operations and create a more complex setup, so to automatize the daily operations, looking for decrease the effort we do when performing such operations. As a biker, I want this bike to go faster.

It soon came out that this story could not be split on several releases, as we needed to do core changes on the site, whenever we would deploy we needed to do it all at once, or at least a big deal at the first time. As a biker, I need to use my bike on my daily commute.

So we created a attack team, this is, a team with people from developer, operations and product teams, this commando hold the needed meetings to define what exactly we were about to deploy, and then, the three devs defined a list of tasks that needed to be done and started coding.

Whenever each one of the developers found some trouble, they paired with another to solve the issue and if any doubt came out, a chat room with the rest of the attack team helped to clarify how things where supposed to work. We built a new engine in the garage, without pulling nothing out from the bike.

At some point, the development branch was ready to be deployed so we reserved a testing environment to use it for testing. We deployed the issue and we checked that the happy path was working as expected. Using a beer can as fuel tank, we started the engine to check how well it was doing.

Time to test. We decided to split effort, so while I was updating the automated test suite with the new features, the ops team member was testing that it was working as expected, for doing this, he created a set of tests with examples of usage and checked that all the results where the expected ones.

We found some bugs that where solved and deployed in no time, and we also found some improvements that would be nice to have on next iterations. After all, this is the first one and for a limited amount of time it is going to be okay to have some rudimentary controls... as long as we build them later. Somehow new stories are quite a valid result of a testing session.

Then we met again. I explained what I did automate, he explained what he had tested, what was working. We came out with new questions, we found that another test case would be nice to automate, and then we had a conversation about how the feature could break. we designed new tests to learn about what would happen is things where badly configured or what could go wrong. We found out that the pass to production would be a tricky question. We needed to prepare the bike before we changed engine. We decided the steps to take.

Our new tests found new bugs, so while we automated the last test, the bugs where solved, deployed and tested.

At that point the build was green and we had the definitive +1 from the operations team.

It was time to deploy. We pushed the button and waited the time to run the scripts, we deployed to staging environment... And the deploy failed.

Well, the deploy went well, but we needed to run a rake task to set things up and this was failing due to some conflicts on the last merge.

The unit tests where all green but the integration tests failed because of the failure on the deploy. Then we looked for the cause, fixed it and deployed again to staging.

This time we had success on the deploy. But the time frame to deploy to production was over, and as next day was Friday, and we don't deploy on Friday.

Uh, well... at least we have a rule that says that we don't deploy on Fridays.

This rule, is a agreement the team made at some point in the past. They all agreed that it was better not to deploy on Fridays, to avoid trouble during the weekend. But time has passed since, and now we have a more automatized deploy process, with better tests and better monitoring. So now we are more confident about the deploying process. More confident about jumping our own rules.

So we deployed on Friday morning, what the hell, this is why we test, we check and we monitor!

And the deploy went fine! Operators started using this feature and now they need less time to perform the same actiona, we found a couple of minor bugs once in production, and none critical enough to justify the time we should have needed to catch them before release.

We also sent an email to all the company explaining the new feature, because we like to communicate when we manage to deploy a big story like this one. We like to tell that our bike runs smooth and faster now!

As a tester, I took a look to the requirements, I automated some smoke tests, I helped designing and performing tests, I looked for the deploy process asking when and how we should deploy without causing damage. All that is testing, all that has a result in the quality of the work that we deliver.

We did a nice work, we did deliver a nice feature called #482, it's time to celebrate!

Credits: 
I found the pic of the Vespa here
The idea of the bike analogy is from this book
The other pics are from our peertransfer office.
The team I work with is a great team!

Out of the loop

For two weeks I been on vacations, quite out of the loop.

And it has been great! Kids decided to grow up. Aksel started walking by his own, Karen is a lady who can dress up by herself and Erik has found out that he can read, and that superpower can be used anytime anywhere!

I got time for taking coffee with old friends and for taking long walks, I think I managed to complete all the stuff I wanted that was not related with a computer :) how great is that!

We also went out for a ride, you know, you are not a biker if you don't ride... just as you are not a tester if you're not testing.

This year 2012 has been awesome. In may I went to London to do the RST training with Michael Bolton and also I was lucky enough to meet Tony Bruce on the London Tester Gathering. On october I went to the Barcelona Testing Open Session where I fired myself as a speaker. I am also being accepted as student on the Miagi-do, we'll see how that goes.
Working as fellow tester in Peertransfer has been a great oportunity to learn a lot of things, just to know that there is so much more to know out there. 

Plans for 2013 also looks great. We are setting up the Valencia Tester Gathering with a little help of many friends. Also the Let's test and Eurostar conferences are on my radar for this year... we'll see how this goes.

So, I wish you a happy 2013, it's good to be back.

If I have seen further...

 ... it is by standing on the shoulders of giants. (Issac Newton)

I'm almost one year in peerTransfer, being the QA Engineer, and I like this company.
I was the first and only tester the dev team ever had, so we have spent many hours talking about what my role should be and what the team and the company is expecting from me.

This has been evolving as we found new needs. I been testing new features we have been rolling out,  and also reviewing the requirements of new stories, taking care of the automated test suite, estimating when needed, testing production setups... and having fun!

Last week the dev team faced a change in the code. Operations team needed to refactor their  process and we needed to make changes in the code to follow along.

But once we saw the requirements, we still had a feeling about... something was still missing. The requirements we're okay, but because the change was big, three devs and I went to a meeting room to try to find out how to roll out the change without impacting the operations flow.

After some debate, we started to suspect that the model we where following was not exactly the right one, we had some discrepancies about what we where asked for and what we allready knew about operations crew was doing.

... so we asked two of the operations guys to join us on the meeting. After a few questions to pinpoint our doubts, I made a simple question:

"Please, can you explain again what is the problem you have."

And there it was, the problem the ops team was clear, and now we understood better the mission of this change.

One of the devs came out with a fast workaround that helped mitigate the impact of the time we would need to perform the change, and by doing so we where buying some time to develop the full feature.

After this meeting, we talked to the product team, to help getting the new requirements and putting the needed story on the kanban board, and start working on the solution.

This meeting was not scheduled. The time we spent on it was not measured in order to keep track about how much time each of us spend at the end of the month in meetings, we did not have a previous plan for the discovery, there were no managers managing the meeting.

We where just a bunch of professionals trying our best to get a solution...

and we did it, or at least, we came out with a plan, and the willingness to try our best.

you see, working with these people make me proud. It feels nice when you are standing on the shoulder of giants and you can see further.

How to do a conference.

This month I will be attending two events. the Barcelona Testing Open Day and the VLC Testing. So while I'm riding the train to the first on them, and given the effort of attending to them, here goes the list of the things that you should do when attending to any testing event.

First of all, this list is not mine, is Matt Heussers, so I'm just writing it down here for my pleasure of reviewing the points.

Find the original here. http://www.softwaretestpro.com/Item/5689/

Before the conference:
Read the brochure.
Pick up a 'vibe'.

Find a problem the conference might solve.
Attend with the intention of learning a solution.

During the conference:
Get the information you need
Conferences are for conferring, don't forget that.
Focus on your mission, not in the conference agenda.

On the ride back home:
pick three things you should do different based on what you have learned at the conference

Think about it!

One piece at a time

Last winter I bought a LEGO airplane for my kid. I had the box waiting for a rainy Sunday evening, and when that happened, we spent the evening putting pieces together. But as winter was gone, the pieces ended up in a box, with many others, and the instruction sheet was also gone for good.

But somehow, the pieces remained in the box, and this weekend my kid and I talked about building that plane again.

So we downloaded the instructions from the LEGO site and today we where supposed to build it up again.

Once I got back from the office, we opened the laptop with the instructions, we searched in the box for some pieces and there we went...

But we had not all the time of the world, so at some point I guess I started building the plane by myself, you know, just to get that done.

Then this 4 years old fellow told me... " Dad, I got an idea, how about you giving me the pieces so I can put them together. Just explain me how I have to do it, and I'll get it done."

Darn it! he was right, I knew how to build it, but it was not me who wanted to learn, that was him gently asking for it.

I passed what I had in my hands to him, and started searching in the box for the next pieces.

After some time, we managed to get all the pieces and build the plane. For this, we both where proud.

"Brrrmmmmmooarrrr"

Being a tester is a constant need to learn. Because there is so much you just don't know, that at least you should be learning something in order to become a better tester, and to get proud of the work you are able to deliver. And to learn, you need to practice, and you also need somebody to teach you, to help you think.

Rob "Social tester" Lambert has written a series of posts in his blog about testing a website. Each one of those articles are about something to take care, or something to know when you should be testing. And not enough of this, he has putted all the 36 pieces in a box, in a free ebook that you can download over here:

http://thesocialtester.co.uk/36-days-of-web-testing-the-ebook/

I was folowing the blog for some time, and I even opened some bugs with the advices found there, so all I can say is...

Thank you Rob for finding the pieces and passing them over.

From scrum to kanban

This morning James O'Sullivan wrote: "An agile team without the intense need to improve themselves at more than a superficial level is not an agile team."

And you know what? he is right, and I'm going to tell you a story.

When I started in peerTransfer, the team was doing scrum and I was the first tester they ever had. The stories where written in Pivotal and devs used to push to staging once every week. As every dev was developing in local, it made sense to put me to test in staging environment, along the product team, but this had some drawbacks.
- If a feature was finished on day one, it had to wait all the week to be released to staging, and since having developed features waiting for being tested is a form of waste, this was no good.
- For my point of view, I was always one week late, because I had to wait, when bugs started to arise, the devs had to go back to the code and try to patch whatever came out before it went to production.
- As I had one week to test, I needed to prioritize my testing, the more risky the feature, the sooner I had to start, and guess what, sometimes bugs came out because the features that had less priority had less time to be tested.
- If the pass to staging got delayed one day, for whatever reason, I had one day less for testing, because product team wanted to have the features rolled out to production on time.
- If one of the features went really bad, the whole release could be impacted because of this delay, adding pressure to the team.

We could do better.

So the devs implemented a board available on our site,  because the one with post-its was not suitable, being half of the team in Boston and the other half in Valencia. Then we moved the Pivotal stories that where ready to develop... and we deleted the rest of them as the future is yet to define!

Now, every story gets developed in a branch. this branch is deployed to a clone environment so I can test the feature. Then we merge and then we deploy.
I get stuff to test right out of the oven, and if I need some help, the devs are ready to pair test the features, because they still remember what they just done, and they still have not started with anything else.
We deploy several times every day, so if one feature needs more time, because of bad requirements or bugs being found, we still can roll out the small features or patches and keep testing the features that are giving us trouble.
We develop out own tool for the kanban board so we don't have to pay license for it, and we are able to develop whatever product or business people needs to keep working, we are a software developing team, and that is what we do!

As a tester, I must have some principles, some rules to follow. The ones that I know that make sense for me, are James Bach's Testers Commitments . And since we do kanban, we all do better, even I do.

and still, we all know we can do better :)

Hacknight at peerTransfer