Team Performance Management Part II
Having set the stage in part 1 of this post, I will now describe specifically how we’ve set it up. I should note that template we have come up with applies to all team members, no matter what their specific roles (QA, BA, or developer).
Team Performance
This is the focus of our performance management document. We using a scoring system (built into the software we use) which can be weighted, so this portion represents 50% of the score. As must as possible, we’ve included quantifiable criteria for evaluation. Some examples of the things we measure:
- the consistence of sprint velocities
- the number of sprint commitments met
- code coverage
- code ‘quality index’ (a sum of several code quality measures)
There are also some more qualitative criteria (positive and negative) which we look at, such as:
- Stakeholders recognize consistent, feasible innovation in providing useful software products
- Software products are recognized by stakeholders as meeting the needs of the business
- Customers regularly raise concerns about team performance, product quality
Individual Contribution to the Team
This section represents 25% of the scoring. Here we found quantifiable criteria hard to come by. Instead we look for certain behaviours our actions (again, positive and negative), such as:
- Contribute to team knowledge by conducting at least one learning session (lunch and learn or otherwise) on a relevant topic in the last six months
- Seek to gain and demonstrate new skills and knowledge to make yourself more useful to the team
- Clearly support the team in achieving its goals; take personal responsibility for and understanding of the team’s goals
- Provide feedback to others in a constructive and insightful manner
- Demonstrate an unwillingness to listen to others’ ideas
- Fail to share ideas and information on your work
Individual Goals
The final 25% of the performance score is reserve for individual goals. Here is an opportunity to set out specific goals and individual may wish to accomplish, or otherwise perhaps goals related specifically to their roles on the team.
And that’s it. We set this up in the summer, and we’re still actually in discussions around what some of the numbers should be, but the structure is there. It’s a little early to tell how it will work, but the concept and approach appear to have been accepted by the team, so far.
Team Performance Management Part I
Like most organizations bigger than a small business, we have a performance management process in which every employee participates. The structure of the process is focused on individual goals and performance metrics. Unfortunately, managing performance at the individual level doesn’t work well in a team environment. In Scrum, peoples’ focus needs to be on how the team is performing and delivering to support the business, not the individual. In fact, using an individually-focused performance management process not only doesn’t work, but can actively work against progressing as a team. As a team member, consider how individual performance assessments would impact your motivation to help the team look good, at your own expense … it doesn’t.
There isn’t a great deal out there on performance management in Scrum or for a team. A lot of what I’ve read just dismisses formal performance management as a waste of time or counter to Scrum best practices. This is especially true when the Scrum Master, who should be in a supportive role for the team, is asked to conduct or participate in performance reviews of team members: it doesn’t work very well. But the reality is most of us have to adhere to corporate policies, and therefore have to follow the process our HR groups have laid out. My current situation in thinking about this problem is that I am not the Scrum Master, and some of the team members report directly to me, and some report to a manager who works under me. I am in a situation where I have to do performance reviews, though, and where I need to these to support team performance. So, what do to, what to do … .
There are a couple of critical issues to deal with: (1) the process itself, as well as (2) any incentive plans that are tied into it. Fortunately (well … maybe not so fortunately) my organization doesn’t have a high-valued bonus system tied into personal performance, so at least I’m not working against that. For this post, I’m going to focus on the second issue: building a performance management document which supports team work.
My perspective on the performance management process is that is gives a manager two tools: (1) a framework to have conversations (sure, that’s something you shouldn’t need a framework for …), and (2) a document which describes how or what a staff member needs to accomplish. This second point is the critical consideration for framing an effective performance management document. It needs to be effectively written to support team-focused work.
Written properly, the document can provide appropriate incentive to perform in different ways, although this depends on the stakes involved (e.g. is there significant financial incentives at stake?). My experience is that different people respond differently to this document — some argue every point and demand concessions, even if it isn’t a high stakes process, and other are more relaxed about it. Nonetheless, even in a low-stakes situation with employees who are laid back about the process, the document and how it’s structured can impact behaviour.
So now we arrive here: how does one craft a performance management document which effectively supports a team focus for individual members? I’ll describe my take on this in the next post.
I’ve just finished reading Rework. I follow the 37Signals blog pretty closely and agree with most of what they say, so I (like many, many other people) have been eagerly anticipating the publication of this book for several months. I read their last book, Getting Real, and loved it.
What did I like about Rework? At it’s heart it supports lean and agile concepts really well. The authors advocate working at a sustainable pace, a lack of detailed, long-term planning, going for quick wins, and making decisions as late as possible, for example.
It’s also an easy read. I read it mostly during my commute to and from work, so in 20 minute blocks, and I got it done in a few days. It’s easy to absorb and is written in a casual style (another lesson from the book: “Sound like you”). Each chapter is broken into discreet essays which I don’t think ever exceed three pages. It’s a great way to break up a book.
The most important thing: they’re right. They take apart so much of what goes on in a typical business environment and tell you it doesn’t have to be that way. They tell their readers that things can be done better and simpler. I think it would take a lot of courage to follow everything they suggest — and in a lot of companies it would be impossible without great leadership at the top, but I’d be surprised if there aren’t a few things anyone can take away and start using, even in the most structured corporate environments.
One thing I found frustrating was that it was a little too light on details for my taste. It felt thin (which I suspect is something they were going for). In rare instances they would back up their statements with examples from other companies, which helped to add some weight to what they were saying. I would have loved to have seen more of that. That ‘thin-ness’ presented many of the things they write about an almost off-handed nature, which undermined some valuable ideas.
I’ve read reviews describing Rework as a recipe book for running a business. I would disagree, it’s not a recipe book. I think it’s closer to a philosophy book. I also think there may more to the book than I got out of my first reading — I have no doubt I’ll pick it up again in a few months and have another look.
Overall, an excellent read — especially given the low time commitment. With so much packed into it, why wouldn’t you read it?
Agile PM Scorecard
… or the six things I review with my steering committee.
Every three weeks I meet with my steering committee to review the status of our system redesign project. Over the course of a year what we talk about and the information I provide has changed. We started talking in a more general sense on scope, budget, and quality, but in many cases we lacked real statistics to support our conversations. As well, given how much I was hammering the importance of quality, my committee wanted more quantifiable data to support what I was saying. I finally hit on the idea of a pseudo balanced scorecard to show project data. After a few iterations we’ve finally settled on the most valid and concise data which works for us:
- Budget. This is an obvious one. Each sprint we spend money on resources (mostly). I report on actual versus planned.
- Scope. This is a little more complex, but the use of Version One helps. The scope of the project is tracked in story points. Specifically, we look at the total remaining scope from the last report, then do some math: (1) story points (SP) added through new stories, SP removed (through re-estimation or backlog grooming), and SP burned down — leaving us at a new total. What we’re looking for here is a trend of falling scope, obviously.
- Remaining Scope per Release. The project is divided into somewhat-logical production releases. Our production releases are set at the quarter level, although the scope may or may not fit into that time frame. We use this data to manage the project / product at the release level, determining if we need to redefine the scope of the release, either adding or taking away functionality. As part of this, we estimate the total sprints remaining to complete the release; this includes a best guess at taking into account unknown and yet-to be added SP (our unknown unknowns).
- Un-estimated Stories per Release. This is so we know what’s out there, and represents one aspect of risk against the releases.
- Quality. Reporting on quality is probably the most difficult part of the scorecard. It is challenging to explain to executives the intricacies of software quality, and coming up with straight numbers to quantify the system’s quality isn’t easy, either. That said, for better or worse I show them: (1) code coverage, (2) defect count, and (3) a ‘quality index’, which is a formula I developed from statistics generated out of our continuous integration style checker.
- Velocity. This is reported in the first section, but we expand on it here, and include a rolling average. This is often the focus of our discussion. I’ll leave it at that
Overall this makes for a meeting where we can get into some good discussion as to why some data looks good or bad. I can use it as a focus to ask for additional resources or help, or we can discuss realistic target dates around functionality. The level of transparency this project has is beyond any other project in our organization, or any project I’ve every run. Sometimes is makes for some difficult conversations, no doubt, but those conversations can be had sooner rather than later.
Thinking about Scrum and Blogging
Clearly I haven’t posted in a while, which is of course breaking one of the cardinal rules for blogging: post often. I have enjoyed blogging over the past year and have found it helpful in thinking through some of the major challenges I experienced in learning about Scrum and trying to lead my department to greater agility. I found as my difficulties decreased so did my blogging. I am re–thinking how I use this blog and website. In the mean time, I’ve listed the top posts since starting the blog. I will take these to be topics people are interested in, and have already started rewriting some of them with some more detail and the added benefit of a little more experience.
Hiring …
For the first time since we adopted Scrum, I am looking for staff. Specifically I’m looking for a new development manager (the current one is moving out of country) as well as a programmer. My perspective on hiring has changed dramatically in the past year.
For starters, I believe it’s important to find people who’ve worked using Scrum and XP practices before. For my manager, I need someone who can coach the team, and for the programmer, I need to know that he or she has ‘drunk the agile cool-aid’ — I don’t want to have to do a sales job on working this way. I’m finding this has replaced technical skills as my primary evaluation criteria. I just hope I’m not being too unrealistic.
I’m also looking forward to a different type of interview. I’ll still conduct the ‘get-to-know-you’ interview, but I’ve become more interested in the growing trend of more realistic interviews — for example pairing programmer candidates so see how they approach a more realistic task and environment. I know there will be limitations to this, but it’s a step beyond hiring someone based on how they do answering your questions in a board room.
So far it has been an interesting experience. The job market in Vancouver is obviously slower than it has been, which means I’m seeing more resumes than I’ve seen in the past (probably 20-25 for the manager and 30-35 for the programmer). What I’m not seeing is quality — most people are wildly under-qualified or ‘wrongly-qualified’, making me wonder if they’ve even read the job description. Every time I hire I experience this problem, but I guess I’m just seeing it on a bigger scale with more candidates on the market.
Right now my biggest concern is that I’m being too unrealistic. Will I ever find a programmer who can write a cover letter addressing the qualifications I’ve asked for? Will anyone who actually has actually worked as programmer in the past apply for the manager role? Only time will tell … .
Scrum: A Year in Review
It was early October of last year that we took the first steps towards adopting Scrum and undertaking the challenge of becoming more agile. I figure this is a good time to review the year, the progress we’ve made, and the challenges we still have to overcome.
Original intentions
My original intentions with Scrum were to adopt a methodology that forced issues to the surface and gave us some flexibility in dealing with them. Specifically, I felt: (1) my staff weren’t working together as well as they could, (2) we had serious quality issues with our applications, and (3) projects lacked visibility with the business. I think that these were largely symptoms of underlying issues, and I hoped Scrum would allow us to bring those issues to the surface and deal with them. So …
How did it work out?
Are my staff working better together?
Definitely. There is a greater appreciation for each others’ roles. We’ve largely eliminated the perspective that work can be thrown ‘over the wall’ (i.e. requirements over to development over to QA) with no interest in how others do their job. There is a greater focus on the team and accomplishing something together. And no one works in isolation with no understanding of what the person next to him or her is doing.
But we’re not quite there yet. From my perspective we’ve come maybe 50% of how far I’d like use to go. I’d like to see more critical thinking and discussion, and still more of a focus on the team. I’d also like to see more thought going into and a stronger commitment to the sprint goals. But there is no doubt in my mind that the Scrum framework can facilitate improvements in team work, and with my staff I think it will continue to do so.
Has quality improved?
We get some positive marks here, as well. Code coverage and documentation is probably five or ten times what it is in our legacy, pre-Scrum systems. Regression bugs (i.e. those that crop up outside the execution of a story) are rare — maybe one or two a month. As we are about to put the first piece of the system into production, I have great confidence that it should be essentially bug free.
But we still need our test coverage to be higher, and we need the quality of our tests to improve. The team is still weak on Test Driven Development (TDD). While we are making improvements, our focus on quality must continue to grow. Again … we’ve come maybe 50% of how far we should go.
Has visibility increased?
In this area we’ve made incredible progress. The quantification of user stories (we estimate using t-shirt sizes but I convert them to numbers) and the use of Version1 to manage our backlog have provided the project team and leadership with an unprecedented view in the project, our progress, and where we have to go. In addition, Scrum has given us the tools to work more closely with the business, with stakeholders (not just the product owner) joining us for planning meetings and stand-ups. I think their perspective on the work we do has changed … and just maybe become a little more sympathetic to the challenges of software development.
My Lessons Learned
Changing peoples’ perspectives is hard
You can never underestimate how hard it is for people to change their perspectives. This has been one of my biggest challenges — getting my staff to see their jobs and our systems (products) from a different perspective. My perspective on their roles has certainly changed: it’s clear to me know how important it is to have staff who aren’t just coders, who can see the big picture, who have good judgment, and who take it upon themselves to learn continuously (and not just about the latest technology).
I have definitely seen a change in all of my staff, but I have to admit I’d like to see more.
Training is critical to make a change like this
This is an obvious one, but I’ve learned it nonetheless. Unfortunately we undertook a change towards Scrum just as the financial crisis hit, and as a result I had very little resources to train my staff. Specifically I would have got them training in Scrum, and at least a class on test-driven development. I also would have liked to have money to send them to conferences (specifically the annual Agile Vancouver event). If I were to undertake this again at a different company, a lack of sufficient training funds would be a non-starter.
Get your systems in place first
By systems I mean proper development and test environments, build servers, et cetera. It’s been a year and we’re still building out some of the infrastructure we need to be really agile (like a proper user acceptance testing environment). I’ve read discussions of the value of a ‘sprint zero’, which is used to set up all the infrastructure. For us, we could have taken use a sprint zero through two or three to get it all set up. It has impeded us at various levels since day one. (Although, admittedly, we didn’t know everything we wanted or needed when we started, and some of his has been dictated by how the project has developed.)
Managing expectations is critical
Typically I’m pretty good at this, but there’s always room for improvement. I was the one who sold Scrum internally, and in so doing established certain expectations with my bosses concerning what we could accomplish. While I’m sure they see the value, I should have spent more time educating them on what we were doing and what they could expect from it. I should have managed expectations better — but I was walking the fine line of selling the change enough that they’d buy in, but not so much that they’d think this would be a cure-all.
What’s next?
I have a few goals for the next six-to-twelve months. As the team matures and our processes continue to stabilize, this will open up opportunities for additional improvements in how we work.
A greater focus on quality
From day one I’ve told the team that quality is non-negotiable. That said, if everyone isn’t on the same page as to what quality means, it becomes negotiable anyway (we did define it, but there’s always room for interpretation). We have recently improved the metrics we use and will take more proactive action with these metrics in hand to increase system quality. This will involve some more learning within the team, and definitely some greater discipline.
Lean development
I recently read “Implementing Lean Software Development” by Mary and Tom Poppendieck and it was awesome. I had looked into lean concepts before, but it didn’t really seem to fit where we were as a development group (in other words I don’t think we weren’t ready for these ideas). With a year of Scrum under our belts, now is the time to start maturing and getting a better understanding of some of the valuable ideas and practices lean offers.
Scrumban
This is related to lean development, but it involves more process changes to how we approach our work. I see the value of continuous flow, but I don’t think the team or the company is ready to go there. In six months, maybe … . I definitely see this as a natural progression from where we are today.
Stepping Away
From day one I’ve acted as the ScrumMaster. As the department director I’ve known this goes against best practices — managers shouldn’t be ScrumMasters. Given our company culture, as well as resources we had at hand, I chose to take on this job anyway. I think I have been reasonable successful, although I have plenty of examples of where my position has caused problems. I will be stepping away somewhat, letting another team member take on many of the responsibilities. I still plan to be involved from more of a coaching perspective — I probably still have the greatest technical knowledge of Scrum — but I won’t be as involved on a day-to-day basis.
And that’s all …
The past year my perspective on software development, team work, and project management has changed dramatically. Scrum is good.
Adding a Second Scrum Team
Over the past few weeks we’ve been trying to figure out the best way to integrate a second team — sort of — into the project. My understanding is that there are some standard practices around expanding to a second Scrum team, which involve either splitting the first team and adding members to each to create two new teams, or otherwise simply wholesale adding a second team. Our challenge is a little different, and I haven’t found any ideas out there that speak directly to this.
To recap, the team currently consists of four programmers, one QA, on BA, and one interface designer. We’ve managed to free up some additional resources, so I now have two more programmers. The questions, is, what do I do with them? Eventually we’ll get three more programmers, as well as another QA, but they’re committed to other work right now. It’s at least several months off.
My original thinking what that I would have these two new programmers form a second team, or at least the nucleus of one, until the others are ready to join. They would simply share the expertise (QA especially) they needed from team 1. It isn’t working exactly as planned, though. They’ve essentially joined the first team, which isn’t something I wanted — it makes for a nine-person team, which is too many.
My thinking in approaching this, originally, was that I didn’t want to break the first team up and disrupt all the painful team building we’ve gone through. But they’ve been disrupted already (the additional two programmers have a lot of catching up to do in the technology). In this situation I see I have a few options:
Re-Form The Teams
I could potentially add another QA staff. In doing so, I could take one of the programmers from team one to team two, so that we have two teams of three programmers and one QA each (BA and design would be shared between the teams). This goes counter what I’d like — as I mentioned I didn’t want to break the team up.
Carry On As Is
Stay with my current approach and wait and see and hope if team two, such as it is, finds its own legs and starts to operate in a more self-contained manner.
Add the Staff to Team One
My final option is to just add these resources to team one and live with the larger team.
So …
I started writing this post a couple of days ago, and after re-reading it and having a couple more days to think, I think my best option is to re-form the teams into two three-programmer, one-QA teams. We were going to have to take the pain sometime, and it might as well be now.
Promoting Quality in Scrum
It’s been a while since I posted. Things in the project have settled down somewhat and I find we’re no longer struggling with many of the basics around how to be an agile team. I don’t think we’re there yet, but we’re definitely past the ‘storming’ phase of team formation.
At our last sprint review I discovered something quite shocking, though — the team had largely given up on test-driven development (TDD). It wasn’t a conscious decision, but came about because they thought it was adding too much time to their development efforts. Everything I’ve seen and read suggest that long-term this won’t be the case. After I learned this I went looking for arguments against TDD. There are some out there, but not a lot, and most of them aren’t very persuasive. TDD isn’t just about the number of tests — it directly informs the design and quality of the code and how developers approach a piece of work. I haven’t been convinced that it isn’t the best move for us.
More thinking on this led me to the conclusion that there appears to be a disconnect in how Scrum is executed on a day-to-day basis compared to the generally accepted idea that Scrum promotes quality. One of the driving forces behind my team’s abandonment of TDD was the burn-down chart and task board. They were acutely aware of what they had committed to, and felt there was no way they could reach their goal if they didn’t speed things up but not doing TDD.
The biggest, most obvious piece of information in Scrum is the task board, which only shows what or how much is being done. One of the most critical aspects of a story is defining the “done” in order to support quality goals. But I can’t see any straight-forward way to represent quality to the team. There are signals: does the build fail, are stories being completed to the “done” criteria., et cetera There are also metrics: code coverage and cyclomatic complexity, for example, but neither are an overall indication of quality.
Of course there should be tools in place to protect the quality — automated tests as a start, attentive QA and/or product owners, but clearly these aren’t enough. We had a successful sprint, all of our stories were done, new tests were written and they passed, but still there’s a good chance that overall quality wasn’t where it needs to be because TDD wasn’t practiced.
Reaching our committed sprint goals is important, but not if it forces staff to cut corners, even a little bit. Over the coming weeks I’ll be working with our QA as well as the rest of the team to figure out the best way to disseminate quality information about the project. In the mean time, TDD has resumed.
