The fallacy of time tracking

I constantly see developers advocating the power of being and doing agile, how agile methodologies deliver all this benefits for the team and how our clients are going to get better value for their money, faster times to market, etc. However I am left with the feeling that it is often forgotten that we are still running a business and we have to make some profit in the process or we are all going home.

So when the opportunity to talk about these matters with people who are faced entirely with the business side of running Agile rises, I quite enjoy it. It opens a window into a way of seeing things that allows me to understand where they are coming from, what their main concerns are and what the reasoning behind their positions sometimes is. It is a fresh perspective, a new pair of eyes and I like to embrace that whenever I can.

Profitability, ROI, budgeting and agile contracts are recurrent subjects that worry people from the business end of the agile equation and that are not all that well explained by most advocates.Maybe it is because of where they come from, their lack of experience in the subjects, or simply they think it does not affect them. Buy just like a rowing eights team, for an agile organization to function at its best, everyone has to be on the same boat and rowing in synchrony to achieve their goal.

 

That is all good. But why is this post titled “The fallacy of time tracking”?

A few days ago a friend of mine who is a P.O. was looking for some way how to calculate the profitability/ROI of his team and the projects they are working on and this is where the idea for this post came to be. 

Let there be time tracking

If you worked at one of the old-school big companies here in Spain, or come from an MBA you are probably thinking... What's so hard about it?

Just ask the team to start writing down on an excel spreadsheet or the back of the cards the amount of time they took to complete the task and the role of the person. Then your friend can later look at it, crunch the numbers and have the metrics he wants. I mean, it is not rocket science. We have been doing it like this for who knows how long now! And that’s exactly what my friend’s first reaction was!

Who hasn't had to fill in the excel spreadsheet, maybe even something fancier: a Visual Basic application, or that web page in the corporate site... Hell, I even had to create one of those :)

It sounds like a great idea! However, as popular as per task consumed time is... Tracking time spent on a task is a flawed and most of the times harmful way of measuring the profitability of a project or team.

Here are some of the reasons why, in my opinion, this is so.

Time tracking is inherently imprecise.
Forget about the precision. If you think you are doing this because you will get a precise result, abandon all hope and go cry in a corner, because you will not get it. It is extremely complex, if not impossible to track exactly how much time one spends doing something.

We humans are really bad at tracking time. What may seem minutes could really be hours if you are in a state of flow and minutes can seem like hours when you are not engaged in the task at hand or under stress. Add to this: interruptions, context changes, etc... All this little things that we should be keeping track of that per se may not seem like much, but that definitely adds up to time.

Yes. There are tools that track time, like stopwatches, mobile phones, time tracking apps. And yes, they help. Maybe they can track how long you have been using a certain application, of if there is mouse movement, etc. Sadly they are not quite there yet when it comes to mind reading or detecting if the user left.

Then there is matter of the “per role price”. The fact that usually different roles are most of the time “priced” at different levels and the reality that often the same person is undertaking more than one role, makes the task of tracking real cost per hour even more difficult since sometimes you are working as a developer, other you are “the architect”, or a tester, etc. This makes it not only about the time you work, but also what kind of tasks you are performing, since for your company it is not the same from a costs perspective.

Time tracking is most of the time just done wrong.
People forget. We forget what we did. We forget to write it down. We forget to start the timer. We get interrupted and spend time in things that are not part of any particular project. Things like answering a question from a coworker, the meeting we got called without being notified, chilling out on Facebook, or drinking coffee, etc.

Of course from a business perspective we are expected to work a certain amount of hours each day, and if it's not in the time sheet report, it doesn't exist. So, what usually happens is that this time, goes in the report under whichever project we are supposed to be working on, or under the project that is doing the best so far.

This is curious effect. It seems that just by existing, this report acts like a pressure point. Is it forcing the guy filling in the hours to “lie” about what he is doing? Well, you may argue that’s nonsense because no one is actually forcing him. But social arrangements that produce responsibility are arrangements that create coercion, of some sort.

What about crunch time? Does it go in the report too? Does it cost the same as regular hours?

Time tracking can be manipulated and corrupted.
I have often seen people manipulate the reports by compensating from other tasks or other projects in order for everything to look neat. They report perfect estimations and perfect execution. Of course, not just because it looks nice, but also because of the bonuses they will be getting for not having any schedule deviations.

These kinds of associations are what makes time tracking the preferred tool used to sharpen the stick or sweetening the carrot at the risk of killing intrinsic motivation and undermining the corporate culture. They create politics at the workplace and in my opinion these politics are waste of energy and time that gets in the way of actually performing.

Time tracking is not a core business activity and it is extremely time consuming.
Let me ask you a question. Have you seen anyone who thinks that tracking where his time gets spent is what he gets paid for? Unless you work at RescueTime.com or some other company that creates time tracking software, this doesn’t hold true for you.

Even better, have you seen one of these spreadsheets with a special project that says: "put here the time you waste filling in the time you spent doing real work"? Yet it should be there right?

Sadly this is time you end up charging your customers for and that you could have spent delivering value. It is hard to believe any customer would be willing to pay for it, if they knew the truth.

Time tracking is a lagging indicator.
There are leading indicators and lagging indicators. The firsts provide you with information before the particular change you are after actually happens. I in a way they tell you about the future.

On the other hand, lagging indicators provide you the information only after the fact. Time consumed per task belongs to this category. On its own it doesn't reflect the "potential" project deviation until it has already happened.

When I think of this I am always reminded of the Windows progress bar.

Everything would be going smoothly while copying a file until it got to around 99% and then the bar would simply stop moving. Parked at that last 1% for orders of magnitude more than what it had taken to get to the other 99%.

That's what happens to projects and tasks tracking over and over when focusing on the time spent. Instead of: “How much time have you taken already in this task?” a more helpful question to ask would be: “How long do you think it will still take you to complete it?”

Why do we drink the poison then?
Everyone is fixated with using hours gathered at the individual task level to know the profitability of the projects. I wonder if this is a reminiscence of the use and abuse of Gantt charts and project planning tools.

Why are we so prompt to attempt going down this “time tracking” path?

Sometimes I have heard people say: Well, we need to know because we charge by the hour, right? To which my answer usually is: Not really. No.

The notion that you charge by the hours someone is ___insert favorite software-development-related activity here___ is at best naïve. Unless you don't really care about your customers, or running a successful business, you are not supposed to be charging by the hour.

You should be charging (and you probably are) by the customer's perceived value. In other words, you charge by how much value your client sees that you are offering him with your product or service. From this point of view, your per hour cost will only be the lower bound to your final price. Often people construct their prices from this lower bound, sometimes, so that they can completely ignore it... *sigh*

There is another common response that comes up and it is this: Our people need to be supervised and time sheets are the way we do that.

If you treat people like children, they will behave like children. In my experience if someone is not performing as expected in a mature team doing agile, it will be spotted right away by the rest of the team and dealt with accordingly. I would go as far as to say that, if your team needs someone breathing on the back of their neck in order to perform, you have bigger problems to attend to.

It is not all bad… Well yes, but not always.
There are places where the benefits of time tracking overcome the disadvantages. For sure in some cases you can have some degree of time tracking without it being prejudicial. Don’t get me wrong. I’m not against tracking time or any other metric as long as it fulfills a couple of requirements I find indispensable.

• Recording the metric should not impact the team in a noticeable way.
• The metric in its self doesn’t get perverted. Carrots and sticks anyone?
• It’s not a vanity metric.

Is there another way?
The starting point to overcome this fixation and avoid most problems derived from it I think is to instead of starting with a metric and work our way backwards, ask ourselves what is the real motivation behind the particular metric we want. Only then we can choose the right things to measure and when to measure them. It may as well be that we already have the tools needed to calculate what we need.

Do we want to know the project's profitability? Why? Or is it the overall team profitability that we are interested in? Is it because we want to know if the project will stop being profitable and if so, when is it going to happen? Or on the other hand we are trying to figure out what the base cost for this project is in order to provide a budget?

Each of these questions is in its own is a subject worth exploring in future posts and I hope to not only approach the metrics and calculations of some of these KPI in an agile environment and how to use them best for portfolio management, but also, and more importantly, the motivations behind them. 

If possible maybe raise some questions that I feel are fundamental and get you thinking. Until then, see you soon!

Integration tests for your database code

I hear a lot of people talking about tests and I have been to a couple of events where speakers have given presentations on the subject. Everyone talks about unit tests, and TDD and BDD, Continuous Integration. However, I don't know if you noticed but database-related integration testing is often overlooked, omitted, or briefly mentioned when talking about tests. It makes you wonder why. Doesn't it?

Why is that fear on the subject. Are database-related tests not needed?

Yes, they are. You need them because there are things you can't simply mock (stress and load tests to mention some). More than that, you may not have other options because you are dealing with legacy code and no time ($) to do a proper re-factor and unit testing. We need them because we should make sure our "whole" system works as expected.

But to be truthful, the main reason would be that database related testing sucks. Is difficult to get right. It is slow compared to other kinds of tests and if not implemented properly it could become a waste of time and a source of headaches.

Yet a lot of the logic we write in our software relies on certain preconditions and behaviors of the underlying persistence mechanism to be correct. Most of the times we think there is no way of testing those assumptions, than to actually use them. Using them ether by running the solution on the developers local system and database or an integration server or something similar.

Before you start screaming and writing me off your list I must say.

I don't couple my business logic with my persistence. What I meant is that parts of our logic rely on behaviors that we sometimes take for granted they will happen as we think. These are things like transactions management, when using Spring's Transactional attribute, or cascaded persistence, etc. all of which could fail at run-time.

Sadly, unit tests can't help us in this regard. It would seem that the only way of actually testing that this behaviors are what we expected, is to execute or deploy the application on a controlled environment. Or is it?

You already know what the answer is.... don't you?

the answer is...

42...  :]

To explain it better, I am going to split the whole process into two scenarios or contexts and attack both in different ways.

Case one: Building Application from Scratch
Like Uncle Bob likes to say, there is nothing like the green field. That bast meadow where you first start to build your "architecturally sound" software. There is nothing there, no mess left behind by others, no constraints. It opens up the door to a lot of different opportunities (including creating a big mess). This is our first scenery. But before we start digging in on the hows and whys, some disclaimers are in order.

I am going to assume you know what and ORM is and that you are using one, and if you are not you have a pretty good reason not to. Either way I will explain what I usually do or would do when I find myself in both situations. That doesn't mean that this is the "best" or recommended way. It just means it is my preference. If you have your own ideas on how to improve the process or maybe a more efficient one: don't be shy and share!

Also, I am going to use Hibernate + JPA for the examples because a friend asked me to, but this would be easily extended to NHibernate, or other ORMs like the Entity Framework code first approach. If you ask for it on the comments I can extend or add new posts to include those too.

What do we want to achieve?
We want to test our persistence and query logic, usually located in the "DAO" layer of the application. Since we are good developers ;) we want these tests to be deterministic, self-verifiable, and order independent.

What do we need?
We need to setup a complete database environment, equal or similar to the one we are going to be using, and then populate it with test data, to be able to assert the behaviors in our test code. We will use Hibernate + JPA + HSQLDB + Spring 3.0.

First thing would be to configure Hibernate to recreate the database schema based on the mappings we have, every time it initializes for the integration tests. I will do this by initializing a new spring context and JPA persistence configuration, just for the tests.



Notice that I am also using an in memory database provider HSQLDB to make things a little less complex. However, this could be any other provider. You would just have to provide any connection details needed and to make sure you have the right permissions for the schema.

Since we are using Hibernate in Java we are going to take advantage of the functionality Hibernate gives us of executing an initialization script called import.sql after the schema update process during its own initialization. You can read about it here and here.

Another way of doing it would be to use DbUnit, of which I will talk more in the next example case.


So now we have our import.sql file ready to be executed at the context's initialization. Here you would place your test's data as a set of insert statements. This will allow us to initialize the schema created by Hibernate from your model with the tests data just before the context is accessible to the tests

As you need more tests and the schema evolves, you will extend and update this scripts.

And that's all! Now you can start writing your database tests.


The good
One of the good parts and the one I particularly like the most, is that the database schema doesn't need to be stored as a script in the source control. Instead it will be stored with the code, since the actual model that you are using is the blueprint used by Hibernate to create the database schema. So any changes in the mapping reflect on your integration tests as soon as the context is initialized and developers don't have to write down migration scripts for the changes which is error prone and boring.

This also means if you change database providers for instance, from MSSQL to Oracle, Hibernate will be the one creating the script for the creation of the schema, and it will be using the dialect of your choice. This is particularly useful when you still haven't made any decisions on what the underlying persistence support will be.

The bad
The main problem I see with this approach is that it only works if you don't have previous data that you need to maintain. But if you do, there is no way (that I know of) to use this behavior on the "update" mode and also keep the "migration" done by Hibernate.

Also, if the test data gets too big there is no way to split the script so that it would be more manageable using some kind of "import" directives. However, I think in version 3.6.10 > of Hibernate you can set the scripts to load for each persistence unit using a configuration property, or through code. You can find more info about it in this Stack Overflow thread and in this Spring Forum thread.

By the way, there seems to be a "bug" with the JpaVendorAdapter that I use in this type of set up, which sets the provider to "update". To resolve this issue just set the generateDdl flag to false. You can find more info on the reasons here.

The Ugly
Integration tests are slow. This solution needs you to instantiate the context and database for each set of tests. It only initializes the context once on each test suite. However, although this is faster than doing it for each test, you may run into order dependency problems among tests because of the data, if you don't write your tests properly. Use wisely.

You can still set it up so that each tests executes with a clean db setup. A solution would be to regenerate the schema an repopulate the tables with data for each test using the SchemaExporter class. I leave that as an exercise to the dear reader ;)

Case two: Building from a Legacy Database System
Now, take into consideration that once you release a version of this application, you are actually moving into  building on top of a legacy system.

In this case you already have a database, it may be from a previous version or from a totally unrelated product. Whatever the case is, I am assuming you can't loose data or in some cases modify the database schema. The create approach in Hibernate won't be of much help here. What do we do then?

Well, in this case we are going to need something else. Here is where DbUnit comes to the rescue.

There are other solutions like the SQL Ant task that executes SQL scripts against the db before you run your tests, but I like dbunit better because I can put the database initialization and finalization into my tests, per test, instead of at build time. Also in most cases I can escape having to write the SQL insert statements letting DbUnit in charge of the dirty bits of generating them.

Whats different from this approach?
In this case, we are going to get a little more control over what is happening on the db. We will still insert the sample data, just that this time, it will be DbUnit the one doing it. To do that, we first need to put this information in and xml file format that DbUnit will understand, so that it may be able to dump it into the db when we tell it to.


The format is pretty simple. All records are hanging from a dataset root object and the name of the node will be the name of the table where you want to insert this information. For specific column values you use the attributes of the node. The actual content to put in the column is the value of the attribute.


The Good
We have per test database setup. This is faster than the method we already talked about but, you still need to do a little cleanup afterwards. A way to avoid this would be to execute each test under a transaction and at the end of the test just roll the transaction back. However, this may not always be possible.

We also get a "semi-raw" code based access to the database for verification. In this sense you get more control over the verification of what is really happening. In case you didn't notice when I was testing the insertion in the first method, I used the findUserById method thus relaying on my own code. While for unit tests this may not be a problem as long as the code you are using has its own unit tests, when it comes to db integration tests I wouldn't recommended it. The reason being that you could fall into the trap of, for instance, thinking that your insertions are working when in fact they could just be cached by the underlying persistence mechanism. Be aware of it.

Finally, you can have both methods working side by side and use them as needed.


The Bad
While DbUnit will insert the data for you, it won't create the schema from the information we give it (not that it could). That's why we are still using the "update" mode with our persistence.

There are also some funny things about the way data gets cleared because it relies on the order the records where inserted. I remember this particular dog biting me some time ago.

The Ugly
DbUnit seems like it has not noticed the rest of the world has changed. By this I mean that you end up writing lots of boilerplate code for your tests that could have been avoided with some properly coded annotations. This is typical of the JUnit 3.X style of tests where you would need to inherit from a TestCase class or something similar. Not cool. However, if you refactor your tests, (like you do, right?) you will end up with little or no repetition.

This is all for now. You can see the full code at this repository on github.

Keep tuned for some more smelly code.. with potatoes on the side soon and leave some feedback if you want to help me improve the quality of these posts ;)

Update 14/04/2012: I renamed the github repository.

Make it stick.

A few days ago I was watching a recorder workshop by Angel Medinilla given at the Barcelona Lean Camp this February. He was explaining lean principles to the participants by having them practice a series of Aikido techniques or movements. But there was something in particular that he said that resonated with me.

"Metaphors are a powerful way of explaining things to people, because they stick, because they create a paradigm." I totally agree. In fact this post is a compilation of some of the metaphors and similes that I tend to use when explaining agile or explaining agile practices to someone. As usual, this is just my personal repertoire, but I would love to hear what you guys think about it or if you have any other examples that your prefer.

I plan to keep updating this post as I find new examples that I like. I hope you find them as useful as I do.

Angry Birds

I think most people know the angry birds game. In this game you have a goal which is to kill the pigs sitting at different positions on the map by throwing birds at them from a slingshot. You also have lot of obstacles that may prevent you from getting to the pigs directly. for example you would have boxes, metal crates, glass, etc. and some other variables like gravity, speed, the effects of impacting an obstacle, etc.

This is a good example in which to analyze and discuss the advantages of adaptive over predictive planning: transparency, risk management, etc. Another good question to ask would be: What if the pigs could move?

Crossing the river
You need to cross a river in a small boat but there are lots of stones on the way that you can't see because they are under the water, this stones will probably delay your arrival to the other side or even sink the boat.

You can discuss how frequent feedback cycles are like allowing you to lower the river's water level so you can see the tip of the rocks and avoid them in time or minimize damage.

Growing Roses
You are growing roses. You care after the plant and you know who long it will take to grow and give flowers. However, you don't know exactly how many flowers it will give. You can compare the plant to a product being developed and the flowers to the scope or features for the product.


Running a Sprint vs a Marathon
This is kind of evident. You would compare having a sustainable pace, flow and  rhythm,  to running a marathon, and burning the midnight oil to running a sprint. At the heart of everything is the question: How good are you when you are tired?

It is easy to define sustainable pace but hard to apply. It doesn't mean you can't sprint once in a while. Marathon runners do it too, at the end line, or to surpass another runner. But if you need to do it every time, you have a problem.

Building a road
Suppose your client wants you to build him a road so that he can go to his beach house. Does it need to be an 8 tracks highway or just a 2 ways paved road will suffice? Would a track in the middle of the woods be enough?

There are different levels of achieving the same end results or goals and building incrementally gives you the opportunity to deliver value (an unpaved road in the middle of the woods) and then improve upon it with your customer's feedback at hand. Maybe the unpaved road is enough. Or maybe at the beginning he thought he needed a highway, but he realizes now he doesn't. This is also a good sample to discuss prototyping vs building incrementally.

A car going somewhere
This sample is particularly centered on the roles of a SCRUM team. Where the whole car signifies the project or product being built. The driver is the product owner who has the vision and knows where car needs to go. The Team is the chassis or engine, and the Scrum Master would be the oil or liquid for the breaks.

This is it, for now, if you want me to include something else let me know. :)

Not all TDDs are created equal.

There is little or nothing that has not been said about TDD. As you probably know if you are a practitioner, there are lots of articles describing step by step the whole process behind it and arguably also the state of mind that TDD aims to promote. However, there is something that I find again and again when I go to Coding Dojos, events like the Global Day of Code Retreat, or even when I am pair programming or mentoring someone.

There seems to be, at least from what I have seen, two main ways to approach TDD. I'm going to call them Holistic and Reductionist approaches. Both of these have their advantages and disadvantages and since it is something that comes up a lot with beginners and seasoned professionals alike it seemed like a good idea to pay some attention to it and write this post. 

So in order to understand it better and just like when you go to practice yoga or music or anything else you want to get really good at, I started paying attention to the way I was doing things. Particularly, behavior and thought processes I followed when practicing TDD from both of these perspectives.

Note: The rest of the post addresses a lot of abstract concepts, and I'm terrible at explaining myself. If you feel like this post would benefit from more examples, or needs some editing, let me know and I will add them or change it.

The Holistic approach.

What do I mean by "Holistic approach"? Holism is described as the idea that natural systems should be seen and studied as a whole and not as a collection of their parts. The underlying principle is that the whole is not just equal to the sum of its parts. 

This is mainly a top to bottom approach when dealing with solving the issue.

When we normally take this approach, what usually happens is that you tend to focus on solving the problem at hand directly instead of the sub problems. 

With this approach you think of things like: outputting all prime numbers in the sequence of the n first natural numbers starting at 2, instead of thinking first of solving how to finding out if a given number is prime or not, or outputting all numbers which satisfy a condition to the screen.

What I have seen is that since you tend to focus on the functionality and results of the principal issue instead of the sub problems, you get results that you can use faster. Most of the time they are partial or incomplete results, but the bottom line is you get more "deliverable" value from the start. Things like: I can create a user, but the process doesn't check all fields that need to be validated, etc.

However, as a side effect, what usually happens is that one tends to add "extras" to the code to help him test. I am talking about things like object properties to inspect inner state in your unit tests, having virtual methods that could be marked as final because you need to override them to test, etc.

Another common side effect that I have detected is, that the frequent mantra of "the emerging architecture", although still true, usually happens at the end, when you think you have a solid grip over your main problem and then you start refactoring your code. 

The inconvenience I see with this is, that if you don't refactor as aggressively and often as you should, this "architecture" may not emerge at all, or it may be deficient. I am not sure if this is good or not, but I personally like to have at least "some" architecture than no architecture, or an "emerging" architecture. Mainly because in both cases it could turn out to be a recipe for disaster if everyone on your team is not on the same page. Like when you have a junior developer on your team or not everyone  has the same skill level regarding TDD or refactorization.

One last thing is that as you are developing, you tend to prevent your target problem's corner cases, but not the sub problems' corner cases. This may lead to other bugs when the functionality gets refactored out or when it gets used by others, since they will tend to assume that it works as expected, not just in the context of the main issue, but in the context of the sub problem. 

To give an idea, following the prime numbers example, if another coder wanted to use your is_prime() function, it would be reasonable for him to expect false when it gets passed a 0 or 1, but if that wasn't part of the main problem's constraints, the developer may not have considered those exceptions, leading in turn to new bugs.

The Reductionist approach.

Reductionism on the other side is sometimes considered the opposite of Holism. In this case you would try to understand the whole by looking at its parts and their interactions. Inherently this is a bottom up approach when doing TDD.

You focus on sub-problems of the problem and when you have solved them, you focus on the interactions. Finally, you refine the interactions so that the results combined provide the solution you are looking for.

Contrary to holism, in this case, architecture emerges from the start, driven by the necessity to decouple particular sub problems. For me this approach makes it easy to loose focus on what your general aim is. It requires a great deal of self control and experience to know when is it enough and you should move on and continue with another area.

However, code gets tested more thoroughly. Corner cases for each sub problem are more evident and they tend to be addressed from the start. So the problems found when the clients of your API made assumptions about how the code worked diminish a great deal.

Nevertheless, focusing on the sub problems usually narrows your thinking. As a consequence you need to do more integration tests in order to validate sub parts and the way they are supposed to be used. You tend to write a more developer-friendly API because you are always thinking as the client of your API when you are writing the unit tests. 

This way of doing things seems to work better when you happen to have a roadmap or well-defined plan of how the pieces fit together. For instance, when you are implementing algorithms, which are usually decomposed in very specific sub problems.

Conclusions?

Well, this is most of what I have observed myself and with a lot of input from the conversations I have had with friends. In the end, to me at least, no one method is applicable to every problem. In my day to day job I usually jump from one mode to the other depending on how it "feels". I'm very interested in hearing what others have to say about this subject so please leave a comment below. 

Until next time and happy coding!!

Promoting change...

I have heard a couple of friends, developers and  IT mostly, complaining about the situation they are in at their workplaces. Some say they have a boring job, others say they don't get paid enough, others are simply not motivated. My question to most of them is mostly the same: are you doing something about it?

Let's face it. We spend a lot of time sitting in front of a computer. In uncountable cases, that time is more than 40 hours a week. If you are going to spend that much time doing something your entire life, you better make sure you like doing it. If you don't, maybe what you are doing is not your thing. Maybe there is something else that would make you happier and that would be more gratifying to you. Something you can also get paid for... *insert joke about prostitution here*

But what if you do like what you are doing? What if your are good at what you are doing but, somehow, because of external factors you are still not enjoying it the way you think you should. Maybe you are even, starting to hate it... Well... that's why I am writing this post.

The truth of the matter is, most of the people I ask this, haven't even tried to solve the problem. They haven't even recognized their problem! They are just complaining, and they are not even complaining about it to the right person!

Whether you are the owner of a small software consulting firm, a project manager, or just another developer, you can always do something about it. There is no such thing as no options when it comes to promoting change in your work environment or organization.

This post is based on personal philosophies, experiences that I have had and some that I haven't, but I have received first hand. It is not intended to be a guide but more of an example on how it is possible to use what you have to improve your current situation and somewhat, a call to action. If you are willing to take the step, you will be surprised of how simple it is sometimes.

The post was written after the fact, so a lot of the things I am talking about here may sound very planned, when they weren't and happened organically. 

Also, it is focused on how we dealt with and approached introducing agile software development to the company I am working for. Some of the challenges, the solutions we found (and are still finding). I imagine the concepts are the same for any kind of change. So if you think: - "Ok it's just for agile" - I urge you to reconsider and take the time to see what you can get out of it that is not Agile Practices or Software Development related. 

Finally, it's an open invitation to anyone in the same situation to share their experiences in the subject. I would love to hear about them.

A little background

Some time ago others and I in the team were really burnt out about how bad our code base was. It was impossible to work with ! Fixing bugs became this (to misquote Dr. Sheldon Cooper) un-unravelable web  of changes all over the place, and as if that were not enough, there was no other way of testing any of the changes worked but to deploy the application and manually test it. We work with Eclipse and a really old version of JBoss, etc.


You see what I'm getting at... Let's just say I was really unhappy about how things were going on that front. Being the new guy on the block there, it was expected that I would find some resistance to making changes from some of the guys in the team, as in fact I did.

However, taking a chance, I started using TDD on my new projects. This saved a couple of hours on testing and bugs fixing. Then with those and the help of another team member, we set up a continuous integration server, a Maven repository, and some other tools for metrics and knowledge sharing. Afterwards, every project he and I started working on, diminished the amount of friction required for set up, working on them, deploying and testing.

Soon, others started asking how they could get to use it on their own projects. It became "mainstream" and eventually the rogue continuous integration server passed from being hosted on the slowest computer ever (the only one we could use...) to become a virtual machine on the company's servers.

The moral of the story being: if you want change, you have to start yourself. How are you going to ask others to change if you are not willing to do what it takes? Sometimes a spark is all you need to set a whole forest on fire. Check out this Derek Sivers TED video. 

Let's talk about change

Change is a strange animal. Almost like a cat. We pat it and care for it. From time to time, it comes over and wants to play with you. However, like a cat, it barely listens when you are calling him. So how come it is so hard to "change" things when you want to, instead of it just happening to you.  

What I have come to realize is that affecting the organization is a matter of knowing how to approach change. Whether it is getting the team to consider a new technology, tool or methodology, it boils down to dealing with change. Why? Because change creates fear. Change creates discomfort. Finally, most changes involve transforming ourselves and the way we see things, and that is just plain scary.

I would say there are mostly four main things you have to consider when you are trying to change something. I am not saying they are the only things you have to consider, but for what I have seen they are the ones that I personally value the most. I will explain them in detail later, but for now, those four things are:

First, the desire to change. Whether you, or the team, or the organization wants to change. Then the ability to change. Are you ready? Do you have what is needed? Followed by the permission to change. In case you need it of course. Then, last but not least, the resistance to change.

Each of these with their own unique traits. I think we could agree that you will find them wherever there is someone defying the status quo.

The Desire

Why would you like to change? If you think about it, your answer will be different than mine, or the person sitting right beside you. But somehow the bottom line behind it is: dissatisfaction, yours or others'. Why? That desire to change, comes from the pain, the discomfort this dissatisfaction causes. With that in mind, to generate change you first need to generate this discomfort with the status quo, or create awareness of that pain. 


Don't get me wrong, I am not talking about making other people's lives miserable or anything like that. In fact, if you want to change, chances are the situation is already there, whether it is caused by external factors, or internal. It may only affect you, or it may affect others too, but you want those problems to surface and be dealt with. Make them. 


Usually when extending agile through a company, the idea comes from the people feeling the pain. Those being us, developers. When that happens it is referred to as bottom-up change. This is where the idea and action starts with the people at the bottom of the structural pyramid.


On the other side, if the change comes from the top of the pyramid, it is called, yes you guessed right: top-to-bottom. I have never seen this type of change first hand myself, but I know a few guys who have. In this case, when the guys generating the change are not directly the ones who have to change, things can get a little complicated.


Top to bottom, bottom up, in any case, everyone involved needs to be on the same page and moving towards the same goal or it won't work. Creating commitment is hard. You may not know how to do it. However, one thing is for sure, whatever you do, be sure not to undermine it, or people's engagement in it.


More about some ways in which you can create awareness below.

The Ability

Now comes the next stage. Suppose you already have that desire to change. You want to move forward with using whatever flavour of  agile you want, or TDD or whatever. But the question is: are you ready?


Just wanting to do something is not enough. Yes, it's a big part of it. I personally think it's the most important part. But sadly, the successful undertaking of anything, rests on top of the ability to execute.

I always remember this great quote when I hear someone ask for something impossible. It's from a children's book.

Being ready to change, is not just about the mental state. You need to have the environment. Here are a couple of things that we have particularly been looking out for: skills, change of beliefs, and barriers.

In my experience, embracing agile comes from two things, a mind set, and an ability to focus on what's important. Ok, I can hear you saying: - but everything is important! - I like to rephrase it like this sometimes: ability to see the things that are not important, to make them not interfere with what is important. Sometimes that involves focusing on them first.


In our case, we didn't have that ecosystem that would allow us to start doing so. No continuous integration server, none or little experience or knowledge about agile, SCRUM, TDD. Lots of uncertainties about business, project and technical constraints, etc. We needed to address those, and more.  However, not all of them before we moved to applying, and not applying all of them at the same time.


You want developers to be writing code, not looking for libraries. You want your newcomers to get in line with the program and start contributing as soon as possible. You definitely don't want them wasting time on things that a machine could do and you want them to join the mindset and leave behind the baggage from years of working differently.


Improve your skills. Which skills? Well, talk to people! Ask them what problems do they have. How can you help? Why aren't things moving forward? I will talk in more detail about what you can do about skills below.


Beliefs are hard to change, but it's not impossible. Having external influence is a great help. You can talk to someone who has already gone through the same process to come and share their experiences with the team. You can also find and recognize success case studies inside the company so that people know what is expected of them. Initiative, inventiveness, resourcefulness, self-motivation are all qualities you want the team to have.


One last thought. Simplify the change process. Avoid bureaucratization or it will collapse on its own weight. 

The Permission & Support

Well, my question here was: to what extend do we need that? I definitely don't know about your particular case, but in our case, we are very result driven. That worked great for us. By that I mean, if you get work done, they won't bother in controlling every little detail, because at the end of the day they trust us to do the job. 


That's great! It is a trait of all of the agile flavors, trust people, focus on them. They are the ones that make the magic happen. But, we also have to make a living. That is a part that, as regularly paid developers (not freelancers), we usually forget. 


A lot of the time when I talk to developer friends about that, I find some disconnection between what they would like and reality. Reality being the company may want to change, they may even think it is a good idea and see the benefits, but at the end of the day it is a business, and they want to make money. You can't forget that. They need to keep the engine running and the way to make both things happen is to put your feet on the ground, and level with it.


Things we looked out for were:
Do they see the benefit? Personally, I think the best way is to show them and use their own language. It may sound harsh, but if they talk numbers, you should talk numbers. Estimate improvement. Get real data, code coverage, defects, estimated time for completion, money saved, money gained. 


Run trial projects and change within the confines of your space, they will help validate your assumptions. If you need it, ask for permission, although I personally prefer to give an apology than to ask for permission. But, be smart about it.


Again, find people who are in their same positions and went through the process, who are willing to exchange their experiences and insights with you so you can gain better understanding.


This and this are nice examples of where you can find that kind of people in Barcelona. 


How does the agile process fit into the commercialization process? Does it have an impact or are you going to keep selling the same way you have and work your magic on the background?


How much of an investment are they, and you, willing to make in the change? Will they pay for training? Will they give you time to work on what needs to be worked on but does not directly produce any income? On your side, will you for instance, be willing to go on your own budget to receive this training or spend a couple of hours after work?


What is it that they do or don't like? Ask, and communicate. 


Lots of other ideas come to mind. The list is huge. Just put yourself in their shoes and ask and answer the questions you would ask.


If you do need permission, there are ways in which you can get it. Negotiate it. You can't expect a big company to change from dawn to sunrise. Set your company's expectations and yours clearly, so that there will be no misunderstandings. Again, create awareness. Do your homework. 


Prepare a presentation for management and deliver it. Selling agile to management, can be hard, so prepare yourself. Give yourself the best chance. Get data. Find success stories. Find people in the community who are going through the same process or who already undertook the change and get them to come and talk to your guys and explain to them the benefits. Explain them yourself to your co-workers. 


Set goals and timings for the gradual implementation. It is easier to sell them a particular plan that you can refine and adjust to their concerns, than to sell the idea of change alone which leaves too many open ends.

The Resistance

There is always resistance to change. It is hard-coded in the laws of the universe. But the way we have found to deal with it, is understanding where it comes from. To do that, you have to listen. Chances are people already want to tell you. Be aware of the symptoms. Fear, discomfort, lack of communication. Listen to what people tell you.

This is where the communicative skills come in handy. Adapt to your interlocutor, put yourself in his/her shoes. Ask the question: what is it that you don't like about it or you are having problems with? And wait for an answer. Finally, think about what the person said and act accordingly, don't just rush to deliver a speech on how good agile or TDD is. If you don't put any thought and heart into it, it will just come out as fake.

Where do you start?

Start at the beginning! Where are you right now? What is it that is bothering you? What is it that you want to change about your current situation? This is simple most of the time. You already know what it is that is bothering you... Solidify your thoughts about it... Most of the time this is usually a symptom of an underlying problem. Use the 5 whys technique to find out what that underlying cause is. Have a clear vision for yourself and others to understand and follow. For instance: what does it mean to have self organized teams? What does it mean that the team or employees are participating?

Next in line is figuring out where is it that you want to go. What is it that you want to achieve? A more interesting job? Better work dynamics? Better salary? Career advancements? Find out what it is, or at least, what it is that you don't want. 

If there is something that I have learned over these past few years, it's that if you know where you want to go, you already have 5o% of the problem solved.

Finally, plan and execute the way to get there! This is tricky because you may at first not be able to see how to change the situation. You may think that some of the solutions are too hardcore for you to consider. They are not. Like I heard someone say not so long ago in a talk: think of it this way, you know how your work life has been the past three days, the past three months, maybe even the past three years... If you don't do anything about it, you already know how it's going to be in the next three days and the next three months... etc.

I am confident by now, you get my point.

Don't get paralyzed.

Now, a word of advice, don't get paralyzed! Yes, plan, but don't fall into the trap of paralysis by analysis. Break the inertia by doing something.

For instance, start playing around with new technologies at home or in your spare time. Maybe even incorporate them into a new project. The important thing here is for you to start doing something and don't stop once you have. Keep thinking about what your objectives are and increase your efforts on getting there.

Create awareness.

Create awareness sounds pretty simple, however, I sometimes hear people complaining they don't know what to do. Ok, here are some tips.


You can, for instance, send management articles about the competition to incentivize them to compete or compare with them.

Invite your colleagues or boss to webminars or seminars. In our case one of the things we did was inviting people to come over to a free seminar for a master in agile project management at laSalle. The presentation was so well delivered that the guys that went there were bursting with energy and enthusiasm about making the change!


Ask for feedback from customers and others in the team or the company. Talk about what works and what doesn't. Take a look at the current process and identify what is not working or could be improved.


Ask, what do you think will happen if you or your organization doesn't change?

Get others involved.

It's sometimes easier if you are undertaking a particular challenge with someone else. If you want to get someone else involved and you think that would help you, go ahead. It may be as simple as asking them! If you are feeling the pain, chances are someone else is walking in the same shoes. It is a great idea!


In our case I had the luck of having the support and help of great guys that were the fuel behind it all. They, more than anyone made everything possible. I'm really glad about that.

Find the barriers.

There will always be barriers and constraints. We need to know about them so that we can lift, and work around them. The base for this is communication. 

Almost the first thing we did after deciding that we wanted the change, was getting together and under suitable circumstances do some debugging of the processes we were following. What works, what doesn't. What were the options and the constraints we were aware of.  

We got together and made a list of all the problems that were affecting us, divided them in recognizable groups, and tried to find distinguishable roots for the problems, some technical, other managerial, personal, etc. so that we would better understand how to change them. 


Improve your skills.

Hone your skills, not just as a developer, but as a communicator. It will be useful.

Learn whatever it is that you need and when you know what you are doing, teach it to others. They say you remember only 15% of what you studied but 90% of what you taught.

Again, in our case, we organized a set of talks on the different subjects that we were interested in like: TDD, SOLID, Designing for test-ability, etc. 


We also sent links of free webminars and other free training sessions and resources like Coding Dojos, etc. to everyone interested.


We started helping  people to interact more with the agile community here in Barcelona, to promote the experiences and information exchange. Also, we used pair programming as a way of shortening the learning curve.


In general, we tried to create a friendly environment for learning. By that I mean, we got books, digital or not and made them visible and available to everyone who needed or wanted them. We also tried to allow time for learning, whether it is after hours (like in our case) or not.

Well, I have other stories from the trenches, but I think it is enough for now. Until another post and don't forget to start doing something about it. Above all things: have fun doing it! I know I am.