Systems Thinking and the Educational System in #Manitoba #agile #teachers

As a father of two younger children, I am taking an increasing interest in the educational system. Saturday during my son’s hockey game, the parents were trading stories of how we were graded when we went to school.

I told the tale of how in Junior High and High School, the teachers used to call out the grades in descending order. I remember it was a painful experience to sit there and hear your name not being called. I also remember how the process set the status of the students in the classroom. There was no guessing as to who the smartest and brightest were. Likewise, there was no hiding those students that were having troubles with the content.

For me, it motivated me beyond belief. I strived to be one of those first ones called. And when I worked hard enough and I was called, the pride and sense of accomplishment I had was unbelievable. And everyone knew that I was competing for the highest marks.

Looking back, I see that this system may have worked for me. But I am sure it hindered the development of some students as well.

New System

To try to mitigate these adverse affects, the Educational System places less emphasis on marks in the early grades. I’ve always had my concerns about the potential downside of this approach. How can I tell how my kids are doing with the subject matter? The new grading system is now less concise and allows for placement into four large categories. I find this grading more subjective and harder to interpret.

That isn’t the point of this Blog though…

It was mentioned by multiple parents of the boys that a new competition has arisen with the boys in the Grade 2 class. Since marks are not disclosed and discussed, the boys compete over the one visible metric – Finishing First.

The boys that finish their work first can move on to other fun activities. They can play or at least choose what activities they would like to do. Although the reports cards report that the boys rush through their work, they are just working in the defined system. By taking the focus away from grades, the boys are displaying the behaviour the system values.

Summary

This is a topic for a future Parent/Teacher meeting. I think I’d like to discuss Systems Thinking and the relation to behaviour in the classroom. The educational system has reduced the focus on grades and now speed is the behaviour the system values.

At least grades had some correlation to the learning of the content and ability to progress in future grades. Speed has no correlation to future grades and success in the real world. Speed only matters once you are correct.

For me, this was a great illustration of Systems Thinking. You get what you measure and value.

For me, I’d love to see an Educational System that measures and values:

1) What has been learned

2) Passion and Effort

3) Questions and Curiosity

By the way, I’d like to see teachers graded in the same way.

My SQL Saturday Experience #SQLsat175

I just recently completed my presentation at SQL Saturday in Fargo. The experience was excellent. The facilities at the Microsoft Executive Briefing Centre in Fargo were outstanding. Very impressive organization throughout the entire event. I am very much looking forward to presenting at future SQL Saturdays.

I was presenting on my experiences installing and configuring SSRS and SharePoint over the last few months. It was a bit of a journey and the presentation was my homage to the multiple Blog posts that helped me through my issues.

It was very gratifying to hear that other people had also encountered the same issues that I had. When you create a new presentation, it is always a nervous experience to learn if people find value in your presentation. Thankfully, the attendees did find the presentation valuable and had run into the same issues I had.

Thanks to everyone who attended and for their great questions and discussion after the presentation. It was very enjoyable, and I met some great people whom  I hope to trade experiences with in the future.

My only regret is that the Microsoft store wasn’t open on the weekend. I think we will definitely need to make a special trip down to Faro again to visit the store in the future.

Thanks again for the organizers for selecting my talk. I was honoured to present with the other speakers. I hope I can speak at other SQL Saturdays in the future.

Cheers

Terry

The #1 competency of a great Project Manager #pmot

I was talking on Friday to Steve Rogalsky (@srogalsky) about my thoughts on the #1 competency of great Project Managers. As soon as I said that I corrected myself and said that it is never that simple. There is not just one thing. That was something that makes for a great Blog title and Blog post, but is not a true reflection of reality.

But since this is a Blog post, I’m hoping you will allow me a little leeway in the discussion.

Project Manager Competencies

So what is the #1 competency for a Project Manager?

Think of the best Project Manager you have ever worked with. What was the one thing you remember about him/her?

If you were going to describe him/her to someone else, what would you say they did really well?

Let me hazard a guess that it isn’t one of these statements…

• Man, that guy really knew his way around MS Project
• Man, she really knew how to create a detailed WBS
• Man, that guy knows how to drive a team and instil a sense of urgency
• Man, she really kept on top of the work and followed a good governance process. I mean her Change Requests rocked!
• and so on…

My favourite Project Manager

My favourite Project Manager was not someone whom I liked or respected initially. I thought that he was weak on having a detailed plan and the change request process. (these were my pre-Agile days) I mean we were taking on extra scope and missing deadlines. Why was he not pushing the client and the team?

As I worked with him, I started to see the real skill he had.

• He never used his authority
• He built relationships with both the client and the team
• He knew when it was time to push and when it was time to be patient with both the team and the client
• He knew a great team took time to build trust and to gel
• He was a facilitator first and foremost
• He had an awesome sense of humour
• He honestly wanted to know how you felt and what you thought

I don’t know for sure, but I think he would have had great Emotional Intelligence

Emotional Intelligence

Emotional Intelligence is defined as:

“Emotional Intelligence (EI) refers to the ability to perceive, control and evaluate emotions.”  - about.com

Emotional Intelligence has four main competencies:

1. Perceiving Emotions: The first step in understanding emotions is to accurately perceive them. In many cases, this might involve understanding nonverbal signals such as body language and facial expressions.
2. Reasoning With Emotions: The next step involves using emotions to promote thinking and cognitive activity. Emotions help prioritize what we pay attention and react to; we respond emotionally to things that garner our attention.
3. Understanding Emotions: The emotions that we perceive can carry a wide variety of meanings. If someone is expressing angry emotions, the observer must interpret the cause of their anger and what it might mean. For example, if your boss is acting angry, it might mean that he is dissatisfied with your work; or it could be because he got a speeding ticket on his way to work that morning or that he’s been fighting with his wife.
4. Managing Emotions: The ability to manage emotions effectively is a key part of emotional intelligence. Regulating emotions, responding appropriately and responding to the emotions of others are all important aspect of emotional management.

(reproduced from  - about.com)

Empathy

To me, Emotional Intelligence comes down to being able to having great empathy for others. For your team mates and clients.

Great Project Managers use this empathy to build relationships and read situations when the project is starting to go off the rails. They understand that it is all about the people and not the technology or process. They honestly care. Care about team mates, the clients, and the solution.

Two More competencies

I also believe that a good Project Manager also has a wealth of experience with the type of project being executed. I’m not a fan of the concept that it is best if a Project Manager is not technical expert or domain expert. The Project Manager also needs to be a Problem Solver.

If you do not have the technical expertise, how can you fully empathize with the team?

If you do not have the business domain, how can you fully empathize with the client?

Summary

Steve was right. It isn’t that simple.

The best Project Manager I ever worked with had these three competencies:

• Excellent Technical expertise
• Excellent Business Domain knowledge
• Excellent Emotional Intelligence

Although it is rare to have all three, I have found that excellent Emotional Intelligence with either technical or Business Domain expertise is a very good indicator of success. And someone I would like to have on my team.

Are you playing #Project Battleship??? #pmot #baot #agile

I was talking in our co-located room last week on a particularly frustrating day to a co-worker. I enjoy talking to Hanaa as she has an adept analytical mind, great technical skills, and a great sense of humour. A rare find indeed.

Anyway, I was talking about my frustration where some discussions and emails exchanges didn’t seem to be moving forward to solving issues. They seemed to be just playing hot potato with who currently owned the issue. This finally came to a head when I tried to schedule a quick discussion to try to solve an issue and one person just replied back that they couldn’t make that time. I expressed my frustration at the lack explanation or options provided by the person. It was just stated they couldn’t make it, and it seemed it was back in my court.

Hanaa then said “Yea, that can be frustrating – this isn’t battleship”

LOL

Project Battleship

What a great analogy. Or metaphor? I can never keep those two straight.

Anyway, what a great image to keep in mind on projects. I think we are all guilty sometimes on focusing on just getting the issue off our plate rather than ensuring our actions are moving closer to solving a problem. I’ve kept this image in my mind ever since. I use it to guide me with all my actions. Are my actions showing commitment to moving closer to solving problems? Or am I just getting the issue of my plate and not moving it closer to resolution?

The image reminded me that we must always focus on two-way dialogue instead of the sometimes easier one-way email communication. It can be so easy to respond to an email with another email that doesn’t help to solve the problem or move closer to solving the problem. In many instances, additional replies to emails usually move the team away from a solution rather than towards it.

Summary

Do you sometime fall into playing Project Battleship? As team mates we must always ensure our actions move the project forward and not backward or standing still.

Focused, bi-directional communication is the only way to ensure the project is successful. Hiding behind our screens and issuing one directional commands or questions in email are an easy trap to fall into.

When is a plan good enough? Provisional Project Planning

As I’ve taken the journey along the Agile path I’ve come to learn two things I am certain about:

1) A detailed Work Breakdown Structure is one of the largest wastes and mistakes that can exist in a project

2) That not creating a temporal plan at some level can also lead to rework and failing late

On the surface, these comments may seem to be contradictory. I seem to be supporting neither traditional or Agile. And I would agree that it can appear in this way. But like most good solutions, the optimal solution is a compromise between extremes.

In my blog about whether Flow can be used on projects, I emphasize the need to plan. (if you are interested, you can find the blog here) I then continue the planning discussion in my Agile Chicken Little blog. (which can also be found here)

My intent of both posts was to communicate my thoughts on the fact that some level of planning is crucial. But when do you know when the plan is good enough? When is it excessive?

Too Much

I must admit I was someone who used to create detailed work breakdown structures. I even used to assign multiple people to one deliverable and have multiple items in progress. Looking back now I see that this was excessive planning for little value. The detailed project plan was also not an accurate depiction of how the actual project would execute.

But even when I started working on more Agile projects, I still sought out a moderate detail plan to try to comfort myself. I thought that we needed a full prioritized backlog of items. I held frustrating meetings with clients asking them if item 101 was more or less important that item 102. (If you were in one of these sessions with me, I apologize and owe you a beer)

Too Little

But as I stated in my prior posts, the creation of a temporal plan is critical to allow the team to think through the project process, dependencies, and possible conflicts. To start to do the project using just Flow may not allow you to fail fast. You may discover a dependency or conflict late in the process. It is possible that an initial plan would have highlighted this issue.

But creating a project plan at the start can create assumptions and issues all on its own. So what to do?

Just right – Provisional Project Plan

It the game of golf, there is a wonderful concept of a provisional ball. In the case where you have already launched a ball into the woods, you hit a provisional in case you cannot find the ball. It is seen as a time-saver to keep the game moving.

I use this analogy when I create my high-level project plans.

My provisional project plans are:

• Only one way out of many to execute the project – it is not the one correct way
• It is high level to get me to when I can look for more detail to refine the plan. (i.e. finding my first ball)
• It sets the expectations with the clients that the plan will change, but that we have at least thought through one way that the project could be delivered.

We then use flow to prioritize the next group of stories in the project. Once they are prioritized we use flow to execute.

Summary

In my experience, these Provisional Project Plans provide the right level of planning. They allow the team to think through the execution of the project without committing to ‘the’ plan. It also then provides the ability to execute in a flow-based structure within the Provisional Project Plan.

Special thanks to Luke Hohmann who illustrated the waste of creating a full prioritized backlog through the use of Innovation Games. By prioritizing and executing in groups of 10-20 items, we are able to efficiently work in a flow mode. (And update our Provisional Project Plan as we execute)

#Agile Chicken Little

I recently came across this video on deadlines and it helped crystallize a Blog Post that I have wanted to write for a long time.

Although the video is quite good an makes some very valid points, I fear the Agile movement is starting to exhibit some concerning signs. There are more and more Agilists proposing that we should not estimate or have any deadlines. I use the term Chicken Little in this Blog entry as I usually encounter exaggerated “the Sky is Falling” arguments when I propose that estimates and deadlines can be applied in a judicious manner and when managed properly, can result in greater value for the team and the project.

“The term Chicken Little has been applied to people accused of being unreasonably afraid, or those trying to incite an unreasonable fear in those around them” – Wikipedia

Estimates

Let’s start with estimates. It seems to be very fashionable to propose that estimates are useless and a wasteful exercise. Usually when estimates are mentioned the following Chicken Little cries are heard:

• You are proposing detailed estimates
• You are proposing estimates made by someone out side of the team
• You are proposing a detailed work breakdown structure
• You will punish the team when they don’t meet the estimates
• We are going to spend a month creating detailed estimates

Of course all of these cries are usually false unless poor project and company leadership exists. (If poor leadership is present, they will find other means to oppress the team without estimates and deadlines) These are exaggerations intended to make a point and portray the worst case when estimates are misused. Typically the opponents of estimates will also state that people are bad at estimating. The Beyond Deadlines video even makes this point.

Although estimating is a difficult activity, no one usually discusses how estimating is a skill. The reason people are bad at estimating is because it is a practice that they may not have been exposed to. No one ever discusses how people get better an estimation the more they estimate. The usual cry is that estimating is hard and we suck at it so we should never do them again.

In our company we have two rules:

1) The team creates the estimates (and plans) in all situations. They are never changed or modified by anyone but the team.

2) Friends don’t let friends estimate alone.

Most of the negative experience and stories around estimates propose situations where people are estimating alone and for the first time. (see rule #2)

Deadlines

Deadlines are starting to become the new cause célèbre of the  Kumbaya Agile movement. The history of the word “deadline” is reviewed as a line when people were shot. It is proposed that if you use deadlines, you must not trust your people and believe that people are inherently lazy. Once again we see the extreme hyperbole to try to make the point. 

Once again the use of deadlines, or milestones and target dates, can be a very useful method for planning a project. The use of them, does not mean that you don’t trust people. In fact, many people I have worked with ask for a planned end date so that they can use it to guide their work and prioritize.

The negative reputation of deadlines lie in the history when deadlines were created by management and foisted upon the team. Like estimates, deadlines or milestones need to be created by the entire team collaboratively.

Behaviour

Many opponents of estimates and deadlines propose that they can affect behaviour and damage intrinsic motivation. I would argue that this again is only evident when estimates and deadlines are used inappropriately.

We don’t need estimates and deadlines to motivate people. We need them to assist in planning the project.

Rationale

Estimates and Deadlines have been used in the past to oppress and punish people. We should be careful to not allow these situations to happen once again. But we do need to be careful that we do not propose elimination rather than moderation.

Just as the extreme application of estimates and deadlines was a mistake, so is the extreme eradication of them.

Why are estimates and deadlines a good thing? Because they require the team to plan. The elimination of both estimates can lead to a team not planning enough and this can have a negative affect on the project. The definition of plan I like is:

“a temporal set of intended actions through which one expects to achieve a goal.”

When we are evaluating project practices we need to be careful to not eliminate practices that are hard for the development team if they deliver value for the clients. We need to remember to do just enough of these practices to allow us to deliver the most value.

Can Flow be used on a project? #Agile #PMOT

I have been lucky enough to be on a large integration project lately. On this project, there are four distinct streams and they are being executed in different ways. I would consider all of them to be Agile projects, but some of the streams lean closer to Traditional to Iterations and some lean more towards pure Flow.

It got me thinking and considering if Flow can really be used on a project. Initially I was thinking that Flow wouldn’t be appropriate to use on a project. I had thought that Flow is aligned better with operational activities as opposed to a project. Typically a project is defined as an agreed upon amount of work that a person or team undertakes in accordance with an agreed upon schedule.

“A project in business and science is typically defined as a collaborative enterprise, frequently involving research or design, that is carefully planned to achieve a particular aim.” – Wikipedia

The main difference I see between the using Flow and more structured approaches like Iterations seems to me to be the lack of a Schedule. (Whether the schedule is high-level or detailed)

So I took off to Wikipedia to find the definition of a Plan.

“A plan is typically any diagram or list of steps with timing and resources, used to achieve an objective. See also strategy. It is commonly understood as a temporal set of intended actions through which one expects to achieve a goal.” – Wikipedia

Eureka.

Epiphany

After reviewing the definition of a Plan, I realized that I was not concerned about the use of Flow as much as I was concerned about the lack of a Plan. If a plan is “a temporal set of intended actions” then starting a project using pure Flow would be starting without a plan.

I am not proposing a Work Breakdown Structure or anything of the sort. But starting a project should require a temporal plan to allow the team to think through the activities, spot conflicts and prerequisites, and create a schedule. (at a level the team thinks is appropriate)

Summary

I think Flow does naturally have its place on very routine and repeatable processes. In can also be used on projects with great success. It does not however remove the requirement for a plan.

 

 

 

#Agile Data Conversion

Last year I presented on Agile Data Warehouse – the final frontier at SDEC12. You can find the presentation here if you are interested.

This year on the same project I have been challenged with another new frontier. Now that we are executing and evolving the Data Warehouse, the question was posed as to how we can run Data Conversion projects in an Agile way. The project has a requirement to convert massive amounts of data from old Legacy systems to a new application. These old Legacy systems are written in Cobol and have existed since the early 1970′s. Basically they are old-school Legacy.

We have encountered the traditional quandary. Data conversion projects and application development projects need to occur in parallel because of the constraints on the project. Then both of these streams of projects need to be validated and tested for the start of Integration Testing. This clearly can be a challenge as changes in one can affect the other. The issue is that if we wait until the end of all the design of the applications before we can consider the specifications for conversion to be materially complete, we would not then have adequate time to be able to develop, convert, and validate and reconcile for the start of Integration Testing.Although the application projects and teams have embraced Agile methods, the Data conversion teams have been quite hesitant. The focus for data conversion has still be to have a complete set of specifications and to test and validate the entire data conversion process.

Once Data Conversion is code complete there is a large amount of work required to reconcile and validate the converted data. Although there will be some test automation, the amount of investigation can be significant to resolve one defect. In short, data conversion validation will take much longer than application validation. Yet both streams need to feed the Integration Test.

Agile?

The question that was posed is whether this was a truly Agile process.

We didn’t deliver frequently and we didn’t minimize inventory. I would say we failed on both counts.

We were tasked to determine how we could possibly supply converted data for Integration Testing.

Epiphany

The epiphany was asking ourselves what an Agile Data Conversion would look like. We discussed that we would be able to do just enough to allow the testing to proceed. It turns out we are already doing some of that for the generation of Sample files when working with the package vendor. We are generating the files in accordance to the current specification and adapting as we learn more. So that is good – early and frequent feedback.

One thing we were not doing in our development was placing some rigor around how we adapted. In this case, doing things in an Agile way actually resulted some more structure. We still had to plan since we were time-boxed into when we needed to be complete. This was a change to ensure we had a plan to adapt rather than just ad-hoc adaption. Which all Agile project can suffer from.

But the one area we were not as Agile as we could be was the reconciliation and validation of the data conversions. We had planned that we had to balance all the conversions before we could say the data was ready for Integration Testing. This was going to result in multiple Mock conversions over many months.

Why?

The Solution

The solution we came up with was that we should be able to reconcile and validate the converted data iteratively. After we are able to convert a full set of data, we are not going to wait until we can validate all the data. We are going to find between 5-10 clients and their associated data that balance and look good. These clients and data will be simple at first. (although we will add complex clients if they balance) We will then add to the clients we use for Integration Testing as we are able to validate and balance them as we go along. If we can’t find 5 clients to balance off the hop, we have a larger issue. And in the spirit of Agile, it is surely better to know that right away that wait until we try to validate the entire set of data.

Ultimately we will end up validating all the converted data but in an iterative way.

I’ll report back as we see results. We also do have a plan B if needed.

Can an #Agile team mate #telecommute?

When Yahoo released the memo about employees not being able to work from home any more, there was the expected backlash against something which has become more and more common. It was especially unexpected given that Yahoo’s new CEO recently had a child and might be expected as someone who would find value in being able to work from home.

If you haven’t seen the story. Here is a link to one of the many articles on the subject:

Yahoo Reaction

Individual versus Team

My thoughts on this matter revolve around the concept of individual versus team performance. In fact, the link to the article provides a perfect quote:

“They didn’t lose my productivity,”

As someone who has a 6 and 7-year-old, I can’t see how someone who works from home could be as effective by working the same amount of time. Little questions and requests have context switches inherent in them. We always talk about the impact of task switching at work. We have to be consistent and also recognize the impact of task switching at home. So let’s assume that the person in question puts in the extra time to be as productive at home as they were at work.

OK.

What about the Team?

But what about team productivity? Unless you are working on a solitary project, having a remote member has to affect the team productivity. People work around the missing person. Instead of an immediate in-person discussion they have to make a phone call, instead of white-boarding they either create an electronic diagram or set up a video-conference. Now what happens when the remote person may not be available due to an urgent issue on their side? Technology can help, but the experience isn’t the same and there is an effect on the team.

But what about remote Agile teams?

My opinion, and it is only my opinion, is that I would prefer to not have to deal with a remote team. Now this isn’t possible in some circumstances, but we can’t pretend a remote team will be as efficient as a co-located team. The only question is how bad it will be.

I’d prefer Agile teams to always be co-located in one room. I don’t want to be in a separate room in one building, never mind some people being at home or across the continent.

Summary

I think Yahoo is correct. The company and teams work best when everyone is together. The collaboration and innovation that happens when people are together can’t be replicated.

I understand that are complexities that make everyone co-located a challenge. But if you gave me the choice, I’d choose everyone in same room every single time.

Every quote I had seen on this topic also had the individual aspect as the primary focus. The quotes proposed that “I” was still as productive. No one ever stated that the “team” was still as productive.

How to strategically prioritize user stories #Agile

One of the reasons I love to read books on Agile is because of the wealth of great ideas out there. It seems like every book I read allows me to discover one or two news ideas. The book I am currently reading is Ken Collier’s “Agile Analytics”. While reading this book I was introduced to the idea of prioritizing project requirements using the Purpose Alignment Model. (The model was proposed by Pixton et al in the 2009 book “Stand Back and Deliver”) Although the principle of getting the clients to prioritize your project requirements doesn’t change, the method does provide a structure and context that can inject new insights into the prioritization process.

Purpose Alignment Model

The Purpose Alignment Model proposes that you separate your project requirements into 4 categories based upon their strategic importance to the business. The categories proposed are:

Business Differentiating Activities – These are project requirements that support activities that are both marketing-differentiating AND mission-critical. These are strategic activities that will add to the growth of the company. These activities are usually aligned with new opportunities. The company risks the loss of growth if these activities are not done.

Business Parity Activities – These are project requirements that support activities that are mission-critical but not market differentiating. These are operational activities that will allow the company to maintain their current position in the market. The company risks losing their position in the market if these activities are not done.

Partnering Activities – These are project requirements that support activities that are not mission-critical but are market-differentiating.  These are typically partnering opportunities in the market to create new opportunity and growth. Unlike the Business Differentiating activities, this activity is looking to find a partner to share the cost of this market-differentiating opportunity.

Who Cares Activities – These are the project requirements that support activities that are neither market-differentiating or mission critical.

Conclusion

I’m planning to use this structure on the next opportunity to prioritize User Stories in the backlog. I believe it will allow for interesting insights into where the development team is spending their efforts in each iteration. I also think the clients would find the information interesting. Especially if we are spending more than 50% of our time consistently on developing functionality to support “Who Cares Activities”.

 

Adaptive Data Model – #Agile or Anathema?

I have seen the concept of an Adaptive Data Model proposed as an Agile method to Data Modelling lately. (Most recently in Ken Collier’s excellent book – “Agile Analytics”) The theory is that you can be more Agile using an  Adaptive Data Model instead of a traditional Data Model of the business domain.

Definition

An Adaptive Data Model is a Data Model that doesn’t model the business data. Rather it is a data model of the data model that models the business data.   The Adaptive Data Model describes tables that contains the meta-data that describes the data model. In this way, the entire Data Model is data driven and stored in a series of tables. The advantages of this approach is that changes to the model can be made by updating the meta-data in the tables. (as opposed to having to generate Alter statements to update the database structures in the database)

A sample Adaptive Data Model is shown below. (Copyright Ken Collier – Agile Analytics)

 

A subsequent layer would then need to be created to allow for the data to be extracted in a traditional sense by the application. (and to be made sense of by the business) Some suggestions for this layer have recommended that this layer could be created with a series of views or stored procedures.

Agile?

Although the Adaptive Data Model does allow for the easy modification of the Data Model, is it Agile?

I propose that an Adaptive Data Model is neither Agile or a Data Model.

According to Wikipedia a Data Model is:

“A data model is an abstract model that documents and organizes the business data for communication between team members and is used as a plan for developing applications, specifically how data are stored and accessed.”

An Adaptive Data Model does not meet this definition of a Data Model. It is a construct created to allow changes to be made to a data model.

In addition, I would propose that an Adaptive Data Model is also not Agile. It does not encourage frequent delivery and iterative development. An Adaptive Data Model is a complex solution that is not easily deployed in iterations to deliver value quickly and often to the business. If anything, it increases the technical debt of the project.

Summary

We as data professionals should be striving to make our processes more Agile and to be able to allow our processes to be refined iteratively like other areas of Software Development. Software Development proposed practices that allowed for iterative development. These didn’t include creating an Object Model of the Object model so that they didn’t have to fully embrace adaptive development practices.

An Adaptive Data Model feels like a short cut. It is hard to be Agile and iterative on data project, but trying to propose an Adaptive Data Model as a solution seems like a wrong turn. We need to find ways to allow our data designs to be change tolerant, adaptive, and test driven.

A Systems Thinking concern

I have been recently doing more reading into Systems Thinking due to the fact that several great articles have been written recently that I have had the good fortune to discover.

Definition

“Systems thinking is a holistic approach to analysis that focuses on the way that a system’s constituent parts interrelate and how systems work over time and within the context of larger systems. The systems thinking approach contrasts with traditional analysis, which studies systems by breaking them down into their separate elements. Systems thinking can be used in any area of research and has been applied to the study of medical, environmental, political, economic, human resources, and educational systems, among many others.”

Performance Feedback

Recently there has been a lot of discussion related to Performance Feedback and Systems Thinking. Much of this has revolved around how it is very difficult to separate one’s performance from the system this person finds himself in. I can personally attest to this. In a previous job, my behaviour adapted to the environment I found myself in. Even more interesting, the company underwent a merger with a sister company a few years before and I could see how my behaviour changed along with the culture of the company. Although I really had no idea that I was changing at the time.

So if have experienced the systems impact on behaviour and performance, what could possibly be my concern?

My Concern

My concern about Systems Thinking related to Performance Feedback is not that the system does not materially impact the behaviour and performance of the individual. It does. But it is a complex relationship.

My concern is that a misapplied system thinking approach might lead to someone thinking that they are 100% at mercy of the system. One can see by the definition above, that this is not Systems thinking. All Systems thinking says is that you need to consider the entire system.

In some discussions, there seems to be a distinction between the individual and the system. The system is really just the amalgamation of all the individuals. There is no us (individuals) and them (system) – we are all the system.

Summary

I’m a big proponent of Systems Thinking, but we need to ensure people use the term and concepts correctly. People will be impacted by their environment, but to different degrees based upon their roles and personality.

Most importantly, we should not hold the system solely responsible for anything. After all, we are the system. If the system is not optimal, our primary responsibility is to help improve it.

Memories of #ADABAS – My first database love

I recently was watching a DataVersity webinar on MongoDB schema design and I had flashbacks to my first DBA job at Investors Group in the early 90′s. You can find the webinar here.

MongoDB

MongoDB is one of the many NoSQL databases available. When I saw that DataVersity was going to have a webinar on MongoDB and how you define schemas in MongoDB I was very interested in learning about the topic. (It is a great presentation and introduction on the topic if you are interested)

One of the most interesting concepts was that you define documents which are like tables in SQL. These documents contain columns which are just name-value pairs of attributes when the data is stored. These documents can then be joined to other documents in one of two ways. Either they can be embedded into the main document or linked to the main document. This pattern is seen frequently is Object Oriented development but it is one that frequently causes issues in normalized databases.

These two methods also reminded me how I sometime like to organize tables in SQL databases along the same lines. I also like to identify which tables are truly independent (linked candidates) and which tables are dependent(embedded candidates) and need the context of other tables to typically be used. For example, Client and Product can be thought of as independent tables while the related tables of client_address and product_rate are dependent as they really only make sense when context is provided by Client and Product. I try to think of tables in these two ways as this categorization helps when we need to do Dimensional modeling.

I was intrigued with the functionality provided in MongoDB to allow for this embedding of other tables or documents.

But I did have concerns as all of the attributes to be saved in these documents are name-value pairs that are defined in each data command. There is also no consistency that is validated or verified on multiple data commands that operate the same documents. Yipe! Unlike SQL which separates Data Modification Language(DML) from Data Definition Language(DDL), in Mongo DB everything is a DML statement. As a result there is a lot of faith and confidence placed in the application to manage the integrity of the data. Although there is some additional work to define the data structure first and then operate on it, I feel that this structure is beneficial and has value.

Love at First Sight

As I was listening to the Webinar, I though back to my experience with ADABAS in the 90′s. After working with Oracle, Informix, Sybase, and SQL Server – I still think about the functionality I had with ADABAS. Unlike the other relational SQL databases, ADABAS did provide functionality that allowed you to embed tables/files in other tables/files through multi-value fields and periodic groups.(Tables are called Files and Indexes are called Descriptors in ADABAS, but the functionality is the same) Unlike MongoDB, ADABAS did provide functionality that required the creation of the tables/files first with their own DDL language and then modification of the data with their own DML.

In addition, ADABAS was able to provide performance throughput similar to what I have seen in any of the top-flight relational database engines. It certainly was high-performing. Now some of that high performance may have been helped by the lack of referential integrity provided by ADABAS. (That was the one drawback always mentioned when ADABAS was compared to other relational DBMSs) Any referential integrity must be maintained by the applications.

In retrospect, I don’t think I realized what a good DBMS I was using at the time.

I noticed that ADABAS now has a community edition that I am currently downloading… Maybe I will see what the technology looks like today.

Summary

Although MongoDb looks promising, I think ADABAS should try to inject their name and product into the NoSQL discussion. They have been providing NoSQL databases for over 40 years. They were cool before it was cool to be cool.

If I’ve peaked your curiosity, here are a couple of ADABAS links:

ADABAS Wikpedia Page

ADABAS Home Page

What dance do you do when your project is Yellow? #PMOT

It seems that everyone is pretty comfortable when it comes to the definition of a project that is either green and red.

Green – everything is going well, on track for schedule, scope, and budget.

Red – serious problems that need stakeholder help to resolve. Probably unable to materially meet schedule, scope, and/or budget.

But Yellow?

Yellow seems to be the colour in the middle where there is a wide range of status and actions.

To some, any amount over budget turns the project yellow. Some only turn it yellow after the schedule, scope, or budget overages exceed 5%.

To some, a yellow project requires an action plan. Some turn it yellow to indicate that it is being watched, but that no actions are required.

Which is correct?

I would suggest that depends on the culture of the project, team, and client. But I do tend to see two types of yellow dances that occur on projects.

The Yellow Dances

Using using the dance metaphor, I am proposing that the two partners in the dance are the project and the project status. The dance that they do reflect the interactions that occur during the lifetime of the project.

The Green Waltz – I use the Green Waltz dance to refer to the more formal behaviour of the project and project status. In this dance, the project and project status relationship is quite formal. The status is usually communicated via status reports. Much of the focus of the project is to do whatever it takes to turn the yellow project back into green as quickly as possible. As a result, this dance is very quick as the change request tempo can become very up tempo. During the course of the dance, the yellow/green transitions(change requests) can come at a furious pace.

Pros – Any exceptions to the plan are highlighted.

Cons – The project may be green at the end and people may think it is fine, but there may have been numerous change requests that only give the appearance of everything being fine. The status does not represent the holistic status, only the current point in time status of current plan against current actuals.

The Yellow Tango – Unlike the Green Waltz, the Yellow Tango is less about seeking to become green and more about understanding the rhythm of your partner and reacting. The Yellow Tango embraces yellow and does not try to guide it back to green. The dance understands that both partners lead the dance at certain times, and tries not to force it back to green without good reason. The Yellow Tango is a slower dance, without the up tempo of numerous change requests. But if a change request is required it usually is a larger, sudden movement and change in the dance.

Pros – the status holistically communicates the status of the entire project, not just at that point in time.

Cons – The reduced formality can result in delayed escalation of issues until they become large enough. This may be an issue depending on the dance hall you are in. Some dances halls are ok with these exaggerated movements, some not so much.

Summary

Both dances are appropriate and neither are right or wrong. Many times, the dance will be stipulated by the client/dance hall.

My personal preference is for the Yellow Tango as I feel it does communicate the overall status as opposed to just a point in time. As with the Tango though, it is a more involved and complex dance. I think it does required a more educated client and team. I also find the  Yellow Tango better for Agile projects when things are just bigger or more complex without added scope. The Yellow Tango more honestly recognizes that we don’t know everything up front and we will manage the items together in clear, open, and honest communication.

It takes a village to install an integrated #SSRS/#Sharepoint environment

On a recent project we selected SQL Server Reporting Services (SSRS) as the reporting solution. During the investigation we also chose to use Sharepoint as a portal/repository for the reports as the client did not have a current portal/repository solution. I should mention that the version of Sharepoint we decided to use was Sharepoint 2010 foundation as it served our initial needs very well.

Although I have installed SQL Server many times, I only have experience using Sharepoint, not installing it. But I thought it would be good to get experience and to learn how to install and at configure a simple Sharepoint installation. I mean, how hard could it be?

The Google Experience

I figured the best place to start would be at the alter of Google. After making a sacrifice of my pride, I began looking for Sharepoint 2010 installation guide. Got one on my first search! After I downloaded it I was dismayed to see that it was over 600+ pages long. It can’t be this complicated to install a basic Sharepoint installation can it? Isn’t there a cheat sheet just to do a basic installation for use with SSRS?

The Amazon Experience

After being somewhat defeated by Google, I went to my second favourite source of information – Amazon. On Amazon, I found the WROX book – “Professional SQL Server 2012 Reporting Services“. After flipping through the table of contents, this book looked perfect. It even had a chapter on how to install SSRS and Sharepoint in integrated mode and how to create reports and maintain content through Sharepoint. I spent some time reading what I felt were the important chapters and then I planned out the installation. Seemed pretty straight forward. But still I had this feeling of impending doom.

The installation experience

The book provided quite a good guide for how to install SSRS. I had no issues at all. Alright, my confidence was building. I then started to install the Sharepoint integration. The guide for the initial installation of Sharepoint seemed straight forward. I followed the instructions and Sharepoint was successfully installed. The guide instructed you to not configure Sharepoint at the end of the installation since we will do that after we install additional SSRS features. Alright. Fair enough.

I then proceeded to install Service Pack 1 for Sharepoint 2010 as instructed. No issues. Alright! Gaining confidence. One step left.

The final step walked you through installing the additional Reporting Services features for Sharepoint. Once again I followed the guide and had no problems installing SSRS for Sharepoint. My installation was complete!

Or so I thought…

The frustrating experience #1

The next step required you to create a Reporting Services service application using Sharepoint Central Administration. Alright, fair enough.

Hold on, when I try to start the Sharepoint Central Administration it provides a message saying we need to configure Sharepoint. Remember in the process where we de-selected the option to configure Sharepoint and we were going to come back to it? Well, the book never came back to it. Thankfully I found a great reference in a blog for how to manually configure Sharepoint 2010. I followed the steps and Sharepoint was configured. You can find the link to that blog here: How to manually configure Sharepoint 2010.

The frustrating experience #2

OK. I was able to configure Sharepoint, but when I tried to create the Reporting Services service application, Sharepoint would not allow me to create a Report Services service application. It was not listed as one of the service applications I could create. Something was still missing. This problem was much harder to track down.

After searching and searching I came across a blog from Sharepoint MVP Liam Cleary. (Liam, send me your email if you read this post. I owe you a bottle of Single Malt scotch) In Liam’s post he clearly documented how you need to run two commands in the Sharepoint 2010 Management Shell to install the core services into the SharePoint Farm and to provision the service proxy.

Once that was done, I was able to create the Reporting Service service application and the installation was complete.

You can find Liam Cleary’s blog post here: I owe Liam Cleary a bottle of scotch

Summary

All in all, the installation of SSRS integrated with Sharepoint was simple once I located the two missing pieces of information. Thankfully there is a lot of information about SSRS and Sharepoint out there. Once I got the environment implemented, I couldn’t be happier. Easy to use and we are creating reports as well speak using Report Builder.

Good thing there was a village to help me configure my Sharepoint farm.

Why #Dimensional Modeling matters

I’ve recently completed a data modeling initiative on a major project. After doing this I’ve come to two major conclusions:

1. The coverage area in Insurance is probably the most devious and twisted area of data that I have ever modeled.
2. Dimensional Modeling should be done on every model to ensure you can simply model the domain.

Why Dimensional Model?

For the model I worked on, there are over 261 tables in the Relational Model. In the Dimensional Model, there are only 25. The process to distill a Relational Model to a Dimensional one is not easy, but the process highlighted problems in my Relational Model that I was unaware of. These issues would have been exposed later during testing or post-production, but being able to identify them in development was extremely valuable.

So what is Dimensional Modeling? although most people have been exposed to the rules of Data Normalization, not as many people have been exposed to Dimensional Modeling. I will try to explain my opinion of Dimensional Modeling, but if you are interested I would highly recommend ‘The Star Schema Reference’ by Chris Adamson. Simply put, it is the most complete and concise book ever written on Dimensional Modeling. Dimensional Modeling is usually done to create a model for a Data Warehouse. This Data Warehouse can then be used for Operational and Analytic reporting. But the process of Dimensional Modeling does not need to be limited to a Data Warehouse. The process itself has value to validate the Relational Model.

The term that is commonly used to refer to a Dimensional Model is a ‘Star Schema’ model. This is due to the fact that most Dimensional Models look like a star. (With the Facts in the centre and the Dimensions around the Fact)

Dimensional Modeling

Dimensional Modeling is the process of taking a Relational Model in some normal form and being able to distill the many tables down to the business objects that the business works with. In most cases this would result in single tables that correspond to objects like:

• Client
• Product
• Sales Rep
• Store
• Sales Transaction

These business objects fall into two categories:

Facts – These are Facts about the business. They typically correspond to events or transactions and have metrics or measures that the business is concerned about. In many cases these fact tables are what the business wants to report on.

Dimensions - These are Dimensions to the Facts. They typically correspond to business objects that interact with Facts. These Dimensions represent how the business describes these objects and how they like to slice, dice, filter, sort, group, and order the Facts. A standard Dimension that occurs in all Dimensional Models that does not correspond to a business object is Time. Time is a dimension for almost all business events and transactions.

Initially, the act of creating these Facts and Dimensions is a large task of de-normalization. Although this sounds like a simply task, it is anything but. A Relational Model with hundreds of tables must be distilled down to a handful of tables. Many to Many relationships must be transformed into simple relationships that are easy to understand and report against. A natural key must also be defined that uniquely identifies a row in the Fact and Dimension. Although this sounds easy, you may find that some Dimensions don’t actually have a natural key. This is a clue that the Dimensional Model needs more attention.

A Dimensional Model also places the modeling of history at the forefront. A lot of attention is spent ensuring that data can accurately represent the changes that occur over time. Most of the time, a Relational Model is primarily concerned with current state.

Note: This is a gross over-simplification, if this is interesting to you I would recommend the book by Chris Adamson on all of the theory and complexity behind Dimensional Modeling. Or you can just start Googling!

Conclusion

The act of creating a Dimensional Model and creating the 25 Facts and Dimensions highlighted inconsistencies and duplication in my model. It also challenged my understanding of the data domain. It raised questions that I realized I didn’t have the answer for. It is easy for inconsistency and errors to hide in data model with 200+ tables. A data model that has 10-30 tables has nowhere to hide. It has the brutal transparency so valuable in Agile.

Why #Dimensional Analysis should be done on every #datamodel

Those of you who have worked with me, know of my fondness for Operational Data stores. I have always believed in the importance of having an enterprise or holistic view of the data requirements for every application. An Operational Data Store seemed to be the perfect vehicle to ensure this happened. Perhaps my fondness was related to not wanting to stray too far from the normalization rules that I knew quite well. In this way, it was a new-ish discipline or context that really wasn’t new.

I always looked kinda sideways at those weird Dimensional modelers with their Star Schemas and Snowflakes. I mean if they really put their mind to it, they would be able to figure out how to solve their data needs with a nice relational normalized Operational Data Store, Only exceptional and massive amounts of data require the Dimensional modeling constructs that these models typically use right? I mean what is so complicated about a model with only 100 main tables? Shouldn’t everyone know how to write SQL by hand?

On my latest project, I have had the opportunity to become re-introduced to Dimensional Analysis and modeling and I have found the process fascinating and very valuable. Besides the obvious benefits that are being realized by being able to model the data in a way that allows the clients to efficiently write and execute queries, there was an unexpected benefit.

Taking a normalized Data Model and attempting to translate it into a Dimensional Model really challenges and validates your data model. It is easy to create a model with a multitude of complex relationships than it is to distill in down to a handful of FACTs and Dimensions. With so many relationships, it is possible to inconsistencies to exist and hide in the data model. I found multiple modeling errors in the process on trying to create a Dimensional model from my relational model. When you distill a relational model down to a Dimensional model, inconsistencies and errors become very apparent in the creation of the FACTs and Dimensions.

Dimensional Analysis also forces you to look at the data in a different way. Instead of a relational/hierarchical way, I find it forces me to look at the data in a chronological way and forces me to consider data changes, data history, and data latency in ways I may not have considered before. Not having to account for data across time and verify consistency at every point is quite a bit simpler.

Summary

I am a convert of using Dimensional Analysis on all my data models for validation of the data model and additional analysis of the data.  I’ve discovered that I need to understand the data better to create a Dimensional Model than a normalized model. More factors need to be considered and creating the Dimensional model with fewer objects requires that the data model has greater consistency, integrity, and cohesion.

Simple is hard. 

 

Top Three Rules of #Agile Software Estimation

Yep, Estimating is hard. It isn’t easy and it isn’t without peril. Unfortunately it is a fact on 95% of the projects we work on. So given that, I’m not sure that telling people not to estimate is productive. There are a lot of misinformation about estimating though. Not all estimates are evil and not all are a waste of time. Here are the three estimating rules that I follow that have served me well.

1. Estimate Iteratively

I can’t figure out for the life of me why people who discuss Agile projects assume that if you estimate you must be BEUF. (Big Estimate Up Front) I always estimate iteratively with the estimate getting more refined as I go along. Usually the stages are:

Proposal Estimate – At the very start of the process when you don’t know much

Plan Estimate - Creating an executable plan and estimate before you start after you know more about scope

Execution Estimate - Re-planning with the final team just before execution

Iteration Estimate - Re-planning before each Iteration

With an iterative approach to estimating you can provide a line of sight to the clients as to what they can expect and you can also communicate how the whole project will evolve. Even the estimates.

2. Estimate with the team

This is a no-brainer. Always, always, always estimate with the team. If your team composition changes, I would re-estimate with the new team. Friends don’t let friends estimate alone. It is as much a learning experience as anything else.

3. Estimate the Minimum Viable Product

This is probably the most important rule. Let me be clear. You have no business starting or being involved on a Software Development project if you can’t estimate that you can at least deliver the Minimum Viable Product. (MVP) Yes, I know it is hard and difficult and fraught with errors, but an estimation based on years of experience if better than no opinion at all.  It can save wasting money on a project that had no hope of success. And it doesn’t need to be a very detailed estimate at the start, just a high-level estimate that confirms the team believes it can be done. (see rule #1)

Some Estimating Falsehoods

There are some common Estimating Falsehoods out there. I have taken these from the Blog post on estimating by Matt Rogish that takes the point that estimating is harmful.

Here we go:

“In really terrible places to work, someone other than the developer will actually do the estimating. This estimate will then be given to the developer as an implicit (or at especially evil places, explicit) time not to exceed. This is such a depressing situation I cannot dare think of it further.” – Correct. This is really a problem with bad leadership rather than estimating. Don’t throw the baby out with the bath water.

“Most estimation is drawn from past experience. What if the developer doesn’t have this experience? How can they estimate? This is where things get tricky. And by tricky, I mean “completely made up.” Sure, for things that have been done before by someone the developer can draw some parallels and make a guesstimate. But what about the stuff that has never been done before (presumably, the secret-sauce that makes you money)? Usually you just take a Wild-Ass Guess (WAG-estimation) and hope you’re not wrong. Given the rarity of being punished for under-promising and over-delivering this WAG tends to be a massive over-estimation.” – This is a common Falsehood. As mentioned in Rule 2, no one estimates alone. Senior developers help the less experienced and share their wisdom. This point also again highlights just plain old bad management.

“Completely spec out the entire system ahead of time, spend a lot of time researching and estimating the problem, determine project dependencies, and you can determine the “finish date” before writing a line of code! It’s seductive and taps into the part of our brain that craves order and dependability.” – See Rule 1 – Estimate Iteratively. I can’t imagine any Agile professional would fall into this line of thinking.

“If the value of software we’re writing is so low, is it worth being written? If the project has such tight time constraints that a schedule slip will doom the project then you’re in a world of hurt. (The solution is agile: work on the most important things first, have an always-working project, and cut scope to hit a date.)

This exposes a major weakness in most software companies: the inability to determine project value. Given a project of unknown value, the conservative business will then attempt to minimize cost. Unfortunately as we’ll see, cost minimization ends up becoming very expensive in the long run.” – This is a strange point to me. We as software developers can’t estimate our projects, but YOU the business need to estimate the value. Well we can’t have it both ways. I agree Agile is the solution, IF we estimate we can complete the Minimum Viable Product within the current budget. See Rule #3.

“I’m not entirely sure what this means, but I’ve never seen the folks who are writing the specs, requirements, etc. ever being asked to estimate how long each spec will take to write. Or have the CEO give a date when the company will hit some arbitrary metric, although see the perversity in public company earnings estimates and reporting.” – I’ve seen both on all the projects and companies I’ve worked for. CEO bonuses are commonly tied to unrealistic estimates on an arbitrary timeframe.

“Ultimately companies require estimation because they don’t trust anyone. They don’t trust developers to deliver results. They don’t trust mid-management to motivate their reports. The underlying reality is that the folks “in charge” are worried that everyone is going to rip off the company and only by holding people to arbitrary deadlines are they assured stuff will get done.

Committing to a date allows them to sleep better at night and shifts the blame from them to the folks on the front lines. ” – Wow. If anyone works at a place like this, leave. But don’t blame the estimates. If you don’t produce estimates, this company will just find another way to pull a fast one on you.

“Estimation is ultimately a futile effort. Software, more or less, is like writing poetry. Or solving mathematical proofs. It takes as long as it takes, and it’s done when it’s done. Perfect, or imperfect, estimation won’t change how long it takes to complete the work. If that sounds horrible to you, then go do something else.” – I could not disagree more. It is correct that estimation won’t change the amount of work it will take to complete something, but that isn’t the point. The point is about giving some imperfect information to the client to help him/her make business decisions. You remember the client right? The one who is going to pay the bills and we are trying to deliver value to right? I would agree that truly unique software solutions are like writing Poetry. I was on a R&D project recently and the estimating was challenging to say the least. But 95% of Software Development projects are following semi-established patterns. The projects are not routine, but nor are they like creating something fundamentally new all the time.

“Estimation actually slows down development because estimation isn’t free. The developer is incentivised to take as long as possible estimating in an effort to not get beaten when they miss the estimate. Why waste that time when the metric the developer is attempting to hit ultimately generates negative value?” – See Rule 1 on estimating iteratively and this is just an example of bad management once again.

Summary

Estimation is difficult, never correct, and is fraught with danger. But if you are like me, the vast majority of my clients require them. So the question is not why can’t we stop doing estimates?,  but rather how can we do them better?

In almost all cases, the problems with estimating can be traced to bad leadership on management. That is where I believe our focus should be.

Why do we #DataModel at all?

People in the Database world take Normalization and Data Modeling as something that should be done without question. I compare it to best practices like versioning software. No one expects that anyone would create software without version control anymore.But more often recently I do get questioned and challenged on why we need to normalize and model data. Is it even required with the cheap disk space, memory, and server capacity available ?

According to Wikipedia and others, the objective of normalization is:

“Database normalization is the process of organizing the fields and tables of a relational database to minimize redundancy and dependency. Normalization usually involves dividing large tables into smaller (and less redundant) tables and defining relationships between them. The objective is to isolate data so that additions, deletions, and modifications of a field can be made in just one table and then propagated through the rest of the database via the defined relationships.”

The rules of Normalization

Normal form	Brief definition	Violation
1NF	A relation is in 1NF if and only if all underlying domains contain scalar values only	First Normal Form is violated when the table contains repeating groups
2NF	A relation is in 2NF if and only if it is in 1NF and every non-key attribute is irreducibly dependent on the primary key – every column must depend solely on the primary key	Second Normal Form is violated when a non-key field is a fact about a subset of a key
3NF	A relation is in 3NF if and only if it is in 2NF and every non-key attribute is non-transitively dependent on the primary key.	Third normal form is violated when a non-key field is a fact about another non-key field
4NF	Relation R is in 4NF if and only if, whenever there exist subsets A and B of the attributes of R such that the (nontrivial) MVD A->>B is satisfied, then all attributes of R are also functionally dependent on A.	Fourth Normal Form is violated when a table contains two or more independent multi-valued facts about an entity. In addition, the record must satisfy third normal form.

In relational database theory, second and third normal forms are defined in terms of functional dependencies, which correspond approximately to our single-valued facts. A field Y is “functionally dependent” on a field (or fields) X if it is invalid to have two records with the same X-value but different Y-values. That is, a given X-value must always occur with the same Y-value. When X is a key, then all fields are by definition functionally dependent on X in a trivial way, since there can’t be two records having the same X value.

The Questions

Now that we have reviewed the objectives and rules of normalization, let us summarize. The objective of Normalization is to:

1. Minimize Redundancy
2. Minimize Dependency

But what if we have extra storage available and storing redundant copies of data is not a problem? In fact, it probably will speed up our query response time. What if we also don’t require frequent modification of data so that having de-normalized data won’t result in update or deletion anomalies caused by excessive data dependency? Why should we still model our data and normalize?

The three reasons to Data Model

Simplicity, Consistency and Integrity, and Future Flexibility.

Simplicity

Every one of the violations mentioned above would require extra code and extra unit tests to validate proper functioning. Depending on the amount of violations, this can become a severe amount of technical debt that will needlessly be required in the software. There is an entire movement dedicated to the elimination of If statements. (www.antiifcampaign.com) Software that is Data Driven rather than Condition Driven is simpler and easier to maintain over the life of the application.

An application that is Data Driven can also automate the creation of their test cases to validate proper functioning of the application. This combined with the enhanced simplicity greatly adds to the quality of the application.

Consistency and Integrity

Even if the solution being modeled can accommodate redundant data and has the potential for minimal update and deletion anomalies currently, significant risk is being assumed by having these potential situations in your data model. How can you ensure that redundant data will be kept in sync and that update and deletion anomalies do not get introduced in the future as people and requirements change? Either this is through additional software development code or by additional processes and latent knowledge in resident experts. Neither of these situations are a good use of time and energy.

This is an example of an application-centric view of the data model. Unfortunately, not all activity on the Data Model can be guaranteed to always go through the application. Data Fixes, Conversions, and enhancements all have the ability to bypass the application’s business logic and compromise the integrity of the data. All it takes is one high value client with inaccurate or inconsistent data to irreparably harm a company’s reputation and revenue stream.

Future Flexibility

Solutions that are data driven and do not have excessive functional dependencies are much easier to evolve in the future. For example, I may have a business requirement to split one account type or combine account types. This type of conversion will be quite routine if I have modeled my data properly and minimized dependencies. If not, the conversion can be quite convoluted and I will probably need to evaluate code before I can determine the implications of making such a change. Then I have to be sure I address and update all the redundant code throughout the application. Just because the situation doesn’t exist currently with update and deletion anomalies doesn’t mean those situations won’t happen in the future.  

In addition, these changes to split or combine account types would probably also require code changes. If the solution was Data Driven, the possibility of these code changes would be minimized. (not to say they would never be required, but the probability of code changes would be minimized)

Summary

A well designed application and user interface will be able to be used with minimal training. It just makes sense and models the clients current processes and tasks.

A well designed data model should also have the same intuitive qualities. It also makes sense and models the business’s data. Not how the application functions, but how the business exists. Modeling the data in this manner minimizes work currently to work with the data and in the future to accommodate change.

In Object Oriented parlance, the Data Model itself should be loosely coupled with high cohesion. Both the Object Model and Data Model should share this characteristic. (Although they will implement it in quite distinct ways)

#Agile Goalie

I came across a quote from Ken Dryden on the role of a Goaltender in hockey and I thought it was a great analogy to what I believe a great Project Manager is. Anyway, here is the quote:

“[A goalie's] job is to stop pucks, … Well, yeah, that’s part of it. But you know what else it is? … You’re trying to deliver a message to your team that things are OK back here. This end of the ice is pretty well cared for. You take it now and go. Go! Feel the freedom you need in order to be that dynamic, creative, offensive player and go out and score. … That was my job. And it was to try to deliver a feeling.”

Although I have read almost all of Ken Dryden’s books I did not remember coming across this quote before. I feel it also communicates the two responsibilities of a great Project Manager extremely well. These two responsibilities are:

1) Stop the Pucks – Manage the project

This is more the traditional expectations from a Project Manager. Create the Project Plan (in whatever format), manage the plan, resolve issues, submit Change Requests, and produce project communications. These are the more explicit expectations of the role. And like Goaltenders, you need to be proficient at doing this responsibility before you can hope to move onto the second. You can’t inspire confidence unless people believe you are competent in the basic role.

2) Encourage creativity – Inspire the team

Once you are able to convince the team that you are competent, you can move to encourage creativity. This is perhaps the greatest attribute of a great Project Manager, to be able to inspire confidence and trust. If a hockey team is unsure of their goalie, they won’t make that daring cross-ice pass, their defencemen won’t pinch, and the forwards won’t make a blind pass to the open wing. They will stifle their creativity and innovation because they are not confident in the outcome if those actions do not go as planned. But when the team knows their goalie has their back and trusts him or her 100%, the offensive creativity and innovation in unmatched. It is no coincidence that every great dynasty in hockey had a great goaltender who inspired that creativity. You can just name the goalies – Ken Dryden, Billy Smith, Grant Fuhr, Patrick Roy…

To extend the analogy even further, the coach will now create challenging and innovative game plans and strategies because he has confident in his Goaltender. Now not only does the team take greater risks that can result in great things, but the project stakeholders will take on more risk because they trust the Project Manager and the team. That is truly a high-performing team.

Summary

Inspiring confidence is two-fold. One is being confident and having the team feed off of your confidence. This is very important and a project with a Project Manager without confidence in the project and team is lost from the start. Two is having the team understand that you are 100% part of the team. Sometimes on projects there tends to be a distinction between the Project Manager and the rest of the team. But when you can sit in the room with the entire team and do project work alongside the entire team, the confidence and energy the Project Manager can instill will encourage the team to accomplish unbelievable things.

If you need any additional proof, here is the classic Ken Dryden pose.

This pose just inspires confidence and communicates that ‘Things are fine back here, I don’t have a concern and neither should you’

How an #Agile Data Warehouse leveraged an #InnovationGame – Iteration 2!

A few weeks ago I authored a post that explained how were leveraging an Innovation game on my current Agile Data Warehouse project. You can find the original post here.

Iteration 2

One of the aspects I love about Agile is the freedom it allows you to be bold as it acknowledges it is impossible to get it perfect off the bat. I find that this encourages people to take a chance and try new things. If it doesn’t work? No problem, we will just adjust and get better as we progress.

My initial blog proposed using a visual Innovation Game with Visual Report Boards to allow for brutal transparency and the management of the data requirements for an Agile Data Warehouse project. Each Visual Report Board had an object at the centre with 6 dimensions around it that illustrated how the object could be reported on. (I affectionately refer to these boards or diagrams as Hexes) Our Objects were the people who the corporation were interested in. It turns out that this method has been successful. It has allowed us to create a visual backlog of data requirements and have the clients prioritize the work and guide the work according to the overall business priorities.

Why am I writing this blog then?

Well it turned out that once we started placing the data requirements or reporting stories on the Visual Report Boards, it became apparent that our objects and dimensions of the diagrams required tweaking.

Initially I had placed a Person object at the centre of the diagram. The dimensions where then aspects of how we need to report on that person. Although this object and dimensions may work for other projects, it did not work for our project. (Actually I think that most Data Warehouse projects would probably make the same change we did, but I’ll let you decide that for yourselves)

Our experience was that most of the reports or data requirements were very clustered on the Visual Report Boards and the diagrams did not allow for the visual communication of what the report was or what data was required. I was starting to worry that this process might not provide the brutal transparency that allowed for the efficient creation of a Data Warehouse in an Agile way.

Just the Facts

After reviewing the requirements, it became apparent I had the wrong objects at the centre! Rather than people at the centre, the objects at the centre should be transactional data. The centre objects needed to be the actual data that was summed, aggregated, filtered, sliced, and diced. Once I made this change, the value of the Visual Report Boards increased exponentially. Now they communicated the content and purpose of the data requirements.

The real indication that I was on the right track is that the Revenue and Expenses Hexes I now had were also the first two Fact tables that were needed in the Data Warehouse! This method of visualization and analysis was aligned 100% with the Data Warehouse design process. Of course the Hexes were Fact table. This made perfect sense. I imagine more Hexes will be needed in the future as we discover the need for more Fact tables.

In addition, we created one more Master Data Hex as some reports and data requirements are not related to transactions.

Summary

I am convinced the use of these Visual Report  Boards and the related use of an Innovation Game enable the creation of a Data Warehouse in an Agile manner. We are executing in Iterations and not increments and the clients are thrilled with the control, visibility, and value they get every 2 weeks. I’ll post another blog once we have created enough of the Fact tables to provide more lessons learned.

SQL Server 2012 – Cheaper than Open Source Database options

On a recent project we were tasked to review and recommend a database technology platform for a new Data Warehouse environment. This quickly turned into a database evaluation not just for the new Data Warehouse, but also for all future applications. The client was currently on Sybase, but was open to the most cost-effective option. The client also had experience with Open Source technologies in the past for both server operating systems (linux) and reporting solutions. (JasperSoft)

The criteria we specified had cost and performance as two important factors. Since we had limited experience with any Open Source databases in a Data Warehouse environment, we felt it would be useful to evaluate the options from a cost, functionality, and performance point of view.

This post is a recap of that process and of the somewhat surprising recommendation.

The Candidates

Since the client already was using Sybase, it was a no-brainer to include that one. It addition to Sybase, we included SQL Server and Oracle from the commercial DBMS’s due to their market share and existing use at the client. DB2 was excluded due to DB2 not currently being used at the client and the perceived higher cost of implementation of DB2. (Rightly or wrongly) It was felt that either Oracle or SQL Server could represent the commercial DBMS’s.

On the Open Source side, the DBMS’s MySQL and PostgreSQL were selected. MySQL was selected due to its large adoption in the Open Source community and recognition for performance. PostgreSQL was selected also for its recognition in the Open Source community for robustness and performance. MySQL and PostgreSQL were the only two Open Source DBMS’s as it was felt they were the clear leaders in the Open Source DBMS’s.

After performing detailed performance tests between MySQL and PostgreSQL, PostgreSQL was chosen as the Open Source DBMS option to compare against commercial DBMS’s. This was due to the fact that MySQL had significantly poorer performance when using the InnoDB engine that guaranteed ACID compliance and provided referential integrity. MySQL has an interesting piece of functionality to allow you to use different database engines that provide different functionality and types of processing. Unfortunately the engine used in many of the published benchmarks do not provide the functionality required by standard enterprise DBMS solutions.

The Requirements

The current Data Warehouse requirements were reviewed and it was determined that the vast majority of the reporting requirements were operational reports delivered in text format. This is probably not unlike many companies out there that are early in the use of Data Warehouses. This is not to preclude the use of more analytical functions in the future, but that was not the use of the Data Warehouse currently.

The current Data Warehouse also stored data in the 10′s of gigabytes. So although this was a considerable amount of data, it also did not require advanced Data Warehouse appliances or Big Data solutions. Traditional Data Warehouse architecture and designs would be sufficient for now and well into the future.

The requirement was that the Open Source solutions must be supported. For example, this meant that we would require support from EnterpriseDB if we chose PostgreSQL. I propose this is pretty typical for larger companies evaluating Open Source.

The Architecture

Due to the current requirements and expected future requirements, the recommended Data Warehouse solution architecture was designed to incorporate both an Operational Data Store and a Star Schema Data Warehouse. The Operational Data Store would be used to generate pure operational reports and would be loaded at least daily. The Star Schema Data Warehouse would be used to generate more analytical reports and would be loaded on a weekly basis.

The Evaluation

The recommended configuration for the evaluation was for two servers each with two Quad core CPUs. This was due to the current workload and the expected increased workload in the future. We did evaluate higher levels of configuration to ensure the cost was linear for future growth.

This was not a small Data Warehouse and is probably typical of most enterprises as an entry into Data Warehouse technology.

We separated our cost evaluation into two sections:

1)       Initial Cost

2)      Ongoing Cost

Initial Cost

Initial cost incorporated the server and DBMS license fees for both development and production environments and also two human factors:

Training – What is the cost of the initial training required for the administrators.

Efficiency – What is the expected efficiency for the administrators in the first three months.

The results can be seen below:

As you can see, the supported PostgreSQL option was significantly more affordable as far as initial cost.

Annual Cost

The Annual cost incorporated the following two factors:

License Support costs – These costs are the support and maintenance costs for both the Production and Development environments.

Estimated Annual Maintenance Costs – It is estimated that there will be additional ongoing effort to maintain the different DBMS solutions as compared to the current Sybase solution. This is due to the different complexities of the DBMS solution, the available tools for the DBMS solution, and the available support in the DBMS user communities.

	Estimated Annual Maintenance Rating	Rationale
Sybase ASE	0%	No additional maintenance effort is required due to the fact that Sybase is currently used
Microsoft SQL Server 2012 Enterprise	5%	No additional maintenance effort is required due to the fact that Microsoft has an excellent set of tools and resources and the environment is similar to Sybase which is currently used Baseline 5% additional cost has been added to account for the fact that a second DBMS must now be learned and supported.
Oracle 11	20%	Additional maintenance effort of 15% is estimated due to the complexity and toolset for Oracle. Baseline 5% additional cost has been added to account for the fact that a second DBMS must now be learned and supported.
PostgreSQL – supported	10%	Additional maintenance effort of 5% is estimated due to the simplicity of the PostgreSQL solution and the relatively limited toolset and resources. Baseline 5% additional cost has been added to account for the fact that a second DBMS must now be learned and supported.

The results can be seen below:

As you can see, the annual ongoing costs end up being in favour of SQL Server.

The 10 year Total Cost of Ownership (TCO) is:

DBMS	Cost
Sybase ASE	$1,173,057.20
Microsoft 2012 Enterprise	$1,019,909.00
Oracle	$4,040,910.00
PostgreSQL – supported	$1,074,025.00

Although SQL Server has an initial license cost, the ongoing support costs are less than supported PostgreSQL in almost all categories.

Conclusion

Over a ten-year period, the TCO for SQL Server is less than a supported version of PostgreSQL. This also includes the server licensing costs. As an added benefit, our solution also required a Reporting Solution and an Extract, Transform, and Load solution – SQL Server provides both of these solution bundled with the DBMS for no additional cost.

From a cost perspective, SQL Server is the clear winner.

For our situation, although cost was important, it was not the overriding factor. In fact, cost accounted for only 10% of the entire weight. The factors and their weights are listed below:

Evaluation Criteria	Weight
MBC Required Functionality	350
Cost	100
Future Commercial Viability	100
High Availability	50
Scalability	50
Product Track record/Future stability	50
Technical Functionality	350
Architectural Standards and Connectivity	50
DBMS standard functionality	100
Data Warehouse standard functionality	150
Security/Encryption	50
DBMS Management Functionality	50
Ease of Maintenance/Management	30
Database Configuration	20
People and Future Flexibility	50
Future Migration/Flexibility	20
Standard Technology/People Availability	30
Performance Functionality	200
Bulk Load Performance	20
Bulk Update Performance	20
Bulk Delete Performance	10
Basic Query Performance	20
Cross Product Query Performance	20
Sub Query Performance	20
Parallel Query Performance	20
Analytical Query Performance	20
Indexed Update Performance	20
Indexed Delete Performance	20
Non-Indexed Update Performance	10
Total	1000

After all the factors were evaluated, SQL Server finished 196 points ahead of PostgreSQL.

SQL Server was selected and I was enlightened. Although people assume Open Source DBMS’s are the most inexpensive options, SQL Server is the leader once you include other factors. In fact, SQL Server was the leader in both cost and functionality.

SQL Server Integration Services (SQL Server’s ETL solution) and SQL Server Reporting Services (SQL Server’s Reporting solution) were NOT included in this comparison. If they were, the results would have been even more pronounced.

I encourage you to do your own evaluations. You may be surprised at the results. I know I was.

Please let me know if anyone is interested in specific evaluation criteria. I’d be happy to share our rationale and SQL performance scripts.

When #agile isn’t the best thing for the client

I am presenting with two other friends on Agile in a non-Agile environment. Some of our first discussions on the matter was commenting on how the topic itself was confrontational and somewhat elitist. Almost putting people would could do more pure Agile above those that had to compromise on how they implemented Agile. In many situations, I had seen the discussion of these projects being referred to ‘bastardizations’ of Agile. How dare we do less than what was written in the books? How dare we do less than what was proposed by the experts?

Our first thought was to again reinforce that Agile is about finding better ways, no matter where you are on the Agile/Waterfall continuum. Our second that was to reinforce that using Agile in a non-Agile environment should not be viewed as a lesser implementation of Agile. As long as the client is getting the most value from the application of Agile, it should be viewed as successful as a more pure Agile project.

What is Agile at its heart?

When we discussed what Agile is at its heart, we came up with two things:

1. Reducing Inventory (Whether they be documents, features, or wasted processes)
2. Shorten Feedback Loops (On deliverables and on the use of features in Production)

Who decides Value?

Often times I believe we as Agile professionals get caught up in determining what is valuable for our clients. This was a major fault with the Waterfall processes, and I fear Agile in falling into the same trap. In the past, Waterfall projects had less frequent interactions with clients and the projects and professionals were expected to make decisions for the business. One of the first benefits of Agile that I saw was placing the full determination of priority and value clearly back into the hands of the client. For too long, projects had wrestled the full determination or priority and value away from the clients and the processes Software Development projects used were considered mandatory and not open for debate.

But now I fear that we are slipping back into that black and white Worldview. But instead of the Software Development professionals informing the client that we know what is best for them, Agile Software Development professionals are informing other Software Development professionals that we know what is best for them. And if some projects and professionals are not as Agile, they clearly have bastardized Agile.

Three statements I heard in the last week were:

• Business Analysts have no value
• Estimates have no value
• Documentation has no value

From whose perspective? I would be hard pressed to find any client I have worked with in the past twenty years that would agree with these statements. I agree that doing documentation and estimates excessively can provide limited value, but in the end the person that determines that value is the client.

The statement that we came up with for the presentation speaks for itself:
“More Agile processes can deliver less value for some clients.”

 We present at the Agile Winnipeg Users Group on May 10th. Hope to see you there and hear your questions and thoughts.

Top 10 practices for estimating Agile projects successfully

I’ve had multiple posts in the past that provided my passionate opinion on why I believe we should estimate Agile projects. (Both from the perspective of the client and the team) But rather than get into that discussion once again, I thought it would be more valuable if I shared the 10 practices that we have used over the past 6 years that have allowed our teams to successfully estimate Agile projects.

Context

I work for a company that bids competitively on projects against other companies. In almost all situations we have to create estimates to provide to the client or else we would not be successful in winning the business. So estimating really is a fact of life for the type of business we are in. Although it might be nice to proceed on a project without an upfront estimate, that really isn’t a luxury we are provided. I believe we are in a similar boat to the majority of people out there.

The Practices

Over the past 20 years I have honed my estimating skills with the experiences on projects. Over the past 6 years, this estimating has been predominantly for Agile projects. Although we have not been perfect in my estimating, I am quite proud of our track record and feel that we have been correct far more than we have been incorrect. Where we have been incorrect, we have augmented the estimating process so we won’t be incorrect in that way again.

The practices are not in order of priority as I had a real problem trying to do that as I think they are all important. So without further ado:

1) Remember to estimate Project Management and Technical Management

We frequently ran across missing these factors on early estimating attempts. This is usually done by everyone as they work on estimating as the work is easy to forget. It is a critical part of the estimate to ensure that these estimates are included so that we don’t scrimp on these tasks and cause larger issues for the project.

2) Don’t plan on your Technical or Application Architects being any more than 20% on construction tasks

This one continues to plague projects I estimate. No matter what the work is, the Architects will always want to work on their fair share of construction tasks. It is a noble desire and will return great benefit if they can work more on construction tasks. But I wouldn’t bet on it. In fact, I would even consider reducing this to zero depending on team size and complexity of the solution. If the Architects can work on more construction tasks, just consider that a bonus.

3) Remember to estimate for meetings and other factors in your schedule estimate

OK, now you have the estimates so lets just plan the hours out on the calendar right? Not quite. Remember to reserve some time every day for meetings, stand ups, and other non-construction activities. In addition, remember to plan for at least some time for vacations, holidays, and sick time. Use your statistics from your company to generate a rough factor for the project. It will just reserve a percentage of your schedule at the start, but then you will have the wiggle room when people start dropping like flies during flu season.

4) Hold a risk workshop and include the weighted estimates in the overall estimate

Mention Risk Workshops and most people yawn before you have completed the sentence. But they have great value. They are a great team building exercise at the start of the project as well. But ultimately, the output from the Risk Workshop should be used to generate an estimate and apply that estimate to the overall project estimate. By calculating the impact and probability of Risks occurring you will generate an estimate of the Risks that will likely occur. My experience is that if you don’t include these estimate, you will frequently address these risks occurring with Change Requests. The project can be much smoother if the estimate are included up front and you don’t surprise the clients. Yes, we won’t know all the Risks at the start, but it is better than assuming there are no risks at all. As Dr. Phil says “How is not estimating any Risks at all working for you?”

5) Estimate at multiple levels and triangulate

Probably the most important practice. We never just create one estimate. We typically estimate bottom-up (object level), deliverable (mid level), and top-down (schedule level). We use metrics for the bottom up and mid level estimates. (how long does a simple CRUD screen take, moderate report, etc..) Then we compare all three estimates and rationalize the differences. It is amazing how this rationalization uncovers estimating oversights.

We also then use Planning Poker sessions as we execute the project to confirm requirements and plan our iterations.

6) Proactively get information required to provide an estimate

Many times there are factors that may cause the estimate to change greatly. (Data Conversions, Legacy interfaces, unknown technology) Ask these questions early on and factor them into your estimates.

7) Say No when you don’t have information to provide an estimate. But be proactive and say what you need to provide an estimate.

If you don’t have the required information to properly estimate, then don’t. But don’t just say you can’t estimate. Provide the details of what information you need to be able to provide an estimate. I know on past projects we have guessed many times on estimates where in hindsight, the client would have gladly provided all the information we required.

8) Track estimates and actuals and use the metrics for future estimating. (both within the project and for future projects)

Yep. We need to track estimates. Sorry. These metrics will then be used to determine on average how long certain objects will take and the percentage that certain areas of the project will require. (Like Project Management!)

We also hold retrospectives and review iteration estimates and actuals so that we can revise our estimates as soon as possible within the project and taken them into consideration for future iterations.

9) Factor educated contingency into your estimates.

We have 17 contingency factors (and counting) that are weighted and applied. (at the team’s discretion) If nothing else, it is a least a checkpoint for the team to think about whether certain situations exist that may affect estimating. (new team? data conversion? new technology? complex interfaces?) Don’t just apply a standard 10% blindly to estimates. It has to be based on reality.

Like Risk, these contingency factors should then we applied to the estimate. I would also recommend that you don’t show what part of the estimate is Risk versus Contingency versus base estimate. Clients will then say that the contingencies won’t happen so just subtract that estimate. <sigh> The estimate is holistic.

10) Execute in a manner that respects that estimates are not actuals. They will be incorrect.

And after all the estimates are done, don’t execute in a manner that makes Captain Bligh look like a level fellow. Don’t micro-manage, realize that these were only estimates – some will be higher and some will be lower. Very rarely will any be totally correct. Let the clients trade requirements and stories as priorities change! You could not possibly know all the scope up front. But also don’t just agree to add new scope. A lot of issues about estimating do come down to good old-fashioned scope control. The estimate should be a guide and a placeholder for a discussion, not a roadmap.

Summary

My experience is that is we do all of these practices, our estimates will still be wrong. But much less wrong. More importantly, these practices help us to discover better ways of estimating. Discovering better ways doesn’t only apply to development practices, but all project practices. Ultimately these estimates also help the team members to feel better about their work as well. No one like to miss estimates and managing projects in a manner that respects that the estimates are not actuals will alleviate many of the negative connotations with estimates.

<SoapboxMode>

The correct thing is to fix how the estimates are managed, not to stop doing estimates.

</SoapboxMode>

P.S. I forgot one! Here is a bonus practice 11) Estimate that you will be wrong – remember to estimate for refactoring, defect fixing, and some rework. We are all good, but not that good.

 

My #1 Agile Failure

Some of you may have read my personal Agile Adoption Story and possibly thought that it was a good story on Agile Adoption and how Agile can sometime succeed on the first try. I know I’ve read many stories about the pain and suffering on the first Agile projects people have been on.

If you missed the Blog Post on my personal Agile Adoption story, you can find it here:

My Agile Adoption Story

So given that my first Agile project was successful? What was my #1 Agile Failure? Simply put, it was the fact that my first Agile Project was successful!

Explanation

Please let me explain. We came out of that first project with a false sense of security. We thought we nailed this ‘Agile’ thing. I mean we didn’t do many of the core Agile practices and still we delivered on time, on budget, and satisfied the client. This false sense of security was at a personal, project, and perhaps even a company level. We thought that we could deliver again with different teams, different problems, and different technologies.

I often say that I would have rather failed slightly on the first project so we knew we still had a way to go. Unfortunately that didn’t happen.

Why did we succeed then?

I believed we succeeded primarily due to the expertise of the development team and the expertise of our client/Product Owner. Having these two factors can address shortcomings in many other areas.

To recap, here are the Agile Practices we did not follow:

• Test Automation
• Continuous Integration
• Kan Ban
• User Stories
• Planning Poker
• Pair Programming

I would not even dream of being on another project without at least 5 and possibly all 6 of these practices. We really had a project that we led in an Agile way but that did not use XP practices, Agile Requirement practices, or any Test Automation! In addition to these 6 practices, I have learned about many other Agile Practices that I also would not be without on a future project. Some of these are:

• User Story Mapping
• Paper Prototyping
• User Design Studio
• BDD
• TDD
• And many others

But it took me a couple of hard projects after this first successful project to teach me that the hard way. I’ll leave it up to a future post to tell the story of a couple of the hard projects.

My Agile Adoption Story

My first experience with an Agile project that was more Agile than Waterfall was in 2006. I phrase it in that way as all projects I have been on have had some aspects of Agile. But I digress, back to the story…

In December of 2005 a fixed price RFP was released for a new Parks Reservation Service that the Government of Manitoba required completion for the spring of 2006. (April 3rd to be exact) We were very diligent in planning to even respond to the RFP as we knew what a challenge the requested time frames would provide. So we met with the proposed team and held a risk session/planning session/estimating session to determine if we felt this was do-able and if we all were comfortable with bidding. It turned out that we thought this was certainly do-able, but it would be a challenge right from the start. We took a vote in that session and it was unanimous that we should bid. We thought that we needed to execute the project in a way that was more Agile than we ever had done before to be successful. This meant to us at the time:

• Team Chemistry and continuity
• At least 1 client on site full time with full decision making authority
• Daily stand-ups
• Fully empowered team (This is normally the way we operate)
• Very lean Use Case requirement documents.
• Demos at least every week to show progress
• Team Planning and Estimating
• Visual Project Management

What were we missing?

You may have noticed several current key Agile practices that were not in our list. Such as:

• Test Automation
• Continuous Integration
• Kan Ban
• User Stories
• Planning Poker
• Pair Programming

But before we get to that. Let’s provide some context for the project.

The Problem

• Dimensions of Parks and Campgrounds
• 57 Campgrounds
• About 5,500 campsites
• 11 Campgrounds currently computerized
• Requirement was for a Web Application that need to be used by three types of clients:
• General Public to make reservations over the internet
• Call Centre employees to make reservations on behalf of campers.
• Campground attendants to manage inventory, make reservations on behalf of campers, and check-in/check-out campers
• Web Application needed to have the following main components:
• Welcome/Login
• Shopping Cart
• Campground Management
• Inventory Management
• Reservation Search
• My Account
• Reports
• Reservation Maintenance
• Reports
• Policy Configuration
• Integration with online payment provider

Kick off to Deployment in 92 days

To meet the required deadline, we made the following project execution decisions:

• We partnered with three other companies to provide expertise so we could hit the ground running. (We could have learned this expertise but we did not have the luxury of time to do so)
• We did not have experience in test automation and felt it would be too risky to learn on this project. To mitigate this we allocated an Application Architect to manually perform Quality Assurance activities daily.
• We allocated an additional Technical Architect to Performance Test the application for the first three months.
• We leveraged Protegra Frameworks to speed up development velocity.
• We had a Technical Spike to validate and confirm what GIS mapping solution we would use.
• We separated the project into three phases to be able to deliver the functionality when it was required:
• Phase 1 – Public and Call Centre Focus
• Functionality focused primarily on the reservation process and reservation maintenance (i.e. making reservations, changes, cancellation, payment, refund, etc.)
• Phase 2 – Campground focused
• Focused on management of campgrounds (i.e. check campers in, check campers out, move campers, camping permits, revenue reconciliation, etc.)
• Phase 3 – Admin site enhancements
• Focused primarily on providing more back office support to Parks head office users

The Solution

The Solution was a C#, .NET Solution. The database was SQL Server and the architecture was a standard three-tier architecture.

Project Details

• Project team included over 20 developers at maximum
• First Phase was delivered in 3 months.
• Included investigation and integration of GIS mapping technology
• Phase 1 alone was over 7,400 hours of development work, all three Phases were over 12,000 hours.
• Quality of the application has exceeded industry standards with only a few defects since go live

The Outcome

The solution was deployed to production on time on April 10th, 2006. For those of you keeping track, this was a week later than initially planned due the time required to train the users. The solution was ready to deploy on April 3rd. This was after the project learned a month before go-live that the code had to be frozen two weeks prior to go live to allow for the acquisition of PCI certification to take on-line payments.

The project was awarded the PMI project of the year (Manitoba Chapter) for 2006.

What We Learned

• Client decision maker on site is key to a Lean project.
• Liaise with the clients as much as possible
• Empowering the development team was crucial.
• Traceability of requirements is key with formal organizations
• The team succeeded because they trusted one another

What we would do differently

• Quality could be further improved with the use of TDD, automated test and build tools
• Form in sub-teams of 6-8 to maximize efficiency. Assign a Client decision maker to each one of these sub teams.
• Leverage Visual Project Management to an even greater extent
• Develop in Iterations, not increments. We really developed fully functioning increments.
• Use the following Agile practices:

• Test Automation – We mitigated this risk by assigning one of our top Application Architects, but I would not recommend this approach unless you have no alternative.
• Continuous Integration – We probably got away with this because we coded in increments. If you are doing iterative development, I believe you need Continuous Integration.
• Kan Ban – We did not use a traditional Kan Ban board. We now use these on all projects.
• User Stories – Use Cases were still too large and they were really why we developed in increments. I don’t think you can deliver in Iterations without User Stories.
• Planning Poker – We had a very senior team who felt very comfortable in a joint planning session and challenging each other’s estimates. I wouldn’t recommend this going forward though as this team was very unique.
• Pair Programming – Strangely enough, this is one Agile practice that we rarely do. We typically use it more for a mentoring opportunity. The book is still out on this one though as I would like to try it on a project.

Why we succeeded

In some ways we succeeded primarily because of the excellent people and team we had. I believe that is still a key factor in the success of projects. That said, Agile practices do result in better solutions and everyone on the team realizes that we could have had an even better solution and project with them. (And certainly a more stable pace)

I believe we also succeeded because we had the client decision maker on site. This was not just any decision maker though. The Client was a person that had lived and breathed working with the old system, creating the RFP, and knowing what she wanted from a new system. In short, our client on site was also our Product Owner. Very powerful…

Check it out

If you want to check out the solution, you can find it at the following link:

http://prsasr.gov.mb.ca/