The Data Warehouse Project Discovery Phase

Last time I introduced the trademarked “Seven Ds of the Database Lifecycle,” which is the methodology that I’ve developed and which I use when I’m engaged in database projects. Throughout 2007 and 2008 I’ve coached you on implementation techniques for the data warehouse. This means we’ve covered the basics of Step 3, Develop. Now it’s time to go back to the beginning, to Step 1, Discover, and look at some techniques that you can use during the discovery phase.

A data warehouse is a strategic tool. You can use it to give your organization an edge against the competition. The marketing managers can use it to better understand the buying habits of their customers and propel a business to the forefront of its business sector. The warehouse can also expose shortcomings within an organization that need attention. For example, in "BI Without the Data Warehouse”, September 2008, I explained how important it is to analyze raw transactional data in order to reveal the gaps between perception and reality. As the keeper of this corporate asset, you need to plan carefully and you need to start from the beginning.

Keys to Success

Think about your data warehouse project as if you were a publisher of a magazine. Determine the success metrics, figure out what’s going to please the reader (or user, in this case), and consistently deliver good and interesting content. While you might not be able to deliver something new every month, you should be able to roll out some new feature or report or enhanced functionality every 4-6 months, on a regular basis.

Success Metrics

Success metrics are based on measurable objects and events, numbers that you can put into a report and present to your business sponsor and the data warehouse steering committee, if you have one. Objective metrics are good because it’s easy to derive numbers from them. Typical objective metrics are:

Requirements Gathering

Gathering requirements is absolutely necessary for a successful data warehouse project. Almost everyone has, at some point in his or her IT career, had to do requirements gathering. Let’s just hit the highlights of gathering requirements for the three project tracks—Technology, Data, and Application Layer—that I introduced in “Seven Steps for Successful Data Warehouse Projects”, April 2009.

Technology Track requirements include:

Data Track requirements include:

To gather requirements successfully you need to communicate effectively and make yourself available to the users. Don’t hide in the computer room or your office, even though you have a thousand tasks to accomplish by day’s end. You need to become a trusted member of your organization’s business community or you’ll never know how your efforts are perceived or where your data warehouse project stands in your organization’s priorities and budgeting. Your data warehouse may be a strategic tool to the company, but it’s also a political target. Get business savvy; stay in the open. And don’t be afraid to remind others of your success; that’s part of effective communications.

User Support for the Data Warehouse

One outcome of gathering requirements is a better understanding of how to please the user community. You want the users to realize the value that the data warehouse brings to their jobs, you want them in your corner, and you want to keep them coming back for more, even suggesting additional features and functionality for the data warehouse.

In addition to enthusiastic users, it’s vitally important to have one or more business sponsors to support you through the data warehouse project. A good business sponsor is an ally, stays engaged in the project, and is respected by all parties involved, both business and technical. While your IT department implements and maintains the data warehouse, ownership of the data and how the data is used, i.e., data stewardship, belongs to the business units. You can’t forget that.

Any sponsor must have a compelling problem for which the data warehouse will provide a solution. The prospective sponsor must have clout and must be an influential leader. And last but not least, the prospective sponsor must have the patience and courage to partner with the data warehouse technology team. You’ll need to work hard to solidify and maintain sponsorship for your data warehouse project. If at all possible, create an advisory board or steering committee for executive governance. Typically these committees are composed of senior business and IT representatives from each sector of the organization that will be affected by the data warehouse. Develop a roadmap with priorities based on what you’ve discovered to be the most critical needs—enhanced reporting, ad hoc analysis, support for daily operations, strategic support. Align the IT tasks with the business needs. Secure buy-in from all members of the committee on your priority list, and get consensus on which priorities are the “right” priorities.

I use a grid to display business value versus the ability to implement in a data warehouse environment (feasibility). For each entry on the list of reports and features to implement in the warehouse I calculate a rough cost, then plot on the graph. For a sample grid, see Figure 1.

The upper-left quadrant is where fabulous features that require Herculean efforts and open-ended budgets will be located. Despite the high business value, stay away from those, especially if you value your job. In the lower right quadrant are the low-value, easy-to-implement features. While not a whole lot of users may care about this set of deliverables, they’re a great training ground because they’re easy to do and will give you a sense of self-confidence as you hone your skills in the data warehouse. In the lower-left quadrant is that group of features that are low-value and not very feasible to implement. Stay away from these, because they’ll bring you no glory and will take away time that you could be spending elsewhere. (See the web-exclusive sidebar, “Some Musts To Avoid When Developing Your Data Warehouse Project,” for more advice on what to avoid.) The upper-right quadrant is the Sweet Spot, features and functionalities that have moderate to high business value and are moderately to very feasible to implement. The tasks that fall into this quadrant will give you your best return on investment.

Data Warehouse Requirements Checklist

It helps to have a checklist, so that you can keep track of the decisions that have been made and the decisions that still need attention. You can use one checklist for the overall data warehouse project and other checklists for each sub-project (such as a data mart or enhanced feature) as they’re undertaken. Your checklists should include a brief description of the project or sub-project, including the name and date, and a list of the stakeholders’ names, titles, and contact methods. In the checklist enumerate at a high level the business goals that a data warehouse must satisfy and the current problems and functions that such a data warehouse is meant to solve. It’s mandatory to establish success criteria, or else you’ll never be able to judge when a segment of the warehouse is complete and whether or not it is successful. You’ll want to record descriptions of user profiles, or roles, so that you’ll understand which groups of users will have access to which segments of the data warehouse. It helps to understand the user tasks, both current and those that might be scheduled for re-engineering, and how these tasks will interact or impact the data warehouse. Lastly, you’ll want to identify all the constraints that might possibly interfere with development of the data warehouse, and develop contingency plans if at all possible. To download a web-exclusive list go to “Data Warehouse Requirements Checklist.”

You Must Remember This

There’s no way to guarantee success in a data warehouse project. Many fail, but many succeed. Remember to think like a publisher and figure out what pleases your users and give it to them. Deliver a consistently high-quality product, and don’t make them wait too long to get it. Be politically smart. Start small and deliver the results quickly, in installments, as a magazine does. Keep your eyes and ears open so that you’ll know for yourself how your efforts are being received. In so doing, you’ll not only become a trusted member of the business community within your organization, you’ll keep the users happy.