Business Intelligence Project

Phew, it's being a month since I last updated this tiny blog. Sorry mate, I was stucked in a jungle of confusions and disturbances, now still.

We've heard about so many business intelligence projects failed despite relatively little success stories shared by major players in the market. One common fallacy about successful BI project is not about how flexible or friendly is the tool but the overall process that leads to the eventual outcome. In data mining projects, it's a known fact that the process framework is more critical or makes more business sense than just sneaking at mere churned numbers. We have SEMMA and CRISP-DM process methodology just to name a few.

To be fair, project failures in this context do not mean absolute technical failures but refer to the inability to meet constraints, key performance indicators or explicitly stated objectives. A project can considered failed due to expectation mismatch of final deliverables, user resistance to adapt or high total cost of ownership.

Anyway, BI effort is a Product, not a Project.

The formal definitions of project and product are well defined in IT management literature. Briefly, a project has well defined deliverables with final delivery dates which is usually once off. In a project, vendor delivers something then possibly station a few support staffs onsite and done. From the perspective of vendor, yes, a BI "project" can be treated as a project. However rightfully any business intelligence application owners must see it as a product. Very important, the mindset has to be right.

I'm not really a hard core proponent of big bang design phase guy. Planning and design is a must, yes I agreed but not to the extent of every nitty gritty details. My view is that development methodology that involves a big chunk of time allocated for plain design works is not practical and inefficient. Iterative and incremental is still the way to go. The longer that you plan ahead the more deviate the outcome will be compared to your original plan.

Changes need to be embraced.

If BI effort must be considered as a/part of a product, then ideally the vendor commitment to such product shall be time driven, not work driven. And why is it you might ask? Think about it, if the commitment is scoped by works, one obvious characteristic of such arrangement will be vendor resistance to work changes. Any potential works that can result in significant business value to the organization simply dropped or "overlooked" due to the stringent time frame or slight technical difficulty to meet the deadline.

Secondly, the fact is real business requirement changes faster than the frozen paper works. The whole problem with something called scope creep is again merely due to the idea of scoped by works. Scope creeps must be embraced. Customers only realize certain direction of BI usages or more efficient/effective ways of utilizing BI after they know more about it. That's why customers with prior failure in BI efforts are more demanding because they know at least more compared to previous on how to make better cases for BI. This trial and error way is costly. I propose a non-trivial business intelligence to be bounded primarily by time, making the scope of works secondary though still significant.

A scenario. Organization ABC signed up a 2 years service contract with a BI solution provider called XYZ. The arrangement shall be like this, given a set of business requirement that can be materialized in the solution within a month, the vendor starts a common lifecycle to manage the deliverables. Plan, analysis, design, development, test and deployment. A month later, the targeted users can start to make use of the new analytical capability and derive more requirements from it then combine with the latest requirements drive by other factors such as market trends, regulatory compliances and new technology enablements. Here it forms the second payload to start the next iteration of efforts. We must remember that all these efforts are part of a BI product meaning the core of the product (deliverables from the month 1) can be changed, removed and enhanced to accomodate the version 2 requirements. Technically, this means that original tables can be dropped, combined or enriched with more attributes or data behaviors. All ETL jobs are vulnerable to changes. At this point, you might argue that why can't I design nicely and properly beforehand even before the lifecycle of month 1 started? You can't or properly speaking it might not make full business sense. A product must be enhanced only when real needs arise, critical bugs/issues found, significant feedbacks and new corporate directions.

Quantitatively, let assume deliverables from month 1 effort are 10 jobs, 5 reports, 20 tables and 3 cubes. For month 2, assume 5 new jobs added/3 old jobs must be refactored, 2 cubes added/1 cubes refactored, 4 reports added/1 reports refactored and 10 tables added/10 tables refactored. The numbers suggested here are superficial but you might see the pattern as time passes, more the development iterations that the objects (cubes/reports/tables/jobs) went through, the more stable it will be and the business value of each can be justified and maximized. 2 years duration thus might turn into a 24 iteration of BI "project" cycle.

One major advantage of this approach is that the analytical capability of the corporation along with its BI users is truly increasing and now they are more trained to adapt business changes and more flexible to face competitions.

To emphasize again, the experience process is more important than the outcome.