Reading through the comments, several words are flashing across my mind. Greed – just because you can gather data, doesn’t mean you have to. Focus – know what you want to know and then gather the data you need to know it. Don’t try to answer every question that pops into someone’s head. Different departments have different needs, questions and answers. Trying to compile all the data to suit all of them creates those infinitely large forests. It may sound cool to be able to access any little detail that grabs you but databases cannot and should not be all things to all departments. Way back when I was publishing trade directories we had multiple reasons for wanting information on sales, some for the editorial department, some for the advertising department and some for the circulation department. Our weekly reports were inches thick. Reams of paper wasted. Chop down a few of those trees and re-distribute the data. Don’t try to extract the answers to basic questions from complex data.

As an Account Executive at Ketchum Communications in S.F., one of the most important lessons I learned was the value of ACTIONABLE research. We never, ever presented research results of any kind to clients without an accompanying action plan. It was a rule, you could look it up. There are two kinds of cube farms: One is the office cubicle setup in many businesses, and the other is a collection of data parameters that are defined by three descriptors – height, width, and length, i.e., a cube. In most cases a data cube involves the three parameters of time, demo/psychographics, and questions asked. By doing one’s homework and precisely defining cube parameters — while avoiding the word “all” — good researchers are miles ahead in providing actionable research. It’s a discipline. This is where good researchers spend several nights working on their rationale. It’s the definition of the question — and the cube — that is most important. One of the things I most admired about Catalina Marketing during my five years there, and about… Read more »

Bill makes two great points on how to begin to solve the core issue of how we make cost-effective use of the detailed information from the business to improve business results.

Here are a couple of additional ideas that build on those involving benchmarking and the development of a set of decision processes.

>Recognize that there’s variation in all data sources, which means that it’s necessary to be able to tell the difference between what’s normal variation and a real change in the data. The use of quality control statistics and the statistics that underlie 6 Sigma programs could be of real help to a retailer here.

>Create a way to systematically develop and test hypotheses which focus laser-like attention on specific information and can provide clear answers to questions. This hypothesis process could easily be embedded into the decision processes, that Bill is calling for.

I think it was Einstein who said “Information is not knowledge.” Nowhere is this more true than in a data warehouse. Like with any research project it is important to decide what types of decisions will be made based on the data. Then formulate the specific questions needed to retrieve the relevant information, and only then go to the warehouse looking for answers.

Also, habit is a powerful tool. The best data warehouse reporting is often created so that it is easy to read and regularly delivered. Just like inventory levels, margin and other metrics tracked by most businesses it is the trend overtime information that is often the most meaningful. Trends can be your best source for benchmarks.

Disclosure: RAM Communications is on retainer with Teradata, one of the leading providers of data warehousing solutions. There can never be too much information, just like someone can’t be too beautiful (although Nicole Kidman comes close). Wal-Mart’s data warehouse is now more than half a petabyte, equivalent to the number of characters in one billion books. Still, they don’t have enough data to ensure they’re in stock on all items all the time. The key to using a data warehouse effectively is to deploy analytics that make the most of the information being captured. This requires a project team composed of both business users and IT folks, so the reports and alerts created by the system drive optimal processes throughout the organization. One of the best examples of how a data warehouse can provide insightful and actionable information is the beer and diapers story. The legend is that a number-cruncher examining retail checkout data discovered a strange correlation: a higher-than-expected pairing of beer and diapers in afternoon transactions. Presumably, the data indicated that young fathers… Read more »

You can never have too much information. But you have to know what to ask and to have someone to make sure you are getting usable information from the raw data.

You can have the most efficient data warehouse — hate the word warehousing — but it’s not going to do any good if everyone in your organization just lets the reports pile up on their desks because they don’t know what to do with them.

It’s been compared to getting a drink out of a fire hose. From the perspective of mining the data in our own PathTracker® data warehouse, there is a two step process that is helpful. First, find all of the patterns you can and create a catalog of these patterns, with as detailed understanding of WHY each pattern occurs as you can. This is the basis for a set of “rules” that give you a rich understanding of the data. Follow this with an examination of all the stuff that doesn’t fit the rules. These exceptions provide an even richer understanding of what is going on.

The real value comes when you can then predict, if this happens, then that is going to happen. Now you are in the driver’s seat, in a position to say, “let’s do this, so that we can get that to happen.” If “that” is more sales, you’re now in a position to mint money. 🙂

Best practices in data warehouse structure are very complex, and not appropriate for this space. If anyone out there is looking at the issue: invite a consultant. It WILL pay for itself.

I agree with the moderator’s comments in general. Absolutely, the business must address the information it will require to make decisions and implement those decisions. Remember to address this issue across multiple time frames. Information requirements for immediate day-to-day tasks are quite different from weekly or monthly business analysis tasks, which are quite different from seasonal planning tasks.

The front end of a data warehouse is critically important, but it can only provide access to information based on the way it is captured and stored.

Most consulting firms have very good methodology for guiding clients throughout this process. It is NOT intuitive. Very few things provide the ROI they claim. Setting up your data warehouse correctly, anticipating the business needs and requirements…this is one of them.

Answering “what if” questions can be extremely important in retailing as the playing field is constantly being tilted by an ever-revising environment and circumstances. So endless data is collected and stored. But too frequently that data can serve as a security blanket for retailers and a profitable business for the collection-providers. That thought is contestable, of course, but think about it.

It is always thus, impelled by the state of mind which is destined to not to last, that we make our irrevocable decisions of input — at least until tomorrow. So, retailers, watch carefully your cost of both information preservation and your on-going operations.

I would agree with Ron, that there is no such thing as too much information. The data warehouse should be exactly that, a warehouse. It’s up to a human being (with the appropriate tools) to extract the information relevant to his or her burning issue. In the past, a lot of emphasis has been placed on creating data hierarchies to expedite this process. However, as the amount and variety of data in the warehouse grows, this can be cumbersome. A more feasible alternative is attribute based analysis. Using OLAP technology and flexible front-end tools such as XP3 from Interactive Edge, it is possible to empower business users to define attributes of products, geographies etc. This flexibility affords business users the ability to ask questions of the warehouse and get answers in terms that they understand. For example, instead of creating a dashboard that highlights the ‘top X stores’ in a predefined hierarchy (e.g. location, sales team, etc.) business users can segment stores by any number and combination of attributes and ask questions such as “what… Read more »