Back in November, I expressed some skepticism about a claim that software giant SAP made in an email newsletter it sent out. The newsletter reported that “a recent study of companies listed on NASDAQ and NYSE found that companies that run SAP are 32% more profitable than those that don’t.” The email included a link that promised “We invite you to see for yourself,” but instead of bringing you to details about the study, the link just brought you to a SAP web page with more marketing pitches. I did a considerable bit of searching but, as I reported, I was unable to find any documentation of the study – either on SAP’s site or on the site of the company that performed the study, Stratascope. But I did find that SAP is a big client of Stratascope’s. I wrote: “Now, the fact that SAP and Stratascope are cozy – and that it’s in Stratascope’s interest to make its client happy – doesn’t mean that Stratascope’s research is necessarily unsound. But it does raise questions – questions that can only be answered through a careful review of the research methodology and results.”
Although I didn’t realize it at the time, the SAP newsletter was just the start of a huge advertising campaign built around the “32% percent more profitable” study. If you’ve opened a business magazine recently, or watched any cable TV, you’ve likely seen the ads. To the best of my knowledge, though, SAP still hasn’t released the details of the research.
Today, Stratascope’s CEO, Bruce Brien, posted a comment to my November post, which I reprint here in full:
It is interesting that Mr. Carr, in his detailed research for this article, never contacted us at Stratascope Inc. The study was commissioned by SAP so it will not be published on our website. SAP owns the study. I would be more than happy to explain our methodology and calculations, as well as our auditing and quality assurance procedures that we use whenever we undertake such a study for any of our clients. All Mr. Carr had to do was ask.
I can also tell you that SAP commissioned the study without any idea what the results would show. The Ad campaign was created as a result of the study and not the other way around.
We also need to understand a little about statistics. Hypothetically, if the average non-SAP client in the study was showing a profit margin of 6%, a 32% higher margin would be only 7.92%. The numbers are not so hard to imagine in this light.
Finally, it is also clear that this is a success by association Ad campaign. There are no claims in any aspect of the campaign that imply that the client’s success is because of SAP.
Although bespoke research always makes me nervous, I’m willing to accept Brien’s assurance of the study’s integrity. Nonetheless, I continue to believe that SAP has an obligation to release the full report so that we can judge the basis and reliability of its claim that the use of SAP is associated with significantly superior profitability. In fact, Brien’s comment leads me to believe that such a disclosure is all the more important because SAP appears to be mischaracterizing the Stratascope study. Brien underscores that the research does not show that the use of SAP causes superior profitability; it’s just associated with superior profitability. Brien goes on to write that “There are no claims in any aspect of the [ad] campaign that imply that the client’s success is because of SAP.” Oh yeah? Here is a transcript of the voiceover from a SAP television ad:
The right software can make any size company more efficient, more agile, more responsive. In short, make your company more. That’s why companies that run SAP are 32% more profitable than companies that don’t.
After receiving Brien’s comment, I shot him an email, asking: “Doesn’t the ‘that’s why’ [in the SAP ad] suggest causality rather than just association?” He replied:
You are correct. I guess I didn’t pay enough attention to the voiceover. We try to make it very clear to our clients that software does not make companies perform better, that software cannot improve your bottom line and that since most software is not free it will cost you money.
If the people running the company make better decisions because of the software, then they can impact the performance of their business. This campaign was played back to us as an association type campaign where smart people who run great companies are choosing SAP and you should too.
Brien’s a straightshooter. I wish I could say the same for SAP’s marketers.
Nick – if we aere to assess from one more perspective, it is surprising that in claims/projections from product companies in general, the value derived through consulting is seldom highlighted. At one level, it may be difficult but not impossible to correlate in exact terms the relationship between IT investments and business success. From a consulting perspective, success/failure would be measured in terms of both output and input value. Output value is measured by how much more business value enterprises can get for intended expenditure. Input value comes from how much enterprises can do the same with IT for less. Consulting /services bridges the gap between vision & reality. Product companies provide a platform with a plethora of choices and combinations. In the recent past the message from most of them used to be – The best of the best practices are embedded inside as repositories. Uncork the genie – road to prosperity is crystal clear. Conceptualizing innovative models and rolling out cost competitive solutions are amongst other things the key elements of success besides organizational readiness & support. It is indeed the truth that top line standard products may not hinder trying out different/multiple models. From an age of fitting oneself to a readyware size, we have transited to an era of virtually unlimited options and means to try almost infinite levels of innovation. That means all the more reason to factor in the role of services bridging the gap – we see that enterprises of late spend lot more time in assessing service provider/consulting firm fitments related to product assessments. It is like discussing about the yin without the yang in place. On top of it any value derived is contextual. Emphasising richness,adaptability, versatility are one thing, but success doesn’t in general flow just along with a dimension and so claims that overlook these – would anyway look incredible. Read this as a post aimed in general about the practices followed in the software industry.
Why should SAP release the study? And now that you are demanding it, it will not release it al all…
Nick:
Glad to see your coverage of Mr. Brien’s openness and also glad he’s honoring his contract with his client. That does not let SAP off the hook, though. This scenario recalls the use of a study purportedly done by Coopers & Lybrand on document imaging. Widely quoted, but I can’t find a traceable citation that would allow the numbers to be verified. What was most likely a fine piece of research is now just a bunch of bullet points floating around on the web. Too bad too. They could help make the case for imaging in K12 schools, but will never get into my reports without a solid citation. In 3-5 years some other researcher will want to see Stratascope’s study and all they’ll be able to dig up is the 32% line.
I quite agree with you ,Nick,
there are little studies indicating the impact of IT adoption in the organizations ,and if SAP has found a way ,this could be could be a hit , as imagine , we could use the SAP model to measure the goodies that such investment will return to companies, and not only for SAP ,but for any IT investment . so please release the methodology