The Broken IT Analyst Market And Evaluation Options For Cost-Conscious DAM Buyers

This special feature was written by Ralph Windsor, DAM News Senior Editor and Project Director at DAM consultants, Daydream.

 

Recently, it was widely reported that IT industry analysts, Gartner, were being sued by NetScout, a Network Performance Management products and services provider on the grounds of Gartner’s alleged unfair business practices. NetScout were placed as merely as a ‘challenger’ on Gartner’s Magic Quadrant vendor assessment framework rather than the more prestigious ‘leader’ classification.

NetScout have cited what they perceive to be a ‘pay to play’ bias towards those vendors who are clients of Gartner’s consulting services rather than solely on merit alone. Some of the language used in the court litigation documents is of a fairly inflammatory nature, for example this is from point two of the “Nature of The Action” section:

“Gartner, an information technology (“IT”) research giant, markets itself as an “independent and objective” company offering actionable technology research from an “unbiased source.” In fact, Gartner is not independent, objective or unbiased, and its business model is extortionate by its very nature. Its substantial success is due to the worst kept secret in the IT industry: Gartner has a “pay-to play” business model that by its design rewards Gartner clients who spend substantial sums on its various services by ranking them favorably in its influential Magic Quadrant research reports (“Magic Quadrant reports”) and punishes technology companies that choose not to spend substantial sums on Gartner services.” [Read More]

Whether or not NetScout will be successful in their case remains to be seen. One perspective could be that NetScout are fully aware that their case may not be successful, however, the PR benefit to them generated by filing court papers has already been significant (and so far at limited expense, especially if they decide to not pursue the case).

An alternative view of NetScout’s lawsuit might be that they exploited this as a PR counter-manoeuvre at Gartner’s expense. Rather than being sidelined as a second-tier service provider, they have cast FUD (Fear Uncertainty and Doubt) upon the Gartner research method and allowed themselves to avoid discussion of possibly legitimate shortcomings in their offer. In so doing, they have converted a PR dud into a more lucrative opportunity to present themselves as innovators and revolutionaries who are willing to take on the analyst establishment. Using phrases and descriptions like “extortionate” or “worst kept secret in the IT industry”, Gartner are portrayed as being like a mafia-style protection racket who have to be paid off by IT business owners if they want to get into the major league of whatever tech segment they operate in.

Although NetScout might have benefited in terms of short-term press coverage, a possible unintended consequence is that many purchasing authorities may now elect to defer investments into network performance technologies while they wait for the dust to settle. That may be an outcome NetScout’s management had not planned for, but if so, it will do neither them nor Gartner any good and there may ultimately be no winners (as is often the case with all types of litigation).

Cash For Quadrants
Setting aside any theorising about the possibly cynical motivation behind NetScout’s lawsuit, many of the accusations made by them will certainly resonate across the IT industry and some may take the view that there is no smoke without fire (correctly or otherwise).

One of the articles discussing this case was by UK VAR publication Channel Reseller News. In their write-up, they chose to use the title: Cash for quadrants? Gartner sued on ‘pay-for-play’ claims.   The headline alludes to a scandal about cash for questions in the UK around 20 years ago where a number of MPs were paid by lobbyists to ask questions in the House of Commons on behalf of their clients.

This parallel between IT analyst firms with the world of political lobbying has been recognised by other people I have discussed this subject with. In particular, the unspoken nature of the arrangement, that this is what you are ‘supposed’ to do if you want to get ahead. There is a widespread perception among vendors (especially the less well capitalised ones) that segments of the analyst market are murky and operate more like bookmakers than scientists, i.e. they follow the money rather than making rational decisions about whether one vendor’s offer really is superior to a competitor. This is unhelpful for end users who are caught between the risks of overblown claims by vendors and doubts about the impartiality of the self-appointed referees.

Peer-To-Peer Analyst Alternatives?
I read many articles from tech analysts with subjects discussing innovation, cloud business models, market disruption and a range of cutting edge topics. Yet despite the subject matter, the analyst market itself appears to be based on some very old and arguably out-moded practices which would benefit from some disruption of their own. I get the sense that enterprise purchasing authorities are getting as frustrated about the perceived poor value (of some) as the vendors are.

The NetScout action has probably been on the cards for a long time and even if it had not been them, someone else would have stepped up to assume their role as Gartner’s adversaries instead. It is less the accusation that a given analyst firm is guilty of dubious ethics and more that their methods appear to be at odds with many of the trends and innovations they comment upon which seems like an unsustainable position.

Not everyone involved in the analyst market follows the Gartner model, by which I mean agreeing to carry out consulting work for both vendors and end users simultaneously (or ‘double dipping’ to use the pejorative expression). There are some companies who make a point of deliberately refusing to accept not only money, but also freebies such as business-class air travel to vendor partner summits, gifts, hotel accommodation etc. Yet with nearly all examples that I can think of, there is a significant gap between a straight unbiased review and the financial commitment needed to get hold of some useful and impartial information. This is being noted up by some commentators in DAM. For example, consider the following written by Tim Strehele on his blog:

“It must be terrible to shop for a Digital Asset Management system. While the Web empowers cheap smartphone, fashion or book buyers – with independent coverage from press and bloggers, and customer reviews on Amazon – it’s not very helpful when you’re planning to spend tens (or hundreds) of thousands on DAM software and need to compare products. The DAM market is a highly fragmented assortment (148 items on my list) of complex products, most of which can be complicated even more by customizing. It’s a niche market with few household names and a very long tail. Even mentions of DAM products by technology journalists are rare; in-depth reviews in the tech press don’t seem to exist at all. And there’s no Consumer Reports issue on DAM systems.” [Read More]

DAM software, even at the budget end of the market, is complicated and has lots of features, so the cost of reviewing products is not cheap and requires a decent amount of the time of a suitably qualified expert. While it might be feasible for a mass-market consumer computer magazine to do a write up of an application like Photoshop for no more than the cover price, they also depend heavily on advertising revenue (in other words, they are arguably as guilty of ‘double dipping’, themselves). In addition, the level of specialist review time they can afford to devote towards some more obscure competing applications is debatable if there is insufficient demand for it and most advertisers would not want to be positioned on a page that few people would read.

Despite the apparent renewed popularity of DAM, the market for paid reviews of DAM solutions is still not a large one.  That even more people would like to get hold of information but have little or no budget to pay anything for it (apart from with their own time) is highly plausible too.

With all that being said, the dilemma faced by buyers that Tim describes is a very real and present one. In an era of Cloud delivery methods, it seems increasingly harder to justify a sizeable expenditure on a one-off report or evaluation for a product that you are favouring because of the flexibility of its contract and ability to avoid a monolithic implementation. The original proposition with these documents was a kind of leveraged investment where you would put at risk 2-3% of the overall project budget in return for a significant saving in the form of picking the right product for your needs. However, project delivery methodologies and the solutions used have moved on and implementations are less a kind of one shot, ‘build it and hope it works’ approach and more the selection of multiple interconnecting components, in combination with coordinated training, change management and internal communications programmes.

As discussed elsewhere in the article (and the discussion link Tim cites) one of the more credible methods to get useful feedback about products is from peer-to-peer events like DAM Meetups where it is possible to freely associate with other users and get some real, unbiased opinions. Not every medium is suited to this (as mentioned in the LinkedIn discussion referred to) but there are an increasingly diverse range of options where expressing an unvarnished opinion has less risk attached. Some of the more flexible analysts (and most consultants) will also adapt the range of options to suit budgets if you do want to go in-depth and need someone on-hand to probe for detailed information. The best feedback and reviews, however, come from people who use a given application every day. Real users and their experiences may offer far more accurate measure of what you can expect, particularly in the earlier stages when you need to narrow down the options avoid reading numerous pages of information about unsuitable DAM applications.

DIY DAM Solution Analysis
Rather than leaving this article with not much more to offer than comment on the opaque nature of sections of the IT analyst market, I would like to share some recommendations that cost little or no money. I would advise anyone who needs to go through this process to exhaust all the free and cheaper tactics before progressing to more expensive professional advice, at least that way you will have a better idea of what they are telling you and can decide if it sounds plausible or not.

Vendor Lists
You can get lists of vendors at the DAM vendors site: damvendors.com (the registered version is free and gives you a lot of information). Tim Strehele’s Planet DAM has a list of 148 vendors websites: http://www.strehle.de/planet_dam/ Capterra also list DAM vendors (note they do take fees from vendors in return for a more prominent position) http://www.capterra.com/digital-asset-management-software/

DAM Meetups
As described earlier, these are very good sources of information since they are widely attended by Digital Asset Managers who may use some or all of the products you are interested in.

Use An Existing DAM
If you work for a larger organisation then there almost certainly will be an existing DAM already in use somewhere, even though it might not be widely known about. Ensure you have audited the existence of these and consider the possibility of re-using them, or at least partnering up with the users to extend it to cover your requirements too.

Vendor Webinars
Vendor webinars allow you to see their product in action, albeit in sanitised form where usually the sales demonstrators will take care to avoid creating errors, faults or producing other unexpected behaviour. If you don’t fancy sitting through the whole thing live, many are recorded these and you can skip over any of the filler content like introductions etc.

Do The Demo First
If a candidate product looks interesting based on the canned materials, arrange an interactive, live ‘walk through’ demo. An hour is usually sufficient. I tend to favour letting the vendor lead the first one, as long as you make it clear you want to see the product only, not a bunch of PowerPoint presentations with screengrabs etc Corporate information should on their website and you can probably scan it quicker yourself than they can read it out aloud from the screen. The best vendor demonstrations are delivered more like training classes where they address you as though you are already a customer and need to get on with using their app.

Scripted Scenarios
Think of the key tasks you need to carry out. Ideally not more than about five in the first instance but with a few permutations using different source files or workflows etc. Get the vendor to demonstrate how the task is carried out. Once the scenarios are done, ask for their training materials, manuals, videos etc also as they all give clues about how suitable this is likely to be. Make sure you read these in some detail before you have become a customer.

Ask For A Sandbox Or Free Trial
You should be able to get at least 30 days free use of a system and some vendors offer a lot more than that. Ensure you have set aside enough time to properly test this, don’t play around with it for five minutes over a lunch break and then forget about it until the day before the trial ends.

Focus On The User Interface
The UI (User Interface) is a crucial component of DAMs. The stock advice is to get something that is easy to use. I would not argue with that, but it also needs to be suitable for your entire range of users. Be aware that your heavy users will supply most of the assets downloaded by everyone else. The UI needs to address both heavy and light usage groups (possibly through different modes or interfaces). The ROI productivity benefits of DAM are won and lost on this element alone. See the section below on needing more than one DAM also.

Keep Detailed Notes
The feedback from the above will act as your substitute for an analyst report. It may well lack the detail and cross-market insight that some possess, but it won’t have cost you anything apart from your own time. You need to be able to store this knowledge so others in your organisation can use it (see the point about existing DAMs) and also so you can check why you though product x was preferable to product y at the time you signed up for it. Even if you do decide to hire a consultant or buy an evaluation guide, I would still recommend keeping your own notes and check what they say against your own experiences. Don’t rely on your own potentially fallible memory exclusively for that task.

Prepare To Need More Than One
A lot of firms aim to ‘solve the DAM problem’ with a single application. That can work for smaller organisations who can define a tighter scope, but for larger ones, I have yet to see any example of DAM where at least one other supporting application was not pressed into service to fill out missing features at some point. This might be as simple as a desktop tool like Adobe Lightroom etc that gets used to do bulk metadata cataloguing before the data is imported in the DAM but it could equally be something designed to handle a certain type of media better than the core system (video being a common example).

Many end users seem to try to seek out single applications as though there is some perfect Mr or Mrs Right fit for their entire range of DAM needs (which they likely still have not fully understood). They are usually disappointed and what happens is compromises get made and anyone whose requirements were not properly met gets told to put up with it or fobbed off with promises of ‘phase 2’ upgrades etc. This generates simmering frustration and resentment which they resolve by using their own budget to fix the problem themselves, whether or not their DAM is compatible with the enterprise one or not. The modern DAM market is highly diverse to match the ever expanding range of features that users are asking for. You are unlikely to buck the trend, so expect diversity and develop plans to deal with it rather than pretending that monogamous DAM is going to be a tenable proposition.

What Else Will The DAM Need To Work With?
Integration and interoperability are key topics in DAM and are becoming ever more important. DAM solutions are like airports or train stations in that assets will be arriving and leaving on a continuous basis. You need to verify that your transport hub of choice can be widely connected to places you need to go and isn’t located in some deserted backwater. Note that means more than ‘can be integrated’ as in paying the vendor’s professional services teams a lot of money to custom build what they should have offered out of the box. Have some ideas of what you might need to integrate with, the methods the third party systems prefer and whether a given candidate has the ability to do it or not. This is sometimes impractical, but asking the vendor to demonstrate an integration pre-sale should give you a good idea of what level of effort might be required if you go further with their product.
While the methods described above can be effective, it is true that they require time to do diligently and if senior personnel are involved, the cost to the organisation might be uneconomic as compared with an evaluation carried out by a professional. An alternative option is to do some of the tasks yourself and call in a specialist like a consultant (or analyst if they offer it) to fill in the missing items or anything you don’t feel fully confident about.

When I work with clients in our DAM consulting practice, the ones who get the best results are those who have a clear idea about what they wish to achieve and how they want us to help them. They save a lot of unnecessary expenditure and we have a better understanding of what they need to do (which usually pleases everyone). Even if you do not plan to use any external assistance, organising your evaluations like you are going to be, is a good way to get you to think about this more clearly and also provides some scope for delegating the tasks to internal colleagues so they can help you get through the work.

Conclusion
As is obvious from the NetScout debacle, the IT analyst market has sustained some damage to its reputation which is likely to have impacted end user trust in the services they offer. But, it would be absolutely false and a gross generalisation to suggest that every firm operating in this sector is somehow crooked or implicated in unethical practices. What these incidents should do is offer an opportunity for everyone to re-evaluate their processes and consider more efficient or effective ways to deliver them.

The world in 2014 offers many more possibilities for cost-conscious buyers to take on at least some of the research work that analysts or consultants would once have provided exclusively. Ultimately, it is technology buyers who will have to be willing to live with any decisions they might make. While it is true that using an analyst opinion as a justification for a purchase decision can offer a porous form of employment insurance if something goes wrong, that rather misses the point. A better approach would be to solicit a wide range of opinions, take some responsibility for assessing the validity of each them and enter a purchase decision in the full knowledge that you know exactly why the vendor selected was the best fit for your firm, because the decision was one you reached yourself based on proven facts that could be directly benchmarked against your organisation’s own individual circumstances.

Share this Article: