Asset Identifiers: The Most Important Piece Of The Metadata Interoperability Puzzle
Over the last few months, I have been involved with an increasing number of DAM integration projects where interoperability with some other third party system has been a key issue. At the same time, I have also had consulting assignments that were not related to Digital Asset Management, but more like metadata management and information architecture projects for financial services clients. What was striking in both cases was the similarity of what they needed to do and the marked differences in the approaches taken. There are some important lessons for those tasked with both devising DAM integration strategy and implementing it from this. Before I discuss those, a bit of DAM and systems integration theoretical background is required.
Most people, even they don’t have much knowledge of Digital Asset Management implementation are aware that digital assets (including metadata-only representations of non-digital assets) need to have unique identifiers so you can manipulate and organise them. There are various ways this can be achieved, but usually a number is used that corresponds to a record in a database of one sort or another. The necessity for unique identifiers is usually widely understood because people use them in their everyday lives (for example, phone numbers, social security identification, driving licence etc).
Even when dealing with entities that have commodity-like characteristics (e.g. bank notes, barrels of oil, gold bars etc) they also will nearly always have some unique number associated with each individual instance. This is essential because both producers and consumers need to be able to track and count them to verify that what has been delivered is what was requested. In the case of digital assets, it becomes even more important because they are likely to have more unique characteristics that make it essential that you can find a specific asset rather than being able to substitute one for another. As such, it is arguable that unique identifiers are the most important piece of metadata about a digital asset since without them, the link to all the other more descriptive data that provides context about an asset is lost and finding them again could become exceptionally time-consuming.
Most people who are users of DAM systems understand the significance of these identifiers, but don’t tend to need to deal with them much. That tends to be true until you get up to the administrator role for most systems and even then its usually only if there is some specific issue with one asset where numbers needed to be quoted to vendor support people to help them resolve a problem.
Where the role of these identifiers begins to take on a far greater significance is for integration requirements (i.e. interoperability between different systems). When these kind of tasks get beyond the discussion stage and move into implementation brass-tacks, usually the first problem everyone has to contend with is how to link from one key entity in a system with another in the corresponding application. If there are just two systems, this is usually not that difficult to deal with, especially if the exchange of data is one-way (e.g. DAM systems loading metadata from elsewhere). In this case, the DAM only has to hold the third party identifier so it can access its data and transform that into asset metadata which can be leveraged by users of the DAM. Fortunately, this remains the most common type of DAM integration task currently and while there will be lots of other thorny issues like mapping data, linking up taxonomies, user permissions etc, these are all engineering jobs which experienced implementation personnel will have dealt with before.
Unfortunately, these type of simple integration requirements will soon scale up to become far more demanding. The reason for this is that as DAM solutions become more embedded inside enterprises, the number of other applications they will need to get peered with is going to increase exponentially. In addition, not only is the DAM going to be receiving more data from a wider range third party application sources, it is also going to be sending it in the other direction too. My expectation is that for some people involved in DAM implementation (whether as vendors or project managers inside client organisations) this is already happening now and these are live issues they already have.
Anyone who has some experience of SAP and other larger enterprise platforms will be familiar with the nature of this problem and that they frequently require dedicated ‘switchboard’ style sub-systems that make it possible to find out where some piece of data originated from and the destination it is going to. In financial services, teams of people are employed for similar reasons to keep systems running and to follow up issues with different data sources etc. In part this is due to the fact that there are many legacy systems built years ago and also because interoperability has been an essential requirement for a far longer period. That might also explain why I think their thinking about this issue is far in advance of many in the Digital Asset Management sector.
I do not believe the DAM software industry (taken as a whole) is properly considering the scale of the complexity of the multi-system integration challenge at present. When I talk about it with the representatives of some firms, the general attitude tends to be that it is ‘just one more identifier’ and another API to read up about. There are two reasons for this lackadaisical attitude. The first is that software people (in any field) routinely underestimate how difficult nearly all implementation work will be (especially where certain quality thresholds must be met to get work signed off). Developers usually tend to build systems without thinking much about support for them as they assume someone else will do that job, even though it usually gets passed back up to them at some point (in addition, it should be pointed out that their managers are usually clamouring to see results in short order too). The second reason is that the DAM industry does not care about and has no clue about how to deliver interoperability that can scale up both in terms of data volumes (depth) and range of counterparties (width). This is because many developers have not had to do it much in the past, but as described, that will change, potentially quicker than many are currently prepared for.
As mentioned at the start of this article, I have also been involved with some financial services clients who had metadata management requirements. The big innovation which is getting lots of money and attention in their industry is blockchain technologies (the distributed ledger which makes digital assets like Bitcoin and Ethereum possible). Being able to use a shared ledger to store records of identifiers cuts out a whole tier of complexity which will potentially save them billions of dollars and make interoperability between service providers a far simpler undertaking. Rather than numerous identifiers needing to be stored and verification processes to ensure there is concordance between them, everyone can just use the one shared transaction identifier instead. A distributed ledger means that no one organisation wields power over everyone else and the standards are all open ones which can be implemented relatively easily. Since blockchains are immutable (i.e. cannot be changed once a transaction is committed) records persist indefinitely for as long as the blockchain exists, this has some major advantages for verifying ownership and also auditing what happened to an item of information and when.
It is true that while a common identifier helps to associate data, it does not solve the problems of mapping taxonomies and schemas from one system to another, so this is not the silver bullet for interoperability, but there is unlikely to ever be one. Thus far, previous attempts at DAM interoperability which I am aware of have gone nowhere because although a lot has been said about how ‘important’ this topic is, in reality, this is the DAM-equivalent of charitable pro-bono work, more is talked about it than work actually gets done. This being the case, getting to the point where just exchanging records is simpler to achieve than storing 13 or 14 different IDs along with each record might be quite a significant step forward.
I don’t know if blockchain technologies will have the same level of up-take in DAM as it seems they might in financial services, however, it does appear there are startups now getting funded to attempt something that could be used in this way. Those tasked with implementing complex multi-product DAM integration projects might want to consider writing identifiers to represent assets to a public distributed ledger (e.g. those used by Bitcoin or Ethereum) since it seems like it would offer the basis of reducing the complexity of the task and reduces the effort involved in agreeing integration frameworks. While wheel re-invention is often a popular exercise in the technology industry, it is not usually a very profitable one.
Share this Article: