Abstract: This paper is intended for all those who recognize the benefits from metadata management including, executives, business users & practitioners. Unlike in the past, where metadata management was always in the backseat, in the present many major enterprises have invested large amount of time and money in metadata management implementations and have reaped significant benefits. When we talk about metadata, we are really talking about “Knowledge” and the only answer to capture and preserve Intellectual capital of organization is METADATA. In a nutshell, this article encapsulates knowledge gained & lessons learnt from multiple metadata management implementations.
Few days before Christmas vacation, being Year end, in a major Bank/credit card company finance executives were very busy in planning their budget for the coming year. A Marketing Director responsible for “Rewards capability budget” was desperately trying to think of ways to derive forecasted liability numbers for reward points unutilized & minimize lead time to fulfill redemption requests from customers. He had to determine cash reserve amount to fulfill anticipated Redemptions. Director started thinking and imagining about both problems at once:
“The need to be able to produce forecasted liability on reward points”
And “Traceability of redemption requests”
The Director arranged for couple of brainstorming sessions with his BI consultants. The BI consultants recommended the implementation strategy and Metadata solution approach at a high level where the executive management could clearly understand their problem area being addressed and saw tangible benefits from the solution. Benjamin Franklin once said: “An Investment in Knowledge pays the best interest”. This was very true here as well. Only after having brainstorming sessions and investigating there current systems, the team was able to understand that they had all necessary information to tackle the issue, but they were not having integrated information across various systems and ability to trace data lineage out of it.
While integrating two or more data points, the link which connects all these data points is the metadata. Let’s discuss this with a real-time example.
Every business activity requires information and in turn produces information’s which adds value. Information is lifeblood of any enterprise. The enterprise might become dysfunctional or inefficient if the information flow does not happen or worst if it is misunderstood. One such classic example is interpretation of “Redemption Request Date”. This information is consumed by many business users in a redemption life cycle. Let’s see how this field is typically defined once, but has different interpretations when it crosses different business users.
- Customer – “Redemption Request Date is the date when the customer requested redemption”
- Credit Card Finance Division – “Redemption Request Date is the date when the authorization/invoice for the redemption is completed by the Bank/card company.”
- Vendor – “Redemption Request Date for the vendor associated with the Bank / card company (like say Jet Airways), is when the money for the invoice is in there account”
- Delivery Division – “Redemption Request Date for the delivery division is when the vendor actually dispatches the item for actual delivery.”
Though all of these data points talk about the same “Redemption Request Date”, but there context and interpretations are very different. From a business perspective, all that matters is if their customers get the promised service at promised time. Only this will increase the customer loyalty with the company and will generate revenues. A metadata solution would allow the business to have a data lineage on there data, in our case the Redemption Request life cycle. At the end of the day, Reward points are liabilities to Banks / credit card companies and hence needs to be addressed and forecasted very precisely & accurately.
A Metadata solution typically should answer following questions for business users with respect to data:
- What data already exists in Data Warehouse and what does it mean today?
- Where is it residing at the moment??
- How did it get there??
- How do I access that data??
Why Metadata Management??
- To Preserve Intellectual Capital: “Knowledge not captured is knowledge lost” which may turn out to be extremely expensive for organizations. Socialization of knowledge is very important.
- To address Regulations and Compliances: Ever increasing new regulations and compliances including Sarbanes Oxley demand documentations of all standard operating procedures and workflows. HIPAA, Basel II are getting strict about security and privacy rules in organizations.
- To achieve customer centricity: Taking companies from product-centric to customer centric is impeded by current DW architecture. Implementation of centralized information system management however, can circumvent this problem.
- To address Data Governance and Data Lineage: A successful Metadata Management is directly proportional to Data Governance and Data Lineage. Managing enterprise data is a challenge, particularly when the data is redundant, inconsistent, not documented, fragmented and difficult to find.
To establish control and insight, organizations will have to first build a metadata management framework (Figure 1) that integrates and leverages their data to their strategic advantage to deliver value. Either an individual or a group should own this. If no one owns up the problem, it will never be addressed.
Metadata Management Implementation Approach
Metadata elements are priceless asset of an organization, given that it represents a knowledge-base that has been created over time by many individuals. Metadata repository (MDR) is the technical backbone that is necessary to implement a knowledge management effort. BI is focused on identifying Key Performance Indicators of performance.
Metadata Sources & Elements
One of the biggest challenges with metadata solution is that it exists in many different sources, both structured and unstructured. Each source has its own metadata repository. Now, to create an enterprise MDM solution, companies will have to move away from the individual tool offering metadata repositories into a full-scale metadata integration tool. Below Diagram shows all the common metadata sources. This consists of both structured and unstructured metadata.
The metadata repository is relational and based on open standards – specifically the OMG Meta Object Facility and Common Warehouse Metamodel. These standards allow us to link metadata across systems automatically and allow us to extend the repository to include new or business-specific content. The repository can sit on relational database platforms and can be extended.
The task of creating an enterprise wide view of the organization is assigned to the BI team — users and developers who understand the current context completely but who need help getting the undocumented and under represented larger context. We know that multiple contexts will still be introduced, but time-to-market force the resolution of one usage context for information at a time.
Step by Step Ready Reckoner for Meta Intelligence Implementation:
A Meta Intelligence solution implementation would follow these steps:
- Gather current status including locations, formats etc which are already in use.
- Create a list of all metadata elements, including where they are, where they came from, who owns them, how it could be used and/or change them.
- Identify metadata that needs to be captured and managed
- Identify a centralized and definitive location where metadata will be managed
- Identify systems to capture structured and unstructured metadata from all possible sources
- Create tools/programs to share and synchronize metadata
- Educate business users about the impact and cost benefit from the metadata solution
- Design and Implement delivery approach
- Manage and monitor usage and compliance of metadata
Even today, less than 50% of the companies have deployed or currently deploying a “Meta Intelligence” solution. Very few statistics are available on deployments of Metadata Management solutions. One of the reasons why most companies have not initiated or have pushed Metadata Management efforts is because they do not understand the potential Return-On-Investment (ROI), the metadata offers. There is no doubt that, Metadata Management initiatives have heavy investments initially and implementation may take a while, but when the ROIs are measured, a lucid picture emerges that enterprises cannot do without Metadata Management in the coming years. While the supply of metadata is important, the demand and usage of the information is imperative for long-tem success of Metadata Management solution.
The following graph shows the ROI with tangible measures and its value of an Enterprise Metadata Management solution. However, success of this is heavily dependent of 3 factors which are prioritized in its order.
- Re-Use: How much of the information is actually being re-used. This would include any naming standards, transformation rules, or modeling templates. This would be an ideal indicator of the MDR maturity.
- Information Exchange: How much of information exchange is really happening?. Measuring this would be challenging but not impossible. Some analytic and trending tools do capture these measures by producing patterns of information’s and assets that are being most frequently used by users.
- Risk-Mitigation: Risk mitigation, which is both from the information management perspective as well as risks and regulations associated with sarbans oxley etc.
The measures and values derived from this curve are very valuable to address the priorities.
“If a Metadata solution existed 15 years back, Y2K issue would not have absorbed that magnitude of time and money in fixing a date type”
An enterprise DSS requires consent and commitment from all of the key departments within an organization. Defining clear, measurable business objectives is critical for building a DSS and justifying its cost-effectiveness to the organization. A metadata repository like many other DSS projects goes through several iterations.
The Metadata Solution implementation will allow the DSS team to analyze the dependencies among related metadata. Though “Redemption Request Date” is defined once, it has dependencies when it crosses different business domains. For ex: A vendor cannot process the Redemption request without having the invoice cleared from the Bank. This Data dependency will force to track the data from source (which is customer requesting the redemption) all the way till the end to the various processes like credit card bank, vendor etc until the customer receives the item/service in hand in the assured time.
Meta Intelligence – Value Proposition
- Single Version of Truth
- Better / Faster /Easier access to a corporate inventory
- Centralization of a “Corporate Knowledge Base”
- Data Dependency && Lineage Analysis
- Improved data consistency
- Data Dictionary: Allows self-service to End users
- Searching: search the repository by name, category, keyword etc.
- Sarbanes-Oxley Compliance: An Audit Proof IT lets the CIO assist the CFO
Metadata Management – Challenges and lessons Learnt
Successful implementation of Metadata Solutions in an enterprise essentially means transforming its information management and sharing approaches in accordance with the metadata strategy. However, the implementation has its own challenges because sharing occurs on a “need-to-know” basis. New solutions are expected to promote a “need-to-share” paradigm that enables all information to be available to all appropriate customers within the enterprise. Below mentioned challenges and lessons learnt are relevant for thoughtful consideration by any enterprise embarking on a Metadata solution implementation journey.
- Metadata Governance, Quality Assurance and Change Management process
- Quality of information stored in the Data warehouse
- Ability to clearly demonstrate of value-added services of metadata solution
- Business support for metadata initiatives
- Key stakeholders support
- Political && financial challenges
Future of Metadata Management – Google.com of Enterprise
While all organizations rely on metadata to some extent, few have a strategy for leveraging it in ways that can maximize sharing and re-use of critical information. The knowledge repository would be a ready reference where there are problems to be solved. This can house lessons learnt on projects from post-implementation reviews including what went right, what went wrong and why, with the creation of organizational best practices. Metrics will be useful to indicate what is and what is not being accessed, and also to give management a feel for the value of the knowledge repository. Personalized Metadata Directory would be as powerful as Google.com for IT which would allow universal search across systems, projects and repositories using appropriate keywords. We can visualize and get a pulse on operational statistics and quality metrics. Metadata management provides rich analysis functionality that allows IT and Business to understand and monitor their systems – just like they do with their customers. After detailed itemized statement feature was introduced by one of credit card companies, customers couldn’t do without it. This forced the competition to follow the trend. Metadata solution has similar potential. Once the executive management is enabled to get innovative services , are able to see Single version of truth and are able to get powerful search capabilities, at that point we can say the a Meta Intelligence solution is truly a Google.com for enterprise.
Image Source: meta-intelligence.com
Raghuveeran Sowmyanarayanan is a Vice President at Accenture where he works with clients to help identify and shape the “right” solutions for their various Business Intelligence (BI) and Information Management (IM) needs. Mr. Sowmyanarayanan also actively networks with industry leading BI & IM practitioners. He launched TDWI’s India Chapter in 2006 and served as its VP for 2 years. He has also organized and led multiple TDWI events in cities throughout India. Raghu began his career developing indigenous ERP products. Later, he got attracted to the concepts of Business Intelligence & Data warehousing and focused in those areas. Prior to Accenture, Raghu served as the BI/DW Lead for Energy & Utilities at Wipro Technologies and the BI/DW Lead for Genpact. Raghu is a well known author and speaker having published several articles in international journals like TDWI, DM Review, IT Toolbox & bicorner.com and spoken at multiple Information Management and Big Data conferences.