Compound Metrics in Web Analytics: Do's and Don'ts

I used to be a big fan of compound metrics.  I basically had thought compound metrics would be the panacea to all diseases known to man-kind.   Especially back in the days when I was doing Artificial Intelligence research in graduate school, we were basically all competing to see on how complex and indecipherable we could make our function f(x) compared to our colleague's function g(x)

However, nowadays, where my primary goal is to measure the effectiveness of a marketing campaign or a web application, and no longer in academia devising complicated  equations, compound metrics, while it may still look impressive, have really lost much of its attractiveness. 

What exactly is a compound metric? And what does it mean in the context of web analytics? Compound metrics, sometimes also referred to as "calculated metrics" , "composite metrics", or  "synthetic metrics" are basically measures that combine two or more independent measures.  These kind of metrics are actually all around us; some examples are as follows:

Consumer Confidence Index: The consumer confidence index is designed to measure the degree of optimism on the state of the economy expressed by consumers based a combination of several measures of consumer attitudes.  Some of the topics used to determine the index includes business conditions, employment conditions, and upcoming family income.  Because these are all different measures by themselves, the combined index value that is formed is considered a compound metric.

Credit Scores: Your credit score is a three-digit number that creditors and lenders often use to make a decision about your credit-worthiness.  It is dependent upon many different factors, such as payment history, level of debt, length of credit history, number of credit score inquiries, etc.  Because these are all unique measures by themselves, the combined value that forms the final credit score is a compound metric.

Intelligent Quotient (IQ): The IQ Score is a very interesting compound metric.  The IQ is actually a ratio of the "Mental Age" (MA) and the "Chronological Age" (CA), which two separate metrics:  IQ = 100 ( MA/CA) . Furthermore, the MA itself actually constructed based on a variety of questions assessing attention, memory, and verbal skills, in which it is also a compound metric based on the different types of components that make up the test.

As you can see, compound metrics is quite popular with government and economic measurements, as well as several situation where a metric is standardized.  One important thing is to note is the distinction between compound metrics and various operations on a simple metric.  For example, taking the average or a percentage of a group of data points of a simple metric does not count as a compound metric.  There must be multiple unique measures combined in order to form a compound metric. 

Compound metrics have started to become very popular in web analytics, primarily because analysts practicing "Web Analytics 2.0" have started to take web analytics to the next step with trying to explain website engagement based on the combination of actions a user does on the website.  The idea is that when we are only looking at engagement of the website as individual actions such as "unique visits" or "click-throught rates", we can't tell the whole story of how the user engaged with the website.  For example, just by looking at the click-through rate doesn’t necessarily tell you if the website was worthwhile, combining the click-through rate along with the time the user stayed on the site would then give a fuller picture of whether the website engagement was strong.  If users stayed longer on the page after clicking through, the page is considered to be more engaging. And as such, compound metrics for "website engagement"  can be devised with the combination of click-through rate and time spent on the page.  The problem with compound metrics is that by blending the two metrics together into one, we are losing much of the visibility into the components that make up the metric which are the basic behaviors that we wanted to understand.  Analysts run the risk of abusing the usage of compound metrics whenever they cannot easily explain the direct actionable impact of a campaign, and need alternatives to show the ROI of a campaign, and it is easiest to do that by combining the metrics in attempt to tell a coherent story. 

Combining click-through rate and time on page can be relatively easily broken out.  However, even more complicated and vague compound metrics have arisen recently particularly in response to the challenge analysts are facing to explain impact of social media campaigns.  Because they cannot directly see the ROI of a social media campaigns in terms of conversions, or even click-through rates, analysts have worked hard to devise other metrics to explain the impact of social media for their businesses.  One particular area that has gained prominence is in sentiment analysis, where the metric aims to classify the polarity of a given brand or website, whether the expressed opinion is positive or negative.  Typically, the method to achieve this is based on several features and content around the web extracted from blogs or review sites or social networks.  The end result is a very subjective and abstract metric that is usually spit out in the forms of some seemingly random number like 3.7 or 19.3, or even worse, -1 or 1, implying negative vs. positive sentiment.  There are two major issues with this kind of metric.  With the case of 3.7 or 19.3, it is hard to understand how to take action against the number, because it is not clear how good 19.3 is or how bad 3.7 is by itself.  Furthermore, a lot has gone into the calculation of the metric and it is completely reduced to an aggregate number without any insights into the process, and it doesn't give you any clue on how to actually change for the better.

Because of the characteristics of the compound metrics, generally, these metrics tend to add more confusion than good.  This is not to say that all compound metrics used in web analytics are useless though, we just need to be really careful with using them.  The first thing to watch out for, is that compound metrics should be treated just like any simple measure.  Avinash Kaushik, a Google evangelist and a well-known blogger on web analytics, emphasized some important guidelines on the required attributes of a great metric.  A great metric should be uncomplex, relevant, timely, and instantly useful . These are fundamental attributes that should apply to all metrics, simple or compound. 

I think because of the complexity of compounded metrics, there are some additional attributes that a compound metric should possess before it can be used.   Firstly, it is very important to ensure that a compound metric is normalized, or standardized, to be within a fixed range.  As mentioned before, an arbitrary number is very hard to take action against without any context.  When a number is normalized or standardized, we know there is a range that we are working with, and if the number is closer to the extreme of the ranges, then we have a better idea of the impact of the metric. 

Along the same lines, because a compound metric is inherently more complex than a simple metric, it is always a good idea to present them in the context of other relevant metrics, or in the context of metrics that are commonly known.  This way, one can immediately relate to the data with the context and not have to think about what the arbitrary number mean in isolation.  One example is to present the metric's unit in terms of money or currency.  Everyone can understand the value of a good dollar, and if you have a metric that is presented in terms of money, more people can relate to it faster. 

A compound metric should also not contain more components than absolutely needed.  The idea of the compound metric is to answer questions that cannot be answered with a single component, but the more components you have, not only do you convolute the data and make it hard to interpret, you also run greater risk of losing the required independence of the individual component and confounding the combined factors.

Given all these criteria, it almost seem like compound metrics should be outright avoided in web analytics.  While in an ideal world, we would be able to explain all behaviors with simple measures based on direct observation, the fact of the matter is that with all the new media such as social media, compound metrics, when used properly, can help us achieve insights otherwise not possible.  Below are some of the examples of compound metrics, that in my opinion, are useful and powerful in giving new insights.

Monetization - $ Index:

I mentioned earlier that putting a compound metric in context of a familiar metric, such as in context of monetary data, is very useful because it shows direct impact to the business objective.  One excellent example is the $ Index, most commonly seen as a metric used in Google Analytics.  It is a measure to see the average value of for page that a user visited before reaching a goal page or converting on an eCommerce transaction.  While there are different calculations of $ Index depending on the tool, generally, the $ index for any page x it is based on two unique components:

Total Goal Value or eCommerce Transaction Revenue as a result of visiting Page X / Number of Unique Page Views for Page X

You can see an example of how this is calculated on Google Analytics

The $ Index is a very useful compound metric.  Firstly, it is closely related to the business objective, helping you map out a clear view of how much each of your page is actually worth directly related to your revenue.  Furthermore, the final unit is not a random, isolated unit of measure, it is kept in the form of dollars, which gives you immediate  context to evaluate the actual magnitude of the metric.  It is also "instantly useful", giving you immediate view of which pages are valuable for you and you can immediately work on improving the overall conversions of your website by targeting the poor-performing pages to improve the bottom-line.

Social Media - Twitter Metrics:

With the openess of Twitter, there are many interesting simple and compound metrics that have been formulated to help identify the effectiveness of one's influence, or clout, on Twitter.  One thing to be really careful with Twitter is the tendency to obsess over several metrics that are not useful and do not satisfy any of the fundamental characteristics needed of a good metric.  If you notice, with Twitter and social media channels, compound metrics are extremely rampant and can easily get carried away.  For example, many of you are familiar with dashboards that show your "Impact" score was at 5.7 and your influence score at 12.3, while your Amplification Score is 75.7%.  

That's GREAT (NOT?)!

Exactly, we have no idea what those scores mean, and how to act upon them or improve our campaigns.  Part of this is because Twitter analytics is so new that there isn't really any standard yet, so it's hard to put a standardized number on those metrics.   Even then, we should really focus on the meaning of each of these compound metrics, instead of obsessing over the "scores".  In my opinion, some useful  Twitter compound metrics are:

Retweets per thousand followers: This is really a ratio of two measures but since it does contain two unique measures, it is technically a compound.  This is a good, simple, metric to use to measure your amplification.  One simple extension is to also add the @ mentions as part of the measure in the numerator, since these are all amplifications tools in Twitter to acknowledge the contents you are posting.   

Churn: Measuring churn in Twitter is usually based on the changes in the number of followers vs. unfollowers over a certain period of time.  If you are actively participating in Twitter, this is a very quick metric for discovering how you are doing overall and can immediately provide useful information for you to continue doing what you have been doing with Twitter or perhaps change course. 

Engagement: Similar to amplification, engagement metrics can immediately tell you how others helping to amplify your brand on Twitter.  However, engagement metrics in terms of Inbound Messages per Outbound Messages can tell you immediately whether your efforts are getting a good return.  Understanding action and reaction is simple enough, and even with a just a ratio we can quickly determine whether our effort is being met with at least comparable results, or are we just a glorified loudspeaker without any engagement with our posts. 

Page Engagement - compounded clickthrough rate analysis:

Avinash Kaushik has for a long time criticized the general term of "engagement" as an excuse, and not a metric.  The main reason is that engagement as a term is a very vague and needs to be more specifically defined for the particular business or website in question.  Each website exists for several reasons and one or more metrics need to be defined in order to measure the engagement for the particular website.  This is where I think compound metrics can help provide a framework for defining good engagement metrics for each website.  Each website usually has an overarching goal.  For example, for eCommerce websites the main goal is usually to drive an online purchase.  The average conversion rate however usually hovers around 2%, maybe up to 5% if you are lucky, but rarely does that number change.  This means for over 95% of the time your users are doing something else on your website.  To understand the users' experience of that 95%, a few "micro" conversions should be defined. 

Engaged Clickthrough Rate: One compound metric that can be used to understand the user experience, is a compounded click-through rate analysis of the pages of your website.  One often looks at click-through rates from the variety of sources that drives traffic into the website, but if you deep dive into the traffic that comes in, a lot of the actual traffic are not really engaging with your website at all and have no intention of ever converting on anything you want them to.  They might have accidentally came to your website, or perhaps are just bots that crawl around the web.  One way to discount those visitors is to put a minimum time spent on site condition in the clicks that lead into your website, and do not count the visits or clicks that don't meet the minimum time spent on that page.  This compound metric immediately cleans up all the visits you are not interested in and focuses in on the visits that actually matter for you.  This metric can then be segmented as you need to for further analysis or comparisons. 

User Task Completion Rate: With the emergence of SaaS (Software as a service) web applications over the past decade, a website is often not just a marketing tool for selling service, but is also the interface that users engage in after the sale.  With that, users can perform multiple actions on the website and it is important to be able to measure how successful the user is while navigating through the variety of features on the website.  In this case, a compound metric for task completion can be devised to measure the user success of your web application.  The actual components though, can vary dramatically depending on your goals.  For a content heavy site, the task completion rate will contain time related metrics measuring how much time a user reads the content.  If there are multiple features in order of importance, then the metric could contain weighted completion giving partial credits to the different features on the website.  For example, if the primary purpose of your website is a dating site matching couples.  You might give higher weight to the couple that successfully scheduled a meeting on your site, vs. a couple that completed a chat session on your site.  Of course, you need to be in control of what success means to you, and the end result must be instantly obvious for you when you see the data. 

We have seen a variety of compound metrics in this post, both in the real world all around us, as well as metrics specifically for web analytics.  My take on compound metrics is that from the outside, it seems very attractive to go crazy on using compound metrics to help measure the success of your website, but it is so easy to get lost in the metric and not have an idea of how to actually use the metrics to act on improving your website, and that should be the first and only thing that should be on your mind when using any metric, whether simple or compound. 

We have also seen some cases of compound metrics that CAN be useful in help you.  Besides following the fundamental four characteristics of any metric, which are uncomplex, relevant, timely, and instantly useful, it is also good to follow the following guidelines regarding the compound metrics:
 

• Generally, metrics that relate to business outcomes, especially in terms of revenue and profit, are good.

• When the compound metric can be easily related to a common unit of measurement, such as the "dollar amount", it is generally better.  The $ Index is an example that we looked at. 

• When forming a new compound metric, it is important to ensure that the metric's scale and range are well defined and standardized whenever possible.  If you do not know how to act when the value is 9 or 454, then that is a major problem and it makes the metric totally useless.

• For new media such as social media, compound metrics can be useful in measuring the impact of the campaigns, but again it is very important to ensure that the metrics are as simple as possible with the end result easily tied to the business outcome.
• Finally, when there is a need to create more complex metrics such as undrestanding the task completion rate of your web application, it should be completely customized for your specific business and you should spend the time to understand what it means for you and your users to have a successful experience on your website. 

That concludes my post regarding what I feel are good uses of compound metrics.  Let me know your thoughts on compound metrics in general and what you think are good compound metrics. 

DRJWT6N5KJEQ