How to measure Search Relevancy
Normalized Discounted Cumulative Gain (NDCG)
In Enterprise Search, ranking results is one of the significant challenges. To evaluate the ranking quality, a laborious study called Normalized Discounted Cumulative Gain (NDCG) can be used. Getting an average using this method requires a large volume of data.
To better understand the perception of ranking by different users, let us consider an example. If we search for “Apple” on Google, the top result is www.apple.com, which is considered a good ranking. However, if a gym trainer searches for “Apple” with the goal of finding the calories in an apple, they may not be satisfied with the search results.
In an enterprise environment, consider two users searching for “Network Manager.” One is looking for a person with this “Network Manager” job title, while the other is searching for software with the same name. In such a case, the search results may be the same for both. However, how can we determine if the ranking is optimal?
Enterprise search engines face the challenge of displaying unstructured information from various sources in the best possible order.
NDCG, or Normalized Discounted Cumulative Gain, is a metric used to evaluate the effectiveness of a relevancy model. It measures the quality of the ranked list produced by the search engine.
To calculate NDCG, you need a set of relevance judgments, which represent how relevant each item in the results is to the user. In practice, you can obtain these judgments through user feedback.
Depending on the nature of your system, you may need a large number of users to generate meaningful insights from your data.
Once you have the ranking of items and judgments of each user, you can calculate the NDCG for each user and each query.
The mathematical expression to calculate relevancy is
I will decipher step by step, and you can find the Excel template at the end.
Imagine, the first result of a query returned by a search engine is referred to R1, the second is R2, and so on. We take a sample of the top 6 search results i.e., R1, R2, R3, R4, R5 & R6, to measure the quality of the results. The order of calculation will be CG, DCG, IDCG, and NDCG.
Each result from R1 to R6 will be given a number between 0-3. 0 is irrelevant and 3 is a perfect result a query may return.
Irrelevant: 0
Partially relevant: 1
Relevant: 2
Perfect: 3
To rank these results, relevancy will be decided by the user, and the sum of relevancy is the CG.
There is a problem, in this method, we see R3 is perfect. Perfect results should appear on top. It should be in 1st or 2nd position, for this reason, R3 is devalued. We need to reduce the relevance of 3 to a lower number. We use the discount method (Relevance / log2(i+1)). Similarly, R5 & R6 should appear at a higher position in the list than R4 (Irrelevant), those will be devalued as well.
The new SUM is DCG.
The most relevant R3 appears further down in the list, resulting in low-quality search results. The relevancy quality score of DCG is lower than CG.
What would be the ideal set of results? That will give us IDCG. In the table below, I arranged the results in the best possible order. The SUM gives us the value of IDCG
Whatever the order of results a search engine provides us with, arrange that in the best possible order. The IDCG is low in this instance since R4 is irrelevant and still appears in the results, which reduces the quality of the search results. We want to compare our actual results with the IDCG. Divide the DCG by IDCG and we get the Normalized Discounted CG.
The higher the NDCG, the better the quality of search results. Assuming the search results are of the highest quality, DCG and IDCG will have the same value, and dividing them will result in the value of NDCG being 1.
It’s critical to consider two drawbacks of this quality check. First, NDCG ignores the fact that all results can be of inferior quality. Suppose all results are partially relevant, the NDCG will still be 1.
Another weakness is that NDCG does not consider missing results. If the most relevant result is missing from the list, it will have no impact on the NDCG value.
Getting input from multiple users is another challenge. To ease user input, I have created a template in Excel. It will help the user to rank the results they received in their search query. It automatically calculates the NDCG. Here is the link to the template.
