De-Duping for Merchants

What is de-duping for online merchants?
It’s a process of comparing, identifying and validating legitimate sales within the marketing mix based on the user tracking data. I have been working on all sorts of online campaigns to generate traffic and potentially sales and I strongly believe that merchants do need to analyse the de-duping process for both their online and offline sales.

Why merchants should de-dupe sales?
I personally de-dupe merchant sales for two reasons. First of all to make sure that merchants only pay once for a valid sale (not in all cases) and the second reason is related to campaign analytics that means to identify which referrer or campaign is contributing to the last click sale.

Please note that you do not need necessarily explore analytics or tagging software at this stage and I am not intending to recommend any. The aim of the article is to provide basic understanding to merchants on why and how should they de-dupe sales? However, it is important to start thinking what would you be aiming to achieve and then choose a relevant technology.

Rules for de-duping campaigns!
Merchants should define a set of rules before start looking into de-duping process i.e. which campaigns should be included in the de-duping process. Merchant do wish to control the cost and stop leakages in online campaign tracking but also do not alienate the paid campaigns those are mainly based on the CPA model. Once you have identified the rule then it would be interesting to list down all the campaigns you would like to add in the de-duping circle.

I have worked for several merchants in the UK and I personally suggest the following rules for online merchants in computers, home and garden and electronics market.

1 – Add your monthly subscribed campaigns that could be PPC, Affiliate, Media, email etc
2 – Add your link building campaigns only for first year i.e. Yell, Directories etc
3 – Do not add random link campaigns you have generated with automated link software or outsourced to a guy from India
4 – Do you want to add your returning customers? Be careful on that as most websites in CPA model work on customer loyalty platform and would like to earn commission on each sale from a new or registered customer
5 – Do not add social media bookmarks. These are typically free and generated by other users
6 – Do not add social network bookmarks but consider your own social application i.e. Facebook or iphone application etc.
7 – Do not add random articles written by third party, users on their website or blog, review services, send to friend forms on website, comments from existing customers
8 – Add offline campaigns i.e. paid magazines, newsletters etc. Well, think again on this…

It does not matter if your campaigns are based on CPA (affiliates), CPC (search), CPM (media) or other hybrid models. It is imperative to add all the campaigns in the mix based on your rules.

One of the important elements of de-duping is to control duplicate sales within affiliate mix if you are running affiliate campaigns from more than one network. Other campaigns are pre-paid or must-be-paid based on the traffic but if you do not wish to add other campaigns in the de-duping process then you might be paying a lot more than you should, primarily on your CPA campaigns.

CPA – Cost per Action/Acquisition means that the bounty is paid when an agreed action is performed such as making a purchase or applying for a credit card etc.
CPC – Cost Per Click means paying agreed amount on per click only, as I mentioned must-be-paid campaigns
CPM – Cost per Thousand means paying an agreed amount for impressions per thousand.
Don’t confuse yourself if it stands for CPM and not for CPT.
Hybrid Models – these are based on combination of CPA, CPC and CPM models.

De-dupe experiment!
OK, you have setup the rules and identified the campaigns at this stage. Now let’s analyse some scenarios based on the sample image below.

You can view each order had several referrals before the users decided to make the purchase. The cost to the retailer on each sale is also shown based on the agreements with the marketing campaigns. I am not sure how many times a user may divert and come back to your website from a different channel/referrer before making the purchase but I have seen between 2-6 visits in Home & Kitchen sector. It would of course depend on your sector, marketing campaigns channels, consumer segmentation and targeting techniques, relevance, product type, price sensitivity, availability, loyalty with the brand and much more. A typical online customer buying weekly grocery from Asda or Tesco websites would make the decision in the first or second visit but may browse a retailer website several times/sessions when installing a new kitchen, matching bed linen colours and booking holidays abroad. Also, a single decision maker would be able to finalise the sale in less visits as compared to when more members/family is making a combined decision. The consumer buying behavior and decision making power are separate topics that will be discussed in some other session so let’s get back to our de-duping process.

De-dupe results!
We have seen that de-duping against selected campaigns could save considerably towards your marketing cost. After de-duping the CPA based campaigns do not generate duplicate sales across networks and only the last referrers are honoured with the commissions. The campaign cost for first sale hasn’t changed because the first three campaigns are based on the CPC model and they must be paid on each click and the last referrer is an affiliate network, based on CPA model that should only be paid when the sale is conducted and they are the last referrer. The total cost before de-duping is £13.90 and £7.70 afterwards, reducing the cost by 45%. This is just an example to show you how you can calculate the cost and it could be more or less based on your campaign agreements. If you do not de-dupe sales but you do have an analytics/tagging package then try to drill down some reports based on clicks/conversions data and analyse in detail. You would probably understand the above concept just by looking at some consumer clicks that finally converted into sales and referred by several campaigns in single or different sessions.

Secondly, you also need to analyse and make some decisions here. Let’s get back to the image above. Google has referred two customers also participated in making the conversion decision and email campaign has also referred two customers but did not mange to convert the customer. I would think of the following scenarios here for a merchant.

1 – Conversion ratio of each campaign if they were the first referrer
2 – Conversion ratio of each campaign if they were the last referrer
3 – Who was the initiator that probably indicates where you are spending heavily OR you have highly relevant audience on that channel
4 – Which campaign is under-performing based on the conversions but they were the initiator i.e. Yahoo in our case. These are your choice of rules how to interrupt that information such as Yahoo provided the first impression to the customer or Yahoo did not manage to convert the customer at first opportunity.
5 – The cost and sale was comparatively lower for order no 3466 before de-duping. Does that mean those campaigns manage to convert a customer in fewer clicks and that combination is more effective and so on..

I will leave you on that to make up your own assumptions because you probably know your customers, campaigns, products and market segment better. Keep these attributes in mind because they will provide you a base and logically understand the rules to process the data with more consequential approach.

How to de-dupe?
Now we have learnt what is de-duping and why merchant should perform it regularly. Let’s get a bit technical and understand how can we de-dupe the data (click/conversion) in some easy steps but we must set up another rule first that is how often should we de-dupe? This is based on how you are de-duping mainly for CPA campaigns where conversion data is instantly passed on to the relevant parties.

Tip: If you only wish to de-dupe against affiliate campaigns then ask the networks about the local cookie and that would solve your problem of cross network duplication. However i strongly recommend investing in other technologies to build strong and long term solution.

Data collection for de-duping
First of all, merchants need an effective conversion tagging solution. I have worked with Tagman and it works perfectly. Please feel free to browse other providers on the internet or eConsultancy report on Website Analytics also offers a list of top analytics packages. Analytics packages are more advanced with additional facilities but our aim at this stage is to tag each click referral to store them in database and then fire the relevant campaign tag. The tagging software allows you to add additional parameters in the URL at the beginning or at the end that would help the technology to track each click, referral and conversion data.

For example your website domain is and the tagging could be done in many ways to identify the campaigns or referrals such as

The first URL has the campaign identifier at the end and the user would be directed on the merchant website. The second URL is directed to a tagging technology server to save the campaign id before redirecting the traffic to their merchant website. Merchants do not have to worry about the URL redirection and campaign keyword tagging because these are based on your choice of tagging or analytics package.

Here is a simple example to show you what and how the data is stored for de-duping? In our experiment image above for order no 3455; the data stored with minimum ease could be as follows.

The session ID shows the unique user, RefID is the campaign name, date and time and who contributed to the conversion on the last click. Of course this is very simple and you would probably view the conversion and all sorts of other data in your analytics/tagging package. The aim is to show you the trend and what you should expect when thinking to designing or purchasing an analytics or tagging technology.

Note: There are several free solutions available on the market. I have not explored in detail if they are effective for this purpose i.e. Google Analytics but you are most welcome to recommend in the comments box below.

We have covered three main sections by now. Firstly, what is de-duping and why is it important to merchants? Secondly, rules for de-duping and thirdly how it should be done? Once you have decided which solution should be used for de-duping, you should not have any problem to setup, configure and test the technology with the technology vendor.

Types of de-duping process
There are two main types of de-duping processes you will encounter with your chosen technology. Here is the brief overview of both with the additional benefits for choosing the right technology.

Manual de-duping
The first type of de-duping process is rather manual that means that the technology tagging system is collecting information in the database and you would be able to view the reports and extract the data but the data is not being de-duped instantly. This normally happens when the conversion codes (tags) from all the campaigns are dropped on the checkout page. Once the sale is conducted, the data is passed on to all the tags nested on the website based on the campaign cookie and also to the tagging technology. The problem is that that the conversion data has been passed on and the de-duping would be conducted manually through CSV or xml files mainly for CPA campaigns.


The second implementation process is exactly the same except one element that makes the real difference.
– Campaign conversion tags are nested on the de-dupe technology database and only de-dupe tags are remained on the website.

- Campaign clicks are directed to website through the de-dupe technology.
- Conversion data is passed to the de-dupe database from the website and then directed to the last referrer. In this case trade doubler will not receive the conversion data and it does not require manually de-duping the sales.
- Yahoo will also not receive the conversion data and the results will be more accurate.
The above case is ideal as it would only provide relevant data to the concerned parties based on the pre-defined rules and also eliminates manually de-duping. Another major benefit is that the conversion reports will produce accurate results.

How to de-dupe offline?
It is difficult to de-dupe sales against offline campaigns. First of all think carefully, do you really want to de-dupe sales against your offline activities? For example I have seen a nice kitchen unit ad in the local newspaper and only managed to memorise the brand name, finally I went to Google and found the website. This is a big argument if merchants should honour the conversion to Google or the local newspaper? If we are still convinced that the last referrer should be honoured then it will be Google. I let the merchants decide here..

The simplest way to track the offline media sales is to offer a unique discount / voucher code. When the code is redeemed on the checkout, the tagging system should be able to identify if the voucher code has been tagged with a pre-defined offline campaign or not? If it has then do not pass on the conversion data to the referrer otherwise let is go to the last referrer but isn’t the argument to honour the last referrer?

- De-duping for merchants is a process of comparing, identifying and validating legitimate sales within the marketing mix based on the user tracking data.
- De-duping can save thousands of pounds for large retailers and provide accurate results for the marketing campaigns.
- Setup rules on the marketing mix and do not include non-paid campaigns.
- Understand the reason, processes and methods of de-duping before identifying a technology to work on.
- Tag each new campaign and inform the referrer about the conversion process.

Good Luck..

Favourite Posts

Affiliate Marketing For Online Merchants

A series of ten blog posts explaining practical affiliate marketing for affiliates. xplaining practical affiliate marketing series of ten blog posts explaining practical affiliate marketing for affiliates.

Affiliate Marketing for Affiliates

A series of ten blog posts explaining practical affiliate marketing for affiliates. xplaining practical affiliate marketing series of ten blog posts explaining practical affiliate marketing for affiliates.

De-duping for affiliates

A series of ten blog posts explaining practical affiliate marketing for affiliates. xplaining practical affiliate marketing series of ten blog posts explaining practical affiliate marketing for affiliates.

Setting up a Cashback website?

A series of ten blog posts explaining practical affiliate marketing for affiliates. xplaining practical affiliate marketing series of ten blog posts explaining practical affiliate marketing for affiliates.

2015 Resolutions

A series of ten blog posts explaining practical affiliate marketing for affiliates. xplaining practical affiliate marketing series of ten blog posts explaining practical affiliate marketing for affiliates.


© 2016 All Right Reserved

Email :

Skype : adeel_ilead

Call UK : +44 207 078 7676