Deduplication Skill – Leo

It is frustrating to be skimming through your feeds and run into duplicate articles.

This happens for example when you have overlapping keyword alerts where two different keywords exist in the same article. It also happens when some sources publish the same articles into different RSS feeds. Finally, it happens a lot when a company issues a press release and other sources publish that press release with some minor changes.

Giving you the tools and control to tune your feeds is something we care passionately about. Today, we are excited to announce the beta release of a new Leo skill called Deduplication.

What is Deduplication?

This skill helps Leo detect that multiple articles are near exact duplicates of each other and cut that noise from your feeds. On the Web version of Feedly, you will see a small notification at the bottom right of your screen each time Leo removes duplicate from your feeds.

Which language does Deduplication work on?

The Leo deduplication skill works across all languages?

Which Feedly Plan does this skill require?

Because processing duplicates at scale is expensive, this skill will be initially rolled out as part of the Feedly Teams plan.

If you are part of Feedly Teams, there is a preference knob in the Leo settings page to disable this skill.

Beyond near exact duplicates

The deduplication skill is focusing on near exact duplicates. These are articles which have 85% or more overlap. We are working on a different skill called Business Events for articles which are reporting on the same event but with different content. In the case of business events, the content will be grouped instead of being removed.

Thank you!

We want to thank you Aymeric Bernard and Iheb Benabdallah for doing the preliminary ML research behind this Leo skill!