[medium] Remove Spam (#79)

When using Medium as a source it is possible that the a lot of items
might be spam when following a tag. To reduce the spam, we filter these
items based on a word list and an calculated score. The score is
calculated by the number of words which are included in the title from
our word list.
This commit is contained in:
Rico Berger
2023-11-26 17:03:54 +01:00
committed by GitHub
parent 03371cf645
commit ffbed73669
+35
View File
@@ -197,6 +197,41 @@ const skipEntry = (
return true;
}
/**
* Skip entries which might be spam. To detect possible spam, we check the
* title of the entry against a list of words, when the title contains 3 or
* more of these words we consider the entry as spam.
*/
const filterWords = [
'cash',
'loan',
'customer',
'care',
'helpline',
'number',
'patti',
'toll',
'free',
'paisa',
'call',
'kup',
'niewykrywalnych',
'fałszywych',
'pieniędzy',
'whatsapp',
];
const title = entry.title.value.toLowerCase();
let score = 0;
for (const word of filterWords) {
if (title.includes(word)) {
score += 1;
}
}
if (score >= 3) {
return true;
}
return false;
};