IOC Deduplicator

A merged indicator list — three analysts' notes, two vendor reports and a SIEM export — is full of entries that are the same thing written differently. Comparing the raw strings finds almost none of them.

So comparison happens on a normalised key instead: refanged, case-folded, trimmed, trailing dot removed. EVIL.com, evil[.]com and evil.com. collapse to a single entry. The first spelling encountered is the one kept, and input order is preserved, so the output still reads against the source list rather than arriving in some arbitrary new order.

Lines split on newlines, commas and semicolons, which covers the shapes these lists actually arrive in. Blank lines are ignored rather than counted.

Twelve lines containing four distinct indicators return four entries, and the status strip reports eight removed. Turning on occurrence counts appends ×3 to any entry that appeared more than once — useful for spotting which indicator every source independently reported.