Authentication – “Are you real?”
Normally sender won’t be sending thousands of emails per day, therefore authentication or challenge method won’t be hindering their flow of production. But for spammer sending bulk mails in thousands, this method would definitely slow down their rate of successful sending and most spammer don’t even provide a valid return address!
The most common form of authentication method is as the method itself called “Self Authenticate”. This method is effective and would have zero or little false positive.
Example would be, Calvin sent Philip a email. Philip’s Anti spam appliance hold on to the email and sent a authentication mail back to Calvin asking him to verify his “Sending”. Once Calvin had verified by a return mail, his email address will be automatically added to the “Permitted sender” list and no further authentication mail will be send to him in future. The questions are “What happen if he didn’t reply the mail? Or if Calvin Anti spam appliance after receiving the Challenge mail from Philip’s Anti spam appliance sent the same challenge back to Philip? Normally for anti spam appliance with self authentication features, would parked these mail that was unauthenticated in various places. These could include “Certainly Spam”, “Probably Spam” or “Maybe Spam”. Depending on the features and functions used by the anti spam appliances, various different scenarios could happen here. Therefore for user choosing self authentication as their spam fighting tool, it is important to know and understand the nature and behaviour of the anti spam appliance against un-authenticated mails.
Greylisting is another form of challenge that is popular in many anti spam appliance. Unlike Self authentication that required user intervention, what is does is that it rejected the mail with a "450 temporary rejection". Most servers will try again after receiving the error. But for spammer that send thousands of mails a day would not do so. Therefore it greatly cut down the numbers of spam in the process.
Such methods would help in prevent spam but can never stop spam completely. Self authentication or Greylisting method can be an additional form of load for the appliance as well. Therefore when considering usage of this method, we have to take into the consideration of the numbers of users and load of the mails.
End of Part 3
Thursday, April 3, 2008
Wednesday, April 2, 2008
SPAM BUSTER! Part 2 of 4
Reverse Lookup – “Hello, who are you?”
Spamming is illegal in almost every countries, therefore almost all spammer used forged “From” address. Such forged addresses normally appear to be from trusted domain such as XX@yahoo.com, XX@gmail.com , etc. Another reason that Spammer forge email addresses is that most ISP have clauses that prevent spamming. Therefore forging of email address prevent ISP from locking down their network. So if we could prevent spammer from forging the “from” address, we would greatly reduce the numbers of spam. So how do we do that?
Reverse lookup basically is a process that associated a Host with a given IP address/ IP address revolve to a given host name. Spammer forgery address normally would not have a pointer record (PTR) to fulfill this requirement. Sender Policy Framework (SPF) is one of the methods used in reverse lookup to prevent email address forgery. SPF is a process where dedicated host are specified in the SMTP transaction stating the allow hosts to be allow to sent mail out of the domain. With SPF enforced, spammer would not be able to forge an email address undetected and action would be taken against forge mail accordingly.
In my earlier blog explaining how email is send, I mention about mail server searching for the assigned email server to received the mail based on the MX record associated with the recipient domain name. Similarly, the reversed lookup communicated with the DNS associated with the re-verse-MX record (RMX) to determine if the email from that particular domain is send by a permitted host. Reverse lookup seem like a good solution, but it is not without its own limitation.
One important thing that we must take into consideration when activating reverse lookup is that the sender's IP address may not be in the reverse DNS lookup record, or the sending server may have multiple names for the same IP, not all of which may be available from the reverse DNS lookup record. An example of such will be users in host-less or vanity domain.
End of Part 2
Spamming is illegal in almost every countries, therefore almost all spammer used forged “From” address. Such forged addresses normally appear to be from trusted domain such as XX@yahoo.com, XX@gmail.com , etc. Another reason that Spammer forge email addresses is that most ISP have clauses that prevent spamming. Therefore forging of email address prevent ISP from locking down their network. So if we could prevent spammer from forging the “from” address, we would greatly reduce the numbers of spam. So how do we do that?
Reverse lookup basically is a process that associated a Host with a given IP address/ IP address revolve to a given host name. Spammer forgery address normally would not have a pointer record (PTR) to fulfill this requirement. Sender Policy Framework (SPF) is one of the methods used in reverse lookup to prevent email address forgery. SPF is a process where dedicated host are specified in the SMTP transaction stating the allow hosts to be allow to sent mail out of the domain. With SPF enforced, spammer would not be able to forge an email address undetected and action would be taken against forge mail accordingly.
In my earlier blog explaining how email is send, I mention about mail server searching for the assigned email server to received the mail based on the MX record associated with the recipient domain name. Similarly, the reversed lookup communicated with the DNS associated with the re-verse-MX record (RMX) to determine if the email from that particular domain is send by a permitted host. Reverse lookup seem like a good solution, but it is not without its own limitation.
One important thing that we must take into consideration when activating reverse lookup is that the sender's IP address may not be in the reverse DNS lookup record, or the sending server may have multiple names for the same IP, not all of which may be available from the reverse DNS lookup record. An example of such will be users in host-less or vanity domain.
End of Part 2
SPAM BUSTER! Part 1 of 4
Let’s get back to the corporate world on spam fighting. Fortunately for many users sitting behind the computer screen, the jobs of fighting spams are left to the IT departments.
The problems here is that there are many companies in the markets providing anti spam solutions comprising of many difference package of solutions. So who do we choose? Many IT department brought anti appliance based on Sales talk. That is the greatest mistake, as a wrong appliance in place will cause much inconvenient on lost of emails, denial of services and lots of false positives. This blog is to help IT department on deciding the technologies best suitable for their environment.
I will not be listing the product brands and company names. Rather, I will list down the technologies available in the market, explain each of them and tell you which combination of the technologies is best in my point of view.
4 Type of key technologies uses in fighting Spam.
Filter – The old school method
Filter is commonly used by most anti spam appliances as one of their tools in blocking spam. Types of filters includes “Word list” or “Spam Dictionary”, “Black List” and “White list” of IP address, “Hash-Table” and “Bayesian spam filtering”. In my personal point of view, filter system are dangerous as it often result if high level of false positive rate especially in the early stage of implementation. It also required high level of user intervention due to the fast evolving changes in spam content and therefore need to fine tune the spam filter rules frequently.
Why did I say high level of false positive? Imagine that we use the word “SEX”. In most cases, “sex’ will be classified as a spam word. But if in the email that contain “Hi Joe, did you catch the show “Sex in the city” last night? This email will be block even though it is a harmless email between two friends. This are just one example, but you will be surprise to see common words listed in a default “spam dictionary” and you can imagine the numbers of mails to be blocked without an intensive level of fine tuning the “word list”.
While we are smart in adding in the key words appearing in the spam mail, Spammer are as good at modifying the words as well. I am pretty sure you people had seen “Viagra” to appear as “V1agr@”, ‘Vi@gra” or “V!agra”, etc. These words escape the “spam dictionary” and therefore result in false negative.
Filter system is effective with frequent fine tuning of the filtering system use. “Word list’, “Black list” of IP, etc must be updated frequently. Bear in mind that filter do not stop spam, it merely stop what you highlighted in your system. Even so, checking of misclassified email frequently is important to avoid missing of important mails.
End of Part 1
The problems here is that there are many companies in the markets providing anti spam solutions comprising of many difference package of solutions. So who do we choose? Many IT department brought anti appliance based on Sales talk. That is the greatest mistake, as a wrong appliance in place will cause much inconvenient on lost of emails, denial of services and lots of false positives. This blog is to help IT department on deciding the technologies best suitable for their environment.
I will not be listing the product brands and company names. Rather, I will list down the technologies available in the market, explain each of them and tell you which combination of the technologies is best in my point of view.
4 Type of key technologies uses in fighting Spam.
Filter – The old school method
Filter is commonly used by most anti spam appliances as one of their tools in blocking spam. Types of filters includes “Word list” or “Spam Dictionary”, “Black List” and “White list” of IP address, “Hash-Table” and “Bayesian spam filtering”. In my personal point of view, filter system are dangerous as it often result if high level of false positive rate especially in the early stage of implementation. It also required high level of user intervention due to the fast evolving changes in spam content and therefore need to fine tune the spam filter rules frequently.
Why did I say high level of false positive? Imagine that we use the word “SEX”. In most cases, “sex’ will be classified as a spam word. But if in the email that contain “Hi Joe, did you catch the show “Sex in the city” last night? This email will be block even though it is a harmless email between two friends. This are just one example, but you will be surprise to see common words listed in a default “spam dictionary” and you can imagine the numbers of mails to be blocked without an intensive level of fine tuning the “word list”.
While we are smart in adding in the key words appearing in the spam mail, Spammer are as good at modifying the words as well. I am pretty sure you people had seen “Viagra” to appear as “V1agr@”, ‘Vi@gra” or “V!agra”, etc. These words escape the “spam dictionary” and therefore result in false negative.
Filter system is effective with frequent fine tuning of the filtering system use. “Word list’, “Black list” of IP, etc must be updated frequently. Bear in mind that filter do not stop spam, it merely stop what you highlighted in your system. Even so, checking of misclassified email frequently is important to avoid missing of important mails.
End of Part 1
Subscribe to:
Posts (Atom)
Posts
