The phishing problem is a hard problem for a number of reasons. Most difficulties stem from the fact that it is very easy for an attacker to create an exact replica of a good site, such as that of a bank, that looks very convincing to users. Previous work  indicates that the ability to create good-looking copies, as well as users' unfamiliarity with browser security indicators, leads to a significant percentage of users being unable to recognize a phishing attack. Unfortunately, the ease with which copies can be made in the digital world also makes it difficult for computers to recognize phishing attacks. As the phishing websites and phishing emails are often nearly identical to legitimate websites and emails, current filters have limited success in detecting these attacks, leaving users vulnerable to a growing threat.
Our overall approach, first described in , centers on extracting information that can be used to detect deception targeted at web users, which is accomplished by looking at features from each incoming email or potential attack vector. This process involves extracting data directly present in the email, as well as collecting information from external sources. The combination of internal and external information is then used to create a compact representation called a feature vector, a collection of which are used to train a model. Based on a given feature vector and the trained model, a decision is made as to whether the instance represents a phishing attack or not. We present a detailed description of our approach, which filters approximately 96% of phishing emails before they ever reach the user.
The remainder of this paper is organized in the following manner. Section 2 discusses previous approaches to filtering phishing attacks, while Section 3 gives an overview of machine learning and how we apply it to the task of classifying phishing emails, and how it could be used in a browser toolbar. Section 4 covers the results of empirical evaluation, as well as some challenges presented therein. Section 5 presents some concluding remarks.8] and Netcraft  toolbars. As reported in , most toolbars are lucky to get 85% accuracy identifying phishing websites. Accuracy aside, there are both advantages disadvantages to toolbars when compared to email filtering.
The first disadvantage toolbars face when compared to email filtering is a decreased amount of contextual information. The email provides the context under which the attack is delivered to the user. An email filter can see what words are used to entice the user to take action, which is currently not knowable to a filter operating in a browser separate from the user's e-mail client. An email filter also has access to header information, which contains not only information about who sent the message, but also information about the route the message took to reach the user. This context is not currently available in the browser with given toolbar implementations.
Future work to more closely integrate a user's email environment with their browser could alleviate these problems, and would actually provide a potentially richer context in which to make a decision. As discussed later in this paper, there are some pieces of information available in the web browser and website itself that could help to make a more informed decision, especially if this information could be combined with the context from the initial attack vector, such as the email prompting a user to visit a given website. This is discussed in greater detail in Section 3.3.
The second disadvantage of toolbars is the inability to completely shield the user from the decision making process. Toolbars usually prompt users with a dialog box, which many users will simply dismiss or misinterpret, or worse yet these warning dialogs can be intercepted by user-space malware . By filtering out phishing emails before they are ever seen by users, we avoid the risk of these warnings being dismissed by or hidden from the user. We also prevent the loss of productivity suffered by a user who has to take time to read, process, and delete these attack emails.7], in which the authors use structural features of emails to determine whether or not they represent phishing attacks. The features are mostly linguistic, and include things such as the number of words in the email, the ``richness'' of the vocabulary, the structure of the subject line, and the presence of 18 keywords. Other examples include the filter built into Thunderbird 1.5 . However, this filter is extremely simple, looking for only the presence of any one of three features, namely the presence of IP-based URLs, nonmatching URLs (discussed in Section 3.2.3), and the presence of an HTML ``form'' element. The Thunderbird built-in filter still only presents a warning to the user, and does not avoid the costs of storage and the user's time. In our implementation and evaluation, we seek to fill this gap in email-based phishing filters. Our approach is generalizable beyond email filtering, however, and we do note how it could be used and what changes would be required in the context of filtering web pages as opposed to emails.
Many people have proposed ways in which to eliminate spam emails in general, which would include phishing emails (see, for example, [17,9,16,27,26,18]). A number of early attempts at combating spam emails were based on so-called ``naïve'' approaches, ranging from ``bag-of-words'', in which the features of an email are the presence or absence of highly frequent and rare words, to analysis of the entropy of the messages. While these approaches looking at the text of the email appear to do well for spam, in practice these approaches often fail to stop phishing emails. This makes sense, as phishing emails are designed to look as close as possible to a real, non-spam email that a legitimate company would (or already has) sent out. As such, it is our belief that to stop phishing emails, we need to look at features selected specifically to detect this class of emails.
Looking at class-specific features is not a new approach in email filtering. SpamAssassin , for instance, has a number of rules that try to detect features common in spam email that go beyond just the text of the email. Such tests include things like the ratio of pixels occupied by text to those occupied by images in a rendered version of the mail, presence of certain faked headers, and the like. Spamato  is another extensible filtering platform that ships with a number of advanced filters, such as Vipul's Razor  (a collaborative algorithm using both URLs and message hashes), that work in tandem to detect spam emails. Our contribution is a new approach focused on learning to detect phishing, or semantic attacks in general. We do this by extracting a plurality of features designed to highlight deception, utilizing both sources of information internal to the attack itself, as well as external sources to gain more information about the context of the attack. Our solution can easily be used in conjunction with existing spam filters. The solution significantly reduces the amount of phishing emails with minimal cost in terms of false positives (legitimate emails marked as phishing).
Our approach, PILFER, is a machine-learning based approach to classification . In a general sense, we are deciding whether some communication is deceptive, i.e. whether it is designed to trick the user into believing they are communicating with a trusted source, when in reality the communication is from an attacker. We make this decision based on information from within the email or attack vector itself (an internal source), combined with information from external sources. This combination of information is then used as the input to a classifier, the result of which is a decision on whether the input contained data designed to deceive the user.
With respect to email classification, we have two classes, namely the class of phishing emails, and the class of good (``ham'') emails. In this paper we present a collection of features that has been identified as being particularly successful at detecting phishing, given the current state of attacks. We expect that over time, as the attacks evolve, new sets of features will have to be identified combining information from both internal or external sources. The features currently used are presented in Section 3.2, with Section 3.3 discussing how these can be adapted for use in detecting phishing web pages. In Section 4 we present a method for evaluating the effectiveness of these features, as well as the results of such an evaluation.3.2 also can be applied towards classifiying a webpage in a browser environment. A spam filter cannot generally be run on a webpage, so the last feature is not applicable, but the other features can still be evaluated with slight modification. For instance, the number of dots could be turned into two features - the number of dots in the URL of the current page, and the maximum number of dots in all URLs linked to in the current page. The same extension could be applied to the evaluation of the domain age feature. One could likewise split the presence of deceptive links into two features - deceptive links on the current page, and deceptive links on the previous page. This technique may enable the use of additional context, and would be especially useful if the user is coming to the attack site from a message in their web-based email interface. Additionally, one might be able to make use of additional context available in the browser and its history in features such as the following. 15] to hide the phishing site's URL in the email (or other attack vector). The browser is explicitly instructed to redirect to a new page, and as such it would be possible to create a feature out of whether or not the browser was redirected to the present page, or whether the user went to the current page explicitly. This information would be available in the context of a browser, but might not be available if only analyzing the source email. 30].
In this section, we present the details of our implementation used in evaluation of PILFER (Section 4.2) and in evaluating SpamAssassin (Section 4.3). The dataset of emails used to perform the evaluation is described in Section 4.4. Certain challenges are present when trying to do post-hoc analysis of phishing attacks, the specifics and impact of which are discussed in Section 4.5. Section 4.6 introduces some terminology, and Section 4.7 shows our results in classifying the dataset.3.2. Once the features are extracted, we train and test a classifier using 10-fold cross validation. (The dataset is divided into ten distinct parts. Each part is then tested using the other nine parts of the data as the training data. This ensures that the training data is separate from the test data, and is called ``cross-validation''.) For our reference implementation of PILFER, we use a random forest  as a classifier. Random forests create a number of decision trees (in our case, 10), and each decision tree is made by randomly choosing an attribute to split on at each level, and then pruning the tree. The exact workings of the classifier are beyond the scope of this paper. We evaluated a number of other classifiers as well, including SVMs , rule-based approaches, normal decision trees, and Bayesian approaches, but the overall accuracies of most of the classifiers were not different with statistical significance. Accuracies for some of these other classifiers are shown in appendix A. For a complete discussion of classifiers and text classification in general, the reader is directed to a machine learning text such as  or .
The age of the dataset poses the most problems, which is particularly relevant with the phishing corpus. Phishing websites are short-lived, often lasting only on the order of 48 hours . Some of our features can therefore not be extracted from older emails, making our tests difficult. For instance, in one of our features, we are interested in the age of domains linked to. We perform a WHOIS query to determine the date a domain was registered, and subtract this date from the date the email was sent according to its headers to determine its age. In many cases of phishing attacks, however, these domains are no longer live at the time of our testing, resulting in missing information. The disappearance of domain names, combined with difficulty in parsing results from a large number of WHOIS servers returning results in non-standardized formats resulted in only being able to programmatically extract registration dates for 505 of a total of 870 distinct domain names referenced in the dataset at the time of writing.
It is not clear whether this dataset is representative of normal people's email inboxes or not, but to date it is the best data we have been able to find. We are currently planning a follow-up study where we will be having users label every email coming into their inbox as either legitimate, spam, or phishing. This future work will provide us with a dataset more representative of real users' inboxes.
It is important to note that misclassifying a phishing email may have a different impact than misclassifying a good email, so we report separately the rate of false positives and false negatives. The false positive rate corresponds to the proportion of ham emails classified as phishing emails, and false negative rate corresponds to the proportion of phishing emails classified as ham. Let us denote the number of ham emails classified as ham (correctly classified) as , the number of ham emails classified as phishing as , the number of phishing emails classified as ham as , and the number of phishing emails classified as phishing as . We then define , the false positive rate, as
|Classifier||False Positive Rate||False Negative Rate|
|PILFER, with S.A. feature||0.0013||0.036|
|PILFER, without S.A. feature||0.0022||0.085|
On our dataset, we are able to more accurately classify emails using PILFER than by using a spam filter alone. PILFER achieves an overall accuracy of 99.5%. with a false positive rate of approximately 0.0013. PILFER's false negative rate on the dataset is approximately 0.035, which is almost one fourth the false negative rate of the spam filter by itself. These results are compared in detail with those of SpamAssassin in Table 1. As seen in the table, the inclusion of the result of a spam filter as a feature to PILFER makes for a significant reduction in phishing emails that get by. While PILFER without the spam filter's input has comparable accuracy to the spam filter, the accuracy obtained by providing the spam filter's decision as an input to PILFER, i.e. the combination of the two, improves the accuracy to be much better than either one alone. This result suggests that the features present in the two are catching different subsets of the phishing emails, and shows that a phishing filter and a spam filter can work well as complementary parts of an overall solution.
Table 2 shows the exact percentages of emails (by class) matching each of the seven binary features. All of the binary features are matched more frequently by phishing emails than by nonphishing emails. For the three non-binary features, their averages and standard deviations per-class are shown in Table 3. These features have higher mean values for phishing emails.
|Feature||Non-Phishing Matched||Phishing Matched|
|Has IP link||0.06%||45.04%|
|Has ``fresh" link||0.98%||12.49%|
|Has ``nonmatching" URL||0.14%||50.64%|
|Has non-modal here link||0.82%||18.20%|
|Is HTML email||5.55%||93.47%|
|Number of links||3.87||4.97||2.36||12.00|
|Number of domains||1.49||1.42||0.43||3.32|
|Number of dots||3.78||1.94||0.19||0.87|
In summary, PILFER can be either deployed in a stand-alone configuration without a spam filter to catch a large percentage of phishing emails with very few false positives, or in conjunction with an existing spam filter such as SpamAssassin for even higher accuracy. If a filter like SpamAssassin is already deployed, then adding PILFER has the advantage of significantly reducing the number of phishing emails making it to the user, while having no significant effect on the number of emails erroneously caught by the filtering system.
One might be inclined to think that phishing emails should be harder to detect than general spam emails. After all, phishing emails are designed to sound like an email from a legitimate company, often a company with which the attacker hopes the user has a pre-existing relationship. Models based on ``naïve'' assumptions, such as certain words like ``Viagra" being indicative of a class of un-desirable emails, no longer hold when the attackers are using the same words and the same overall ``feel'' to lure the user into a false sense of security. At the same time, phishing emails present unique opportunities for detection that are not present in general spam emails.
In general spam emails, the sender does not need to misrepresent their identity. A company offering to sell ``Viagra'' over the Internet does not need to convince potential buyers that they are a pharmacy that the user already has a relationship with, such as CVS or RiteAid. Instead, a spammer can actually set up a (quasi-)legitimate company called Pharmacy1283, and identify themselves as such, with no need to try to convince users that they are receiving a communication from their bank, or some other entity with which they have an established relationship. It is this mis-representation of sender identity that is key to the identification of phishing emails, and further work in the area should concentrate on features to identify this deceptive behavior.
As the phishing attacks evolve over time to employ alternate deceptive behaviors, so does the information available to combat these attacks. The approach used is flexible, and new external information sources can be added as they become available. These sources could take the form of web services, or other tagged resources, to provide additional information to the decision making process. For instance, many phishing attacks include copies of corporate logos, and if one could map a logo back to its legitimate owner's website, that would be valuable information in determining the authenticity of a website or email displaying that logo. As image sharing and tagging services such as Flickr  are increasing in use, it is not unreasonable to think that some day in the near future, one might actually be able to search with an image and get back a description as a result.
There are a number of emerging technologies that could greatly assist phishing classification that we have not considered. For instance, Sender ID Framework (SIDF)  and DomainKeys , along with other such sender authentication technologies, should help to both reduce false positives and make detection of spoofed senders much simpler in the time to come. Looking farther into the future, deeper knowledge-based models of the user and the types of prior relationships she may or may not have with different sites or organizations could also help fend off more sophisticated phishing attacks. Such techniques would likely build on ongoing research on federated identities and semantic web technologies . In the meantime, however, we believe that using features such as those presented here can significantly help with detecting this class of phishing emails. We are currently in the process of building a live filtering solution based around PILFER, which we will start making available to users for testing for further validation.
 K. Albrecht, N. Burri, and R. Wattenhofer. Spamato - An Extendable Spam Filter System. In 2nd Conference on Email and Anti-Spam (CEAS), Stanford University, Palo Alto, California, USA, July 2005.
 A. Alsaid and C. J. Mitchell. Installing fake root keys in a pc. In EuroPKI, pages 227-239, 2005.
 Anti-Phishing Working Group. Phishing activity trends report, Jan. 2005. http://www.antiphishing.org/reports/apwg_report_jan_2006.pdf.
 Apache Software Foundation. Spamassassin homepage, 2006. http://spamassassin.apache.org/.
 Apache Software Foundation. Spamassassin public corpus, 2006. >http://spamassassin.apache.org/publiccorpus/.
 L. Breiman. Random forests. Mach. Learn., 45(1):5-32, 2001.
 M. Chandrasekaran, K. Karayanan, and S. Upadhyaya. Towards phishing e-mail detection based on their structural properties. In New York State Cyber Security Conference, 2006.
 N. Chou, R. Ledesma, Y. Teraguchi, and J. C. Mitchell. Client-side defense against web-based identity theft. In NDSS, 2004.
 W. Cohen. Learning to classify English text with ILP methods. In L. De Raedt, editor, Advances in Inductive Logic Programming, pages 124-143. IOS Press, 1996.
 L. Cranor, S. Egelman, J. Hong, and Y. Zhang. Phinding phish: An evaluation of anti-phishing toolbars. Technical report, Carnegie Mellon University, Nov. 2006.
 N. Cristianini and J. Shawe-Taylor. An introduction to support Vector Machines: and other kernel-based learning methods. Cambridge University Press, New York, NY, USA, 2000.
 FDIC. Putting an end to account-hijacking identity theft, Dec. 2004. http://www.fdic.gov/consumers/consumer/idtheftstudy/identity_theft.pdf.
 I. Fette, N. Sadeh, and A. Tomasic. Learning to detect phishing emails. Technical Report CMU-ISRI-06-112, Institute for Software Research, Carnegie Mellon University, June 2006. http://reports-archive.adm.cs.cmu.edu/anon/isri2006/abstracts/06-112.html.
 F. L. Gandon and N. M. Sadeh. Semantic web technologies to reconcile privacy and context awareness. Journal of Web Semantics, 1(3):241-260, 2004.
 Gilby Productions. Tinyurl, 2006. http://www.tinyurl.com/.
 P. Graham. Better bayesian filtering. In Proceedings of the 2003 Spam Conference, Jan 2003.
 B. Leiba and N. Borenstein. A multifaceted approach to spam reduction. In Proceedings of the First Conference on Email and Anti-Spam (CEAS), 2004.
 T. Meyer and B. Whateley. Spambayes: Effective open-source, bayesian based, email classification system. In Proceedings of the First Conference on Email and Anti-Spam (CEAS), 2004.
 Microsoft. Sender ID framework, 2006. http://www.microsoft.com/senderid.
 T. M. Mitchell. Machine Learning. McGraw-Hill Higher Education, 1997.
 Mozilla. Mozilla thunderbird, 2006. http://www.mozilla.com/thunderbird/.
 J. Nazario. phishingcorpus homepage, Apr. 2006. http://monkey.org/
 Netcraft Ltd. Netcraft toolbar, 2006. http://toolbar.netcraft.com/.
 V. V. Prakash. Vipul's razor, 2006. http://razor.sourceforge.net.
 M. H. Rachna Dhamija, Doug Tygar. Why phishing works. In CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 581-590. ACM Special Interest Group on Computer-Human Interaction, January 2006.
 I. Rigoutsos and T. Huynh. Chung-kwei: a pattern-discovery-based system for the automatic identification of unsolicited e-mail messages (spam). In Proceedings of the First Conference on Email and Anti-Spam (CEAS), 2004.
 M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz. A bayesian approach to filtering junk e-mail. In Learning for Text Categorization: Papers from the 1998 Workshop, Madison, Wisconsin, 1998. AAAI Technical Report WS-98-05.
 Yahoo. Domainkeys, 2006. http://antispam.yahoo.com/domainkeys.
 Yahoo. Flickr homepage, 2006. http://www.flickr.com/.
 Y. Zhang, J. Hong, and L. Cranor. Cantina: A content-based approach to detecting phishing web sites. In WWW, 2007.
|SVM, C = 10||0.0024||0.0019||0.0408||0.0225|
|Nearest Neighbor w/ Generalization||0.0017||0.0022||0.0414||0.0265|
|Alternating Decision Tree||0.0020||0.0018||0.0405||0.0229|
|Pruned C4.5 Tree||0.0019||0.0017||0.0414||0.0235|
|Hybrid tree w/ Naïve Bayes leaves||0.0022||0.0017||0.0412||0.0209|
|Random Tree (1 random attribute/node)||0.0016||0.0015||0.0398||0.0200|
|AdaBoosted C4.5 tree||0.0019||0.0017||0.0414||0.0235|
|AdaBoosted Decision Stump||0.0016||0.0016||0.0748||0.0355|