SoC improve the Spam-X plugin - Revision history

Dirk: language tweaks and clarifications; added a link to the CAPTCHA plugin announcement

2013-03-29T09:05:02Z

language tweaks and clarifications; added a link to the CAPTCHA plugin announcement

Dirk: dropped SWOT and downgraded difficulty to "low to medium"

2013-03-20T11:40:57Z

dropped SWOT and downgraded difficulty to "low to medium"

Dirk: added possible mentors

2012-03-09T18:11:09Z

added possible mentors

Dirk: moved information about other anti-spam services to a separate wiki page

2011-03-18T09:09:22Z

moved information about other anti-spam services to a separate wiki page

Dirk: SoC more Spam-X modules moved to SoC improve the Spam-X plugin: changed title to reflect changed content

2011-03-05T13:55:21Z

SoC more Spam-X modules moved to SoC improve the Spam-X plugin: changed title to reflect changed content

Dirk: API changes

2011-03-05T13:54:09Z

API changes

Dirk: reviewed text

2011-03-05T13:53:29Z

reviewed text

Dirk: filled in part 2

2011-03-05T13:42:04Z

filled in part 2

Dirk: rewriting this idea page again for GSoC 2011 (work in progress)

2011-03-05T13:16:31Z

rewriting this idea page again for GSoC 2011 (work in progress)

Dirk: SWOT; Links

2010-03-06T10:53:33Z

SWOT; Links

@@ Line 8: / Line 8: @@
 == Incentive ==
-Spam-X works very well in practice. But of course, there is always room for improvement and that is what this project is about. We are looking for ways to make using the spam filter more efficient (for the site admin) and also try to extend the filtering capabilities.
+Spam-X works very well in practice. There is always room for improvement of course and that is what this project is about. We are looking for ways to make using the spam filter more efficient (for the site admin) and also try to extend the filtering capabilities.
 From a usage point of view, the handling of long lists of blacklisted phrases and IP addresses could be improved. Also, there are new anti-spam services that we would want to try out.
@@ Line 24: / Line 24: @@
 However, we would also welcome new ideas on how to better handle these issues. Surely, having a sortable list of several hundred entries is not the only possible solution to this? Here's a chance for a student (i.e. you) to come up with a clever idea that sets your proposal apart from the others.
-Other UI improvements that we're looking for (and that should be easy to implement) in this part of the project would be to ensure consistency with the "look and feel" of the rest of Geeklog. Currently, the Spam-X plugin is sticking out a bit, both from the way the admin panels work as well as from how they look. This would probably make a good first task in the project.
+Other UI improvements that we're looking for (and that should be easy to implement) in this part of the project would be to ensure consistency with the "look and feel" of the rest of Geeklog. Currently, the Spam-X plugin is sticking out a bit, both from the way the admin panels work as well as from how they look. This would make a good first task in the project.
@@ Line 37: / Line 37: @@
 This concept, however, can not easily be integrated into Geeklog right now. First of all, Spam-X currently expects either a "thumbs up" or "thumbs down" answer, after which the comment post is either allowed or dismissed. "I don't know" simply isn't supported. So to support this third possible reply, some changes will have to be made to Spam-X itself and to the Geeklog code calling it.
-There's also the problem that Geeklog's CAPTCHA plugin would always display a CAPTCHA to the user. So if Mollom were to return an "unsure" response, the poster would have to solve ''two'' CAPTCHAs, which would be very annoying. So we need a solution for this scenario, e.g. some communication with the CAPTCHA plugin.
+There's also the problem that Geeklog's CAPTCHA plugin would always display a CAPTCHA to the user. So if Mollom were to return an "unsure" response, the poster would have to solve ''two'' CAPTCHAs, which would be very annoying. We need a solution for this scenario, e.g. some communication with the CAPTCHA plugin.
-And finally, since we're going to have to change the Spam-X API anyway, this would be a good opportunity to address a design flaw: Once a comment post is considered spam, it is deleted. There is currently no way to store the post for later review (and possible approval). This is not as simple as it may seem, though, since currently Spam-X simply doesn't know where the post came from - it could be a comment or a story submission, both of which would have to be treated differently.
+And finally, since we're going to have to change the Spam-X API anyway, this would be a good opportunity to address a design flaw of the API: Once a comment post is considered spam, it is deleted. There is currently no way to store the post for later review (and possible approval). Fixing this is not as simple as it may seem, though, since currently Spam-X simply doesn't know where the post came from - it could be a comment or a story submission, both of which would have to be treated differently.
 Backward compatibility has to be considered. There is third-party code out there that uses the current Spam-X API that we don't want to break. An easy, though not the only, way out would be to introduce new API functions.
@@ Line 51: / Line 51: @@
 ''Possible mentors:'' [http://www.geeklog.net/users.php?mode=profile&uid=11721 Tom Homer], [http://www.geeklog.net/users.php?mode=profile&uid=408 Dirk Haun]
@@ Line 59: / Line 58: @@
 * [[Filtering Spam with Spam-X]]
 * [[Other anti-spam services]]
 [[Category:Summer of Code]] [[Category:Development]]

@@ Line 10: / Line 10: @@
 Spam-X works very well in practice. But of course, there is always room for improvement and that is what this project is about. We are looking for ways to make using the spam filter more efficient (for the site admin) and also try to extend the filtering capabilities.
-From a usage point of view, the handling of long lists of blacklisted phrases and IP addresses could be improved. Also, there are new anti-spam services that we would want to try out. And finally, there's also the idea of setting up a new anti-spam service that is based on a "web of trust".
+From a usage point of view, the handling of long lists of blacklisted phrases and IP addresses could be improved. Also, there are new anti-spam services that we would want to try out.
@@ Line 31: / Line 31: @@
 Geeklog currently ships with a Spam-X module for [http://linksleeve.org/ LinkSleeve] (aka SLV). At the time, this was the only free service available that didn't require creating an account, so that it is usable "out of the box".
-Over time, more anti-spam services have appeared (see below for a full list). One of the most interesting - and free - anti-spam services these days is [http://mollom.com/ Mollom], which is associated with the Drupal community (but not limited to Drupal sites).
+Over time, more anti-spam services have appeared ([[Other anti-spam services|see list]]). One of the most interesting - and free - anti-spam services these days is [http://mollom.com/ Mollom], which is associated with the Drupal community (but not limited to Drupal sites).
 One important difference between Mollom and other services is that it can return an "unsure" ranking for a comment post. This means that the post may or not be spam - Mollom isn't sure. So what do we do? Display a CAPTCHA to the poster.
@@ Line 42: / Line 42: @@
 Backward compatibility has to be considered. There is third-party code out there that uses the current Spam-X API that we don't want to break. An easy, though not the only, way out would be to introduce new API functions.
 == Level of Difficulty ==
-''medium''
+''low to medium''
-The usability changes and implementation of the Mollom module itself should be relatively straightforward (there already is a PHP class for Mollom). Changing the Spam-X API will be more demanding, especially since backward compatibility will be an issue. Implementing SWOT will also require more thought and work.
+The usability changes and implementation of the Mollom module itself should be relatively straightforward (there already is a PHP class for Mollom). Changing the Spam-X API will be more demanding, especially since backward compatibility will be an issue.
 ''Possible mentors:'' [http://www.geeklog.net/users.php?mode=profile&uid=11721 Tom Homer], [http://www.geeklog.net/users.php?mode=profile&uid=408 Dirk Haun]

← Older revision		Revision as of 18:11, 9 March 2012
Line 66:		Line 66:

	The usability changes and implementation of the Mollom module itself should be relatively straightforward (there already is a PHP class for Mollom). Changing the Spam-X API will be more demanding, especially since backward compatibility will be an issue. Implementing SWOT will also require more thought and work.		The usability changes and implementation of the Mollom module itself should be relatively straightforward (there already is a PHP class for Mollom). Changing the Spam-X API will be more demanding, especially since backward compatibility will be an issue. Implementing SWOT will also require more thought and work.
		+
		+	''Possible mentors:'' [http://www.geeklog.net/users.php?mode=profile&uid=11721 Tom Homer], [http://www.geeklog.net/users.php?mode=profile&uid=408 Dirk Haun]
		+

@@ Line 72: / Line 72: @@
 * [[Dealing with Spam]] in Geeklog
 * [[Filtering Spam with Spam-X]]
+* [[Other anti-spam services]]
 [[Category:Summer of Code]] [[Category:Development]]

← Older revision		Revision as of 13:54, 5 March 2011
Line 27:		Line 27:


−	== Part 2: A new spam filter module ==	+	== Part 2: A new spam filter module and API changes ==

	Geeklog currently ships with a Spam-X module for [http://linksleeve.org/ LinkSleeve] (aka SLV). At the time, this was the only free service available that didn't require creating an account, so that it is usable "out of the box".		Geeklog currently ships with a Spam-X module for [http://linksleeve.org/ LinkSleeve] (aka SLV). At the time, this was the only free service available that didn't require creating an account, so that it is usable "out of the box".

← Older revision	Revision as of 13:55, 5 March 2011
(No difference)

@@ Line 8: / Line 8: @@
 == Incentive ==
-Spam-X works very well in practice. But of course, there is always room for improvement and that is what this project is about. We are looking for ways to make using the spam filter more efficient and also try to extend the filtering capabilities.
+Spam-X works very well in practice. But of course, there is always room for improvement and that is what this project is about. We are looking for ways to make using the spam filter more efficient (for the site admin) and also try to extend the filtering capabilities.
 From a usage point of view, the handling of long lists of blacklisted phrases and IP addresses could be improved. Also, there are new anti-spam services that we would want to try out. And finally, there's also the idea of setting up a new anti-spam service that is based on a "web of trust".
@@ Line 15: / Line 15: @@
 == Part 1: Usability improvements ==
-In addition to using external services (see below) that rate posts as spam or not spam, site owners can use their own blacklists to fine-tune filtering of spam for their own sites. As a result, however, you'll often end up with long lists of blacklist entries. There are two obivous problems with this:
+In addition to using external services (see below) that rate posts as spam or not spam, site owners can use their own blacklists to fine-tune filtering of spam specifically for their own site. As a result, however, you'll often end up with long lists of blacklist entries. There are two obivous problems with this:
 # you don't know whether or not such a rule is still valid, i.e. used to filter spam
-# the long lists are hard to handle
+# the long lists are hard to use and maintain
 We have (raw and unfinished) patches for both of these issues ([http://project.geeklog.net/tracking/view.php?id=1076 #1076] and [http://project.geeklog.net/tracking/view.php?id=1077 #1077]). So the minimal goal for this part would be to finish and implement these changes.
-However, we would also welcome new ideas of how to better handle these issues. Surely, having a sortable list several hundred entries is not the only possible solution to this? Here's a chance for a student (i.e. you) to come up with a clever idea that sets your proposal apart from the others.
+However, we would also welcome new ideas on how to better handle these issues. Surely, having a sortable list of several hundred entries is not the only possible solution to this? Here's a chance for a student (i.e. you) to come up with a clever idea that sets your proposal apart from the others.
-Other UI improvements that we're looking for (and that should be easy to implement) in this part of the project would be to ensure consistency with the "look and feel" of rest of Geeklog. Currently, the Spam-X plugin is sticking out a bit, both from the way the admin panels work as well as from how they look. This would probably make a good first task in the project.
+Other UI improvements that we're looking for (and that should be easy to implement) in this part of the project would be to ensure consistency with the "look and feel" of the rest of Geeklog. Currently, the Spam-X plugin is sticking out a bit, both from the way the admin panels work as well as from how they look. This would probably make a good first task in the project.
@@ Line 35: / Line 35: @@
 One important difference between Mollom and other services is that it can return an "unsure" ranking for a comment post. This means that the post may or not be spam - Mollom isn't sure. So what do we do? Display a CAPTCHA to the poster.
-This concept, however, can not easily integrated into Geeklog right now. First of all, Spam-X currently expects either a "thumbs up" or "thumbs down" answer, after which the comment post is either allowed or dismissed. "I don't know" simply isn't supported. So to support this third possible reply, some changes will have to be made to Spam-X itself and to the Geeklog code calling it.
+This concept, however, can not easily be integrated into Geeklog right now. First of all, Spam-X currently expects either a "thumbs up" or "thumbs down" answer, after which the comment post is either allowed or dismissed. "I don't know" simply isn't supported. So to support this third possible reply, some changes will have to be made to Spam-X itself and to the Geeklog code calling it.
-There's also the problem that Geeklog's CAPTCHA plugin would always display a plugin to the user. So if Mollom were to return an "unsure" response, the poster would have to solve ''two'' CAPTCHAs, which would be very annoying. So we need a solution for this scenario, e.g. some communication with the CAPTCHA plugin.
+There's also the problem that Geeklog's CAPTCHA plugin would always display a CAPTCHA to the user. So if Mollom were to return an "unsure" response, the poster would have to solve ''two'' CAPTCHAs, which would be very annoying. So we need a solution for this scenario, e.g. some communication with the CAPTCHA plugin.
-And finally, since we're going to have to change the Spam-X API anyway, this would be a good opportunity to address a design flaw: Once a comment post is considered spam, it is deleted. There is currently no way to store the post for later review (and possible approval).
+And finally, since we're going to have to change the Spam-X API anyway, this would be a good opportunity to address a design flaw: Once a comment post is considered spam, it is deleted. There is currently no way to store the post for later review (and possible approval). This is not as simple as it may seem, though, since currently Spam-X simply doesn't know where the post came from - it could be a comment or a story submission, both of which would have to be treated differently.

← Older revision		Revision as of 13:42, 5 March 2011
Line 15:		Line 15:
	== Part 1: Usability improvements ==		== Part 1: Usability improvements ==

−	''~~TBD~~''	+	In addition to using external services (see below) that rate posts as spam or not spam, site owners can use their own blacklists to fine-tune filtering of spam for their own sites. As a result, however, you'll often end up with long lists of blacklist entries. There are two obivous problems with this:
		+
		+	# you don't know whether or not such a rule is still valid, i.e. used to filter spam
		+	# the long lists are hard to handle
		+
		+	We have (raw and unfinished) patches for both of these issues ([http://project.geeklog.net/tracking/view.php?id=1076 #1076] and [http://project.geeklog.net/tracking/view.php?id=1077 #1077]). So the minimal goal for this part would be to finish and implement these changes.
		+
		+	However, we would also welcome new ideas of how to better handle these issues. Surely, having a sortable list several hundred entries is not the only possible solution to this? Here's a chance for a student (i.e. you) to come up with a clever idea that sets your proposal apart from the others.
		+
		+	Other UI improvements that we're looking for (and that should be easy to implement) in this part of the project would be to ensure consistency with the "look and feel" of rest of Geeklog. Currently, the Spam-X plugin is sticking out a bit, both from the way the admin panels work as well as from how they look. This would probably make a good first task in the project.

@@ Line 3: / Line 3: @@
 == Introduction ==
-Comment spam doesn't need an introduction - pretty much every site gets it. Geeklog ships with its own spam filter, called [http://www.geeklog.net/docs/english/spamx.html Spam-X]. This filter can easily be extended by adding modules so that it can either be updated for the spammer's latest tricks or to add support for new anti-spam services. This project is about the latter - creating modules for new services.
+Comment spam doesn't need an introduction - pretty much every site gets it. Geeklog ships with its own spam filter, called [http://www.geeklog.net/docs/english/spamx.html Spam-X]. This filter can easily be extended by adding modules so that it can either be updated for the spammer's latest tricks or to add support for new anti-spam services.
 == Incentive ==
-Geeklog currently ships with a Spam-X module for [http://linksleeve.org/ LinkSleeve] (aka SLV). At the time, this was the only free service available that didn't require creating an account, so that it is usable "out of the box".
+Spam-X works very well in practice. But of course, there is always room for improvement and that is what this project is about. We are looking for ways to make using the spam filter more efficient and also try to extend the filtering capabilities.
-Over time, more anti-spam services have appeared. One goal of this project is to evaluate these services. And to be able to do that, we would need Spam-X modules to support these services.
+From a usage point of view, the handling of long lists of blacklisted phrases and IP addresses could be improved. Also, there are new anti-spam services that we would want to try out. And finally, there's also the idea of setting up a new anti-spam service that is based on a "web of trust".
-== Part 1: New modules for existing services ==
+''TBD''
-Here's a quick rundown of some existing anti-spam services:
+== Part 2: A new spam filter module ==
-==== Akismet ====
+Geeklog currently ships with a Spam-X module for [http://linksleeve.org/ LinkSleeve] (aka SLV). At the time, this was the only free service available that didn't require creating an account, so that it is usable "out of the box".
-[http://mollom.com/ Mollom] is loosely associated with the Drupal CMS. It requires signup and offers both free and for-pay services.
+Over time, more anti-spam services have appeared (see below for a full list). One of the most interesting - and free - anti-spam services these days is [http://mollom.com/ Mollom], which is associated with the Drupal community (but not limited to Drupal sites).
-One specialty of Mollom is that it has an "unsure" categorization for posts where it's not quite sure yet whether the post is spam or not. In this case, it displays a CAPTCHA, so that the poster will have to confirm that they are human.
+One important difference between Mollom and other services is that it can return an "unsure" ranking for a comment post. This means that the post may or not be spam - Mollom isn't sure. So what do we do? Display a CAPTCHA to the poster.
-==== TypePad AntiSpam ====
+This concept, however, can not easily integrated into Geeklog right now. First of all, Spam-X currently expects either a "thumbs up" or "thumbs down" answer, after which the comment post is either allowed or dismissed. "I don't know" simply isn't supported. So to support this third possible reply, some changes will have to be made to Spam-X itself and to the Geeklog code calling it.
-[http://antispam.typepad.com/ TypePad AntiSpam] is, as the name implies, associated with the TypePad CMS. It is currently (still) in beta. The service requires signup and is free ("and will always be free" -- quote from the website).
+There's also the problem that Geeklog's CAPTCHA plugin would always display a plugin to the user. So if Mollom were to return an "unsure" response, the poster would have to solve ''two'' CAPTCHAs, which would be very annoying. So we need a solution for this scenario, e.g. some communication with the CAPTCHA plugin.
-==== Stop Forum Spam ====
+And finally, since we're going to have to change the Spam-X API anyway, this would be a good opportunity to address a design flaw: Once a comment post is considered spam, it is deleted. There is currently no way to store the post for later review (and possible approval).
-Their API has an option to check for a poster's user name, which at first glance doesn't seem like a reliable criterion to detect spam (the API has other options as well). Michael Hampton (author of Bad Behavior) also [http://www.bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/ expressed some doubts] about the service.
+== Part 3: SWOT ==
-=== Discussion ===
+Most anti-spam services work on the assumption that the same sort of spam is going to hit a lot of sites. The bigger a spam wave, the more likely (and faster) it is going to be recognized by one of these services, as they get reports from sites all over the web.
 Once these services get big (i.e. the more sites they have reporting to them), there is a chance that smaller spam waves may not be recognized. So a spammer that only targets a few sites and with a low volume may get away with it. Of course, the admins of a site hit by this sort of spam will recognize it as spam and remove it. But how could they then alert other site admins?
@@ Line 73: / Line 52: @@
 == Level of Difficulty ==
-''easy to medium''
+''medium''
-Implementing modules for the existing services should be relatively straightforward. Some more thought and work will be required for the SWOT implementation.
+The usability changes and implementation of the Mollom module itself should be relatively straightforward (there already is a PHP class for Mollom). Changing the Spam-X API will be more demanding, especially since backward compatibility will be an issue. Implementing SWOT will also require more thought and work.
@@ Line 82: / Line 61: @@
 * [[Dealing with Spam]] in Geeklog
 * [[Filtering Spam with Spam-X]]
 [[Category:Summer of Code]] [[Category:Development]]

@@ Line 3: / Line 3: @@
 == Introduction ==
-Comment spam doesn't need an introduction - pretty much every site gets it. Geeklog ships with its own spam filter, called Spam-X. This filter can easily be extended by adding modules so that it can either be updated for the spammer's latest tricks or to add support for new anti-spam services. This project is about the latter - creating modules for new services.
+Comment spam doesn't need an introduction - pretty much every site gets it. Geeklog ships with its own spam filter, called [http://www.geeklog.net/docs/english/spamx.html Spam-X]. This filter can easily be extended by adding modules so that it can either be updated for the spammer's latest tricks or to add support for new anti-spam services. This project is about the latter - creating modules for new services.
@@ Line 25: / Line 25: @@
 [http://akismet.com/ Akismet] is associated with the WordPress blog platform. The service initially required a wordpress.com account, which made it not suitable for use in Geeklog (asking our users to sign up with a competitor's website would have looked odd). This requirement has since been dropped: You still need to sign up but can do so now from the Akismet homepage. The service is free (commercial options available).
-There is already an older version of an Akismet module for Spam-X. It will probably need a review to check for API changes.
+There is already an older version of an [http://gplugs.cvs.sourceforge.net/gplugs/akismet/ Akismet module] for Spam-X. It will probably need a review to check for API changes.
 ==== Defensio ====
@@ Line 45: / Line 45: @@
 [http://www.stopforumspam.com/ Stop Forum Spam] is a private (one-man?) project. It doesn not require signup and is free.
-The API has an option to check for a poster's user name, which at first glance doesn't seem like a reliable criterion to detect spam (the API has other options as well). Michael Hampton (author of Bad Behavior) also [http://www.bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/ expressed some doubts] about the service.
+Their API has an option to check for a poster's user name, which at first glance doesn't seem like a reliable criterion to detect spam (the API has other options as well). Michael Hampton (author of Bad Behavior) also [http://www.bad-behavior.ioerror.us/2010/02/20/stop-forum-spam/ expressed some doubts] about the service.
 === Discussion ===
-''(more to come)''
+Since a site would normally only use one (or maybe two) of these services, we would also need an option in the Spam-X plugin to disable modules. Currently, you can simply drop new modules into the Spam-X plugin's directory and they will be picked up automatically.
 == Part 2: SWOT ==
-''TBD''
+The services discussed above work on the assumption that the same sort of spam is going to hit a lot of sites. The bigger a spam wave, the more likely (and faster) it is going to be recognized by one of these services, as they get reports from sites all over the web.
 == Level of Difficulty ==
@@ Line 64: / Line 80: @@
 == Further Reading ==
-''TBD''
+* [[Dealing with Spam]] in Geeklog
 [[Category:Summer of Code]] [[Category:Development]]