News Shogun

[Overview] [What's New] [Features] [Versions] [Download]
[Sample Config Files] [Mailing Lists]
[Old pages]

WARNING! I will soon drop support for INN versions less than 2.3. See the news page for more details.

This software is currently in use on our servers for stopping spam and for making us more sound on 2senet. The current version is: [an error occurred while processing this directive]

In order to be a sound site on Usenet-2, an ISP has to be very diligent tracking posts from its users. It should be impossible for spam to come from its users, however spam detection software tends to be iffy at best. Mostly it relies on searching for strings in posts like "$$$", or having a list of spammers that you can block. The first method suffers because it is computationally expensive, and because it is not absolute. The second suffers because knowing your spammer usually means your spammer has sent out 20,000 posts and your newsmaster has had his mailbox flooded by thousands of angry usenetters.

This software relies on only one basic assumption for detection of spam: that spammers want their message seen as widely as possible.

What this software does is authenticate and track posts from a user. It is pure C code, so it is not as computationally expensive as spawning an entire perl process for post detection, and is fairly easy to integrate into any kind of data-transmission software. It is very extensible, and can persistantly store in a user's entry in a database any relevant post information for this user.

This package also comes with several extensions which, when put in place, basically ensure that your site can be Usenet-2 ``sound'' with no further effort on the News administrator's part required.

This package basically comes in two parts:

Back to top

That all said, here's how it works:

When a user connects to the news server, an attempt is made to authenticate the user without requiring a password.. If this is not possible, the remote user must give a password before attempting to post. Once the user is authenticated, a bunch of post-related limits for the user are loaded.. These can include: hierarchies between which this user's crossposts can't occur, number of groups between which crossposts (or followups) can occur in a single message, total number of messages posted, &c.

The package isn't used again until the user attempts to post a message.

When the user attempts to post a message, post monitors related to the limits are invoked on the message. If anything about the user's post runs contrary to the administration's posting policy, the post is blocked and a descriptive error message is relayed to the user.

The post can also be logged to a holding directory, where an external daemon will run into it and send it back to the news server. This can happen, for example, when the user has posted enough times that you want to see his posts, but few enough times that you don't want to block it indefinately.

Mail can be sent back to the user and to the administrators which will describe the reason the post got slowed, and how to get the limit raised.

Back to top

This package comes with several modules for user authentication/resolution, and for post tracking and blocking. Here is a brief list:

Back to top

User authentication/resolution

For the package to be worth anything at all, we have to be able to figure out precisely who the user at the other end of a connection is. This is configurable on a per-host basis, and every authentication scheme can take arguments which will help them to resolve the username.

Password file:
This authentication method requires both a username and a password. When the user enters a username and password, the username is looked up in a configurable password file which has lines of the format:

username:crypt(3)'ed password:ignored

where everything after the password is ignored.

If the encrypted password matches the password in the database, authentication succeeds.

Password database:
This is precisely the same as the Password file method described above, except that the username is looked up in a database where the key is the username, and the value is the crypt'ed password.

This resolution method does not take a password. What it does is search for a suffix in the remote host's hostname and strip that suffix off. The resulting string is interpreted as the username of the remote user.

remote ident server
This resolution method does not take a password. It will contact the ident server of the remote host and ask it who the user talking to us is. If the ident server sends back encrypted output, attempt to decrypt it using a keyfile, and look up the resultant uid in a uid-to-username database. This becomes the name of the remote user.

Back to top

Post tracking/blocking

Now that we know who the remote user is, we have to figure out what kind of auditing and limiting we want to apply to their connection.

Total number of posts
This monitor will check the total number of messages this user has sent out over a 24-hour period, and if they are greater than a certain number, a message "User has exceeded posting limits" is returned to the user next time they attempt to post. Post data is stored persistantly in a post database, so this is not thwartable by the user resetting their connection and posting in a new one.

Number of crossposts
This monitor will check the number of newsgroups in the newsgroups line, and if they are greater than a certain number, the post is blocked and the message "Crossposted to too many groups" is returned to the user.

Number of followups
This monitor will check the number of newsgroups in the Follow-up line (if it exists, otherwise uses the Newsgroups line) of the message, and if they are greater than a certain number, the post is blocked and the error "Followups set to too many groups" is returned to the user.

Exclusive hierarchies
This monitor will go through the newsgroups in the newsgroups line, and if it turns out that all the newsgroups don't fall into a defined hierarchy, the message "Crossposted between mutually exclusive hierarchies" is returned to the user. There is an implied "rest" hierarchy, which consists of newsgroups not defined in the other hierarchies.

Emily Postnews
This monitor will look in the user database for entries specifying posting permissions on certain hierarchies. If any newsgroups posted to are explicitely disallowed, the message "You don't have posting permission in %s" (where %s is replaced by the newsgroups you've attempted to post to but aren't allowed). If there are newsgroups that aren't explicitely accounted for, it will go through a list of hierarchies and associated rules files looking for intersections with the unaccounted for groups. If any hierarchies match, the same error message as above is generated with the groups that have rules appended to the %s part. The rule file associated with these hierarchies gets sent out, with the text of your message suffixed.

Back to top

Other News Patches

We've finished writing a patch to the innd transfer server which will modify the format of hosts.nntp such that you can outright reject an article if it's destined for a particular hierarchy. Russ Allbery modified it, and here is his description of his modifications:

I didn't want the poison patch for hosts.nntp (and in fact exclusions in hosts.nntp in general) to result in entries into the history file even when REMEMBER_TRASH is set to yes. This is because, with Usenet II, one may receive sound articles via unsound paths before sound paths.

This patch is now a part of recent INN-2.0 beta snapshots. If you are running a post May 22 INN, you don't need to download this patch.

Find it at .

[Overview] [What's New] [Features] [Versions] [Download]
[Sample Config Files] [Mailing Lists]
[Old pages]
These pages copyright (c) 1997-2001 Public Access Networks Corp. (PANIX)
All rights reserved.
Written by Aidan Cully
Designed by Aidan Cully and Alexis Rosen
From an original design by Alexis Rosen and Larry Nathanson
Last modified on Monday, 18-Mar-2013 16:56:41 EDT