LEAFNODE

Table of Contents

1. Overview
2. Installation
3. Update from previous versions
   1. Update from versions < 2.0
   2. Update from versions < 1.9.3
   3. Update from versions < 1.6
4. Files and directories
   1. The main configuration file
   2. The filter file
5. Local newsgroups
   1. Moderated newsgroups
6. Deinstallation
7. Frequently asked questions
8. Copyright

1. OVERVIEW

Leafnode is a USENET software package designed for small sites, with a few tens of readers and only a slow link to the net. It is developed on Linux, but has been reported to compile and run also under any BSD flavour, Solaris and Irix.

The Leafnode package consists of several programs, three essential ones and several add-ons.

• Leafnode is the NNTP server. It talks to the normal news clients, and stores readership data.
• Fetchnews is the NNTP news-gatherer. It looks at the readership data Leafnode stores, and selects what groups to pull news from. It also transfers your contributions to one or more upstream servers.
• Texpire is responsible for deleting old and uninteresting news. It deletes all discussion threads that are old and not recently read.

These are the additional tools:

• Applyfilter lets you delete articles from your local news spool which fit a certain pattern.
• Checkgroups inserts the titles of newsgroups into the newsgroup database.
• Newsq shows which news are waiting to be transferred to your upstream server(s).
• Rnews sorts an UUCP batch or articles downloaded by other means into the news spool.

Only groups that someone has been reading in the past week are fetched from the upstream NNTP server. When someone stops reading a group, fetchnews will stop reading that group a week later (this is the default which can be configured), and when someone starts reading a group, fetchnews will grab all the articles it can in that group the next time it runs.

Leafnode's distinguishing features are:

• Leafnode is written with the novice user in mind. It is very easy to install and configure and requires no manual intervention once installed. Leafnode tries very hard to recover automatically from error situations.
• Compared to other servers, leafnode uses very little disk space and bandwidth. Of course this also depends on how many users are accessing your server and is probably not true for 300-user sites.
• The newsspool can be easily manipulated by small scripts because it is maintained in plain file format. Some tiny examples are in the subdirectory tools/ .

Of course, Leafnode also has weaknesses. Some of these are:

• In just any error situation, leafnode tries to fix its problems by deleting the offending article. This can cause leafnode to lose news.
• Due to its open architecture, Leafnode scales very badly.

The current version of leafnode is available from http://www.leafnode.org/ .

There is also a leafnode mailing list. Send mail to

leafnode-list@wpxx02.toxi.uni-wuerzburg.de

with "Subscribe" in the Subject: to subscribe.

2. INSTALLATION

1. Leafnode depends on the PCRE regular-expression matching library. It ships as binary package with many current Linux/BSD distributions, and is also available as port for *BSD. Just install the binary package for your operating system if it ships with one. If it comes without PCRE, don't worry, the installation from source is simple. To install PCRE, do:
   1. Download the current PCRE version from http://www.pcre.org/.
   2. Unpack PCRE: gunzip -c | tar xvf -
   3. cd pcre-3.7
      (This is for PCRE-3.7, for other versions, replace that by the path that tar wrote when unpacking.)
   4. type ./configure && make
      there should be no errors
   5. as root, do make install
   6. as root, do ldconfig
      this may not be needed on all systems, but better safe than sorry
   7. DONE!
2. Leafnode uses GNU autoconf to determine the configuration of the machine it will be compiled on. Type

            sh ./configure
   

   to create an appropriate Makefile and config.h. If leafnode cannot find your PCRE, it is probably an old version without pcre-config, or pcre-config is not in your path. Either add the directory containing pcre-config to your $PATH, or call configure like this (we assume PCRE is in /opt/pcre-3.7):

            env CPPFLAGS="-I/opt/pcre-3.7" LDFLAGS="-L/opt/pcre-3.7" sh ./configure
   

   NOTE: on some machines, when PCRE is linked against a shared library, you will need to add /opt/pcre-3.7 to LD_LIBRARY_PATH or to /etc/ld.so.conf.

   The configure script can take some flags (replace DIR by a directory name, FILE by a file name):

   --spooldir=DIR    To override the default /var/spool/news, where leafnode puts
                     its news spool.
   
   --lockfile=FILE   To override the default /var/run/news/fetchnews.lock, where
                     leafnode puts its lock file.
   
   --prefix=DIR      Leafnode installs itself normally in the /usr/local/sbin
                     directory. If you want to use another directory, use this
   		  flag. For example, if you want to install leafnode in
   		  /usr/sbin, use --prefix=/usr
   
   --sysconfdir=DIR  Default path for configuration is /etc/leafnode when --prefix
                     is not used or set to /usr, and PREFIX/etc otherwise.
   		  You can use this to change the config file location from
   		  its default.
   

   BEWARE: Former versions always defaulted their sysconfdir to /etc/leafnode. Current versions only override sysconfdir to /etc/leafnode if configured without --prefix or with --prefix=/usr. That way, if --prefix=/opt/leafnode, the sysconfdir goes to /opt/leafnode/etc, which is more appropriate.
3. Type

             make
   

   There should be no errors.
4. Create a user "news" if you don't have one.
5. Type

             make install
   

   If you prefer an installation with the binaries stripped free of symbols, use

   	  make install-strip
   

   instead.
6. For packagers: if you want to prefix all installation paths with a common prefix, e. g. to support RPM's BuildRoot feature: You can use make install-strip INSTALLROOT=/var/tmp/leafnode-root, it may change to DESTDIR in a future version though.
7. If you are updating leafnode from a previous version, see the section "Updating" below.
8. Edit $(LIBDIR)/config. $(LIBDIR) defaults to /etc/leafnode, but can be adjusted at build time. For documentation, see below or config.example and leafnode(8). Mandatory edits to the config file are:
   • server -- this must point to your upstream news server.
   • expire -- how many days will downloaded articles remain in the spool.
   Make sure the environment variable $NNTPSERVER or /etc/nntpserver points to your own host so clients will talk to leafnode rather than try to go to the upstream server.

   If you want to use filtering of the incoming spool, see the section on the filter file below.

9. Set up a cron job (as user "root" or "news") to run texpire every night or maybe every week. Here is my crontab line, which runs nightly:

   0 4 * * * /usr/local/sbin/texpire
   

   I did "crontab -u news -e" as root to edit the crontab file, and added this line. Substituting "1" for the third "*", thus:

   0 4 * * 1 /usr/local/sbin/texpire
   

   tells cron to run texpire at 4am every Monday morning.
10. Make sure fetchnews is run at the appropriate time. If you have a full-time link, run it out of cron (as "news" again); if not, run it when your connection to the net is established. If it is run as root it will change to user "news". If you use PPP, you can run fetchnews from /etc/ppp/ip-up (or /etc/ppp/ip-up.local, depending on your OS).
11. We assume that you have tcpd installed, it ships with most distributions. If you don't have it, fetch it from ftp://ftp.porcupine.org/pub/security/, it's in tcp_wrappers, and install it. It's preferable if you compile it with -DHOSTS_OPTIONS, and the examples below assume you did.
12. Choose either of these alternatives:
    • Alternative #1: If your system uses inetd (most commercial unices, *BSD). Edit /etc/inetd.conf so that $(BINDIR)/leafnode is executed for incoming NNTP connections. Here is my inetd.conf line:

      nntp    stream  tcp     nowait  news    /usr/sbin/tcpd /usr/local/sbin/leafnode
      

      Note: some systems install tcpd to a different path, but it's uncommon. Change the first path accordingly if your tcpd resides in /usr/etc or /usr/lbin. After these changes, force inetd to read the changed configuration file by sending it the HANGUP signal. To achieve this, issue the following command (as root):

      kill -HUP `cat /var/run/inetd.pid`
      

    • Alternative #2: If your system has xinetd instead. Add this to your xinetd.conf (this example assumes xinetd 2.3.3 or newer):

      service nntp
      {
              flags           = NAMEINARGS NOLIBWRAP
              socket_type     = stream
              protocol        = tcp
              wait            = no
              user            = news
              server          = /usr/sbin/tcpd
              server_args     = /usr/local/sbin/leafnode
              instances       = 7
              per_source      = 3
      }
      

      Then send xinetd an USR2 signal to make it reread its configuration. See the xinetd.conf(5) manual page for details. Note: some systems install tcpd to a different path, but it's uncommon. Change the first path accordingly if your tcpd resides in /usr/etc or /usr/lbin.
13. Write the following into /etc/hosts.deny:

    leafnode: ALL
    

    and into /etc/hosts.allow:

    leafnode: 127.0.0.1
    

    to protect your news server from abuse. If you want to make leafnode accessible to additional IP numbers/domains, add them in /etc/hosts.allow in the format described above. See hosts_access(5) and hosts_options(5) (if applicable) for more information about tcp wrappers.
14. (optional) If you want to allow read-only access, you can set the NOPOSTING environment variable, for example, put this into /etc/hosts.allow:

    leafnode: 192.168.0.4: setenv NOPOSTING "You may only read."
    

    The contents of this variable are printed at the end of the banner, with control characters (as per iscntrl(3)) replaced by an underscore.
15. Run fetchnews. The first run will take some time since fetchnews reads a list of all newsgroups from your upstream server. In the worst case, this can take up to 60 minutes, depending on how many newsgroups your provider offers and how reliable your modem connection is. To see fetchnews working, run it with -vvv.
16. Run texpire. It will create directories missing after the first install.
17. Read news using an NNTP client (with $NNTPSERVER or /etc/nntpserver pointing to your own host). Select the groups you want to read in the future. You will find them empty except a default article. Reading this article is necessary with some newsreaders to select the groups for further fetching.

    After this, you should have empty files in /var/spool/news/interesting.groups/ for every group you want to read.

18. Run fetchnews again. This run should pick up all the groups you want to read.

3. UPDATE FROM PREVIOUS VERSIONS

3.1 Update from versions < 2.0b8_ma8

The local.groups and groupinfo files and the hash function for message.id have changed from any former version to 2.0b8_ma8.

IMPORTANT:To fix the "remote" groups, run fetchnews -f -- it will take some time, because if refetches the whole active file, but that's needed anyhow to figure the group status (posting allowed/not allowed/moderated group).

To fix local.groups, if you have perl installed:

  perl -ple 's /\s+/\ty\t/' -i.bak /etc/leafnode/local.groups

If you don't have Perl, use a text editor that preserves HTAB characters, change all while space to TAB and insert an additional field reading just y:

Before: local.test some test group
After: local.testTAByTABsome test group

3.2 Update from versions < 2.0

From 1.9.x to 2.0 there were some changes to options in the main configuration file. The options "maxage", "maxlines", "minlines", "maxbytes" and "maxcrosspost" have become obsolete in the main configuration file and have to be specified in the filter file instead. The advantage of this is that you can do much finer selection on these criteria now.

To replace, for example, a "maxage = 5" specification in the main configuration file, you should add the following to your filter file:

newsgroups = *
maxage = 5
action = kill

The leafnode programs will issue warnings if they encounter obsolete specifications in the main config file.

3.2 Update from versions < 1.9.3

From version 1.9.3 on, the groupinfo file is sorted in a case-insensitive manner. To update correctly, do a "make update" as root after you have successfully completed "make install". This will re-sort the groupinfo file. The old groupinfo file will be stored as groupinfo.old just in case something goes wrong. You also have to change your main config file manually as described in the last section.

3.3 Update from versions < 1.6

Between leafnode-1.6alpha and leafnode-1.6, the format of the groupinfo file changed and some files moved to other places. To update correctly, do a "make update" as root after you have successfully completed "make install". This will reformat the groupinfo file and move the other files into the correct places. The old groupinfo file will be stored as groupinfo.old just in case something goes wrong. You also have to change your main config file manually as described in the section "Update from versions < 2.0".

4. FILES AND DIRECTORIES

Leafnode puts its files in three separate directories: The spool directory, the library directory, and the binaries directory. All directories can be changed at compile time.

In the spool directory you find the stored news, the active file and some other short-lived configuration file. It defaults to /var/spool/news and can be changed at compilation time. There are some special directories here; see the leafnode(8) man page.

The library directory contains long-lived configuration files. It defaults to /etc/leafnode.

The binaries directory, /usr/local/sbin by default, contains the executable programs applyfilter, texpire, fetchnews and leafnode.

The user directory, /usr/local/bin by default, contains the newsq program.

4.1. The main configuration file

The main configuration file contains settings important for all programs of the leafnode suite. It defaults to /etc/leafnode/config, but several configuration files can be used at once by employing the -F switch of the programs.

The file contains two mandatory and a number of optional parameters.

NOTE: Global options must come first, server-specific options below the server = lines.

Mandatory parameters

server = news02.bigprovider.com

expire = 5

Server-specific optional parameters

To configure interaction with the server somewhat, you can change the behaviour of fetchnews by setting several server-specific optional parameters. They have to be specified directly after the corresponding "server" statement. A new server statement in the config file will also allow new optional parameters.

username = myname
password = mypasswd

timeout = 30

nodesc = 1

server.name does not process LIST NEWSGROUPS news.group.name correctly: use nodesc

port = 8000

General optional parameters

hostname = host.domain.country

create_all_links = 1

groupexpire = a.news.group 7

groupexpire = *.announce 30

filterfile = /etc/leafnode/filters

timeout_short = 2
timeout_long = 5

timeout_active = 90

debugmode = 260

4.2. The filter file

5. LOCAL NEWSGROUPS

From version 2.0 on, Leafnode is able to handle local newsgroups. Local newsgroups are groups that exist only on your local server but not on upstream servers.

To create a local newsgroup, you have to think of a newsgroup name which should not exist on any of your upstream servers. It is therefore a good idea to start a new top-level hierarchy. You should also make up a description for your newsgroup.

If you choose a newsgroup name which exists already on an upstream server, the newsgroup is not treated as a local one.

Next, you write the name, status and description into the file /etc/leafnode/local.groups using your preferred text editor, which must not mangle TABs to SPACEs. The file should consist of lines in the format

news.group.name<TAB>status<TAB>description

The first column of each line is taken as the newsgroup name; the second as the status (y for "may be posted to", m for "moderated" -- see below for moderator configuration), the third column of the line is interpreted as description. For example, to set up a newsgroup "local.leafnode" which deals with Leafnode's internals, you would put a line

local.leafnode<TAB>y<TAB>Local leafnode user group

5.1 MODERATED NEWSGROUPS

From version 2.0 on, Leafnode also handles moderated newsgroups. These are newsgroups which only certain people may post into, and articles which a regular (non-moderator) user posts, are mailed to the moderator.

External moderated newsgroups are handled by the upstream servers.

To configure the moderators for local moderated groups, an INN-compatible "moderators" file is used. It is named /etc/leafnode/moderators and has the following format:

pattern:moderator

Empty lines and lines that start with a hash character (#) are considered comment lines and are ignored.

The pattern is a normal wildmat pattern, which means that it's a SHELL pattern, the most useful character is the asterisk (*) which matches any sequence of characters. The moderator given on that line applies to all groups that the pattern matches. Should multiple lines match a group, the first match in the file "wins", the second and subsequent patterns are not taken into account. So put the more special patterns first.

The moderator is a regular internet mail address, with the exception that the first occurrence of "%s" is replaced with the newsgroup name, with dots (.) converted to dashes (-) for INN compatibility.

Example:

# This is a comment line
#
# special moderator, hans@example.com is mailed all articles for
# local.special
local.special:hans@example.com
# moderator for all other local groups:
local.*:egon+%s@example.net
# that means:
# mail postings for local.test to egon+local-test@example.net
# mail postings for local.video to egon+local-video@example.net
#
# Note: if you put local.special below local.*, local.special will be
# ignored.

6. DEINSTALLATION

If you want to uninstall leafnode (e.g. because you want to replace it with another newsserver) and have a Makefile available, you can achieve this by doing "make uninstall". This will remove the executables, newsspool and configuration directories. If you only want to remove the executables and leave the spool and configuration directories untouched, use "make uninstall-bins" instead.

There is no need to uninstall an old version of leafnode before updating.

7. FREQUENTLY ASKED QUESTIONS

See the file FAQ.

8. COPYRIGHT

Leafnode versions through 1.4 were written by Arnt Gulbrandsen <agulbra@troll.no> and are copyright 1995 Troll Tech AS, Postboks 6133 Etterstad, 0602 Oslo, Norway, fax +47 22646949.

Leafnode versions 1.5 and higher were written by Cornelius Krasel <krasel@wpxx02.toxi.uni-wuerzburg.de> and are copyright 1997-2000.

Major modifications were made by
Randolf Skerka <Randolf.Skerka@gmx.de>
Kent Robotti <robotti@erols.com>
Markus Enzenberger <enz@cip.physik.uni-muenchen.de>
Matthias Andree <matthias.andree@gmx.de>
Jörg Dietrich <joerg@dietrich.net>
Stefan Wiens <s.wi@gmx.net>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The getaline() and getline() routines are Copyright (c) 2000 Matthias Andree <matthias.andree@gmx.de>. They are licensed under the GNU Lesser General Public License (GNU LGPL) 2.1. A copy of the LGPL is provided in the file COPYING.

The wildmat() routine is Copyright (c) 1986-1991 Rich $alz <rsalz@osf.org>. The original copyright notice, taken from INN 2.2.2, follows:

Redistribution and use in any form are permitted provided that the following restrictions are met:

1. Source distributions must retain this entire copyright notice and comment.
2. Binary distributions must include the acknowledgement ``This product includes software developed by Rich Salz'' in the documentation or other materials provided with the distribution. This must not be represented as an endorsement or promotion without specific prior written permission.
3. The origin of this software must not be misrepresented, either by explicit claim or by omission. Credits must appear in the source and documentation.
4. Altered versions must be plainly marked as such in the source and documentation and must not be misrepresented as being the original software.

THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The replacement vsnprintf() function was written by Theo de Raadt and is Copyright (c) 1997 under the following restrictions:

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.