[leafnode-list] Re: leafnode retrieving previously expired articles

[Matthias Andree]

Am 28.01.2014 12:23, schrieb Robert Marshall:
> Recently in my leafnode installation I've been seeing - from the news
> client end of things - lots of old articles appearing
> 
> In /etc/leafnode/filters I had a filter (maxage) so that articles over
> 50 days old weren't retrieved, I've tightened it up to 20 days and am
> still seeing the problem.
> 
> Typically I see in the log files a connectivity problem and the next
> time lots of old articles are retrieved so it looks as if the last
> retrieved article isn't set correctly after a timeout? eg

Robert,

sorry to see there are issues with leafnode-2.

Rather unlikely - we would have to write *higher* numbers on timeout.

> news.gmane.org: connecting to port nntp
>   trying:    address 80.91.229.13 port 119...
>   connected: address 80.91.229.13 port 119.
> news.gmane.org: connected (200), banner: "200 news.gmane.org InterNetNews NNRP server INN 2.5.1 ready (posting ok)"
> news.gmane.org: checking for new newsgroups
> news.gmane.org: found 0 new newsgroups
> found 0 articles in out.going.
> news.gmane.org: 0 articles posted
> timeout reading.
> news.gmane.org NNTP server disconnected or timed out while waiting for response
> timeout reading.
> news.gmane.org NNTP server disconnected or timed out while waiting for response
> timeout reading.
> news.gmane.org NNTP server disconnected or timed out while waiting for response
> timeout reading.
> news.gmane.org NNTP server disconnected or timed out while waiting for response
> gmane.comp.gcc.java.announce: considering 19464 articles 71 - 19534, using XOVER
> Unknown reply to XOVER command: 211 108 1 108 gmane.comp.emulators.virtualbox.announce

This is a hint that communication somehow goes out of synch, what
fetchnews sees is probably a response to a GROUP command, and what's
worse, it hints to bugs in timeout handling.

Without being able to prove it without a debug log that records a
timestamped NNTP trace, this looks like the responses from the server
are extremely slow, but arrive eventually.  Fetchnews should not be
processing responses that arrive after a timeout.

> news.gmane.org: cannot parse reply to "GROUP gmane.comp.gnome.apps.pan.user": "224 No articles in 71-19534"

This is another hint - 224 is the response code for XOVER.

I'll try to do some more debugging this week; for the time being, can
you try raising the timeout and timeout_client values in your
configuration (add them if not there, see man 8 leafnode for details)
to, say, 600 (seconds)?  If that gets us rid of "Unknown reply to ...
command", that would confirm a bug in timeout handling.

> I'm also seeing this with other news servers (eg news.individual.net)
> where I get this 
> 
> news.individual.net: gnu.emacs.vm.bug last seen article was 14946, server now has 10752 - 10760
> news.individual.net: switched upstream servers for gnu.emacs.vm.bug? upstream bug? 14946 > 10760

Are there hints to desynchronization, such as "Unknown reply|response to
command..." in the logs for news.individual.net?

> Maybe my ISP (virginmedia) are traffic shaping nntp traffic -
> fetchnews seems to be running a lot slower - unless there's a problem
> with my local db? Last year my fetchnews run was taking 2 minutes or
> less and now it's up to 10 mins or more.

Problems with the local "db" are effectively problems with the file
system.  Running texpire with an additional -r option should fix any
such problem.

> Also, when updating my leafnode2 installation, I tripped over the move
> of the spool from /var/spool/news to /var/spool/leafnode - there's no
> mention of this change in the README (or the README.html) and indeed
> they still mention the /var/spool/news location as being the default
> (eg for --enable-spooldir). It also should be mentioned in the text
> produced on a `make install` in install-data-hook

Uh. I probably did not mention that in the logs because either it is
supposed to pick up /var/spool/news if it's there, or because I thought
not to document that in an alpha version.  Thanks for letting me know.
You can run configure with options to override the spooldir to use the
old location explicitly, that should help.

Cheers
Matthias