[leafnode-list] Re: trying to decide which is better

[Matthias Andree]

Eric S. Johansson schrieb am 2008-11-20:

> Matthias Andree wrote:
> 
> > If IMAP (rather than NNTPd) is an option, have a look at Dovecot and if
> > it meets your requirements.
> 
> we're already using dovecot but, I don't think it serves the need for
> a internal conference server, at lease, as far as I know.

Not directly - you'd likely have to add a mailing list underneath.

> > Leafnode uses what INN would call tradspool - files 1 2 3 4 5... in a
> > group directory. So, that would add some 80,000 files over the course of
> > the years. Quick file access can be achieved with modern file systems,
> > such as "dirhash" on UFS, "dir_index" on ext3, and most tree-structured
> > file systems should also be able to deal with this.
> 
> I think in my circumstances a group/[00-ff]/1 2 3... mechanism might
> serve my needs better.  If my hands weren't so crappy and C so hostile
> to speech recognition, I would try to fix it myself.

Unfortunately, exactly that is nontrivial to fix, since several places
of leafnode's hardcode the directory naming structure; that would have
to be consolidated first. I've been thinking about a better code
structure and perhaps C++ or something, but I have too little spare time
at the moment to make sweeping changes...

> (I've been living with RSI for the past 15 years)

That's a pity. When I had initial symptoms (aching forearms) while
writing up my diploma thesis (think master's), I went into the next
store and bought an ergonomic keyboard, reduced my writing efforts for a
couple of days, and insisted that my employer buy an ergonomic keyboard
for me, too. No symptoms for the past years -- so seems it was just in
time.

> I've turned on dir_index and it does make things slightly faster but
> EXT3 still runs out of steam at 32,000 files in the directory.  I'm
> probably going to have to create a loopback filesystem using JFS.

Is leafnode in some place the problem or that large directory?

(dir_index must have been on before the directory creation, else you
need to unmount and e2fsck -Df /dev/foobar to create and populate the
index, ext3 will silently fall back to linear searching if the hash
isn't there)

> > I'm happy to help with patches addressing non-scalability - I'd say do
> > some test runs on a test machine and see if it gets slow with "many
> > messages", let's profile and fix the slow parts.
> 
> well I am going to use leaf node for my first tests of the injection
> code.   I will let you know where things break (unfortunately, I'm
> really good at breaking things)

But OTOH, breaking things in interesting ways makes for the spice in
software development, right? ;-)

> > In what context exactly? I'm not quite sure what you mean.
> > If you need to replace the ID or References, you need to that outside
> > leafnode. If you POST, leafnode suggests a Message-ID that the client
> > can either pick up and use, or the client can generate its own, leafnode
> > doesn't care beyond some basic syntax and duplicate checks.
> 
> I'm generating my own ID and references.  When I post, I get this error message
> 
> nntplib.NNTPTemporaryError: 441 Invalid header "Message-ID:
> silly.test.1227217025392 at silly.com", article not posted

Ah, I see. Try adding angle brackets, that should do the job. Example:

    Message-ID: <silly.test.1227217025392 at silly.com>

This also holds for other headers (notably References:).

HTH

-- 
Matthias Andree