Discussion:
Getting release 0.6 out of the door.
Alex Hudson
2009-08-23 17:39:11 UTC
Permalink
So, after some testing and patching today I think 0.6 is virtually there.

I've written some release notes which Lance has checked over:

http://www.bongo-project.org/Releases/0.6.0

I'm also running 0.6 on my personal mail domain, and I'm thinking that
I'll probably run it for a few days myself before feeling ok about
releasing it - but it looks reasonable at the moment.

I've merged all the patches into releng-0.6 in SVN, so if people want to
test, please test this branch:

http://svn.gna.org/svn/bongo/branches/releng-0.6

At the moment, this is basically what will get tagged as 0.6.0. Once
that happens, we can start messing around on trunk/ again. I'll try to
review patches which should get merged into future 0.6 releases too.

Cheers

Alex.
pfelt-Urct4Ww6//Pz1n+
2009-08-23 23:40:36 UTC
Permalink
I read the backlog from today and see that there are some questions on
queue and logging that i thought that i'd address. the whole queue system
is due for a massive rework as this (along with smtp) is about the last
major nims legacy that exists in the code.

each queue agent connects to the queue and registers a callback via a
tcp/ip port. as a new mail is dropped in the queue (usually due to smtpd)
it enters queue 0.

antispam and antivirus register at queue 0
smtpc registers at queue 7
rules registers at queue 3

The messages Lance saw (http://slexy.org/view/s20IHj5wEM) are all
completely normal (in most cases). every 10 seconds the queue wakes up
and scans the spool directory for mails that need to be sent. this should
generally only be messages that smtp can't deliver though if an agent dies
mail can get stuck. when the agent finds a message that should be
processed, it spawns a thread to begin the message through the queue.
basically it loops over all the registered agents and calls back to the
agent passing the message id. the queue agent then reads in the mail and
processes it however it needs to.

the "failed to aquire lock" messages are the system saying: "i've already
spawned a thread for this message and some agent already has it." i've
seen cases where the threads don't appear to exist any longer an so the
mail never gets unlocked to be processed. the only way to clear this is
to restart bongo*.

this will all hopefully be fixed for .7 when we rework the queue agent a bit.

hope this helps.
Post by Alex Hudson
So, after some testing and patching today I think 0.6 is virtually there.
http://www.bongo-project.org/Releases/0.6.0
I'm also running 0.6 on my personal mail domain, and I'm thinking that
I'll probably run it for a few days myself before feeling ok about
releasing it - but it looks reasonable at the moment.
I've merged all the patches into releng-0.6 in SVN, so if people want to
http://svn.gna.org/svn/bongo/branches/releng-0.6
At the moment, this is basically what will get tagged as 0.6.0. Once
that happens, we can start messing around on trunk/ again. I'll try to
review patches which should get merged into future 0.6 releases too.
Cheers
Alex.
_______________________________________________
Bongo-devel mailing list
https://mail.gna.org/listinfo/bongo-devel
Alex Hudson
2009-08-24 14:02:25 UTC
Permalink
Maybe it would make sense to have a meeting for 0.7. Originally I said
I wanted to do 0.7 for 1st Oct, which looks a bit ambitious but could
be possible. Can we do something like this Thursday, Friday or next
Monday?
Additionally to this, I'd like to declare next Monday to be web-ui-jfdi-day.

Thoughts? :)

Alex.
Alex Hudson
2009-08-24 14:39:29 UTC
Permalink
I was thinking primarily IRC, because hopefully a. there will be a few
of us, and b. we get someone to take notes for free :)

The 0.7 thing I was thinking being much more of a meeting than the jfdi
day, though, which would be more a kind of all day hack.

Cheers

Alex.
I am game for both things. If you need me on the call.
How do we meet?
Lance
pfelt-Urct4Ww6//Pz1n+
2009-08-25 04:55:25 UTC
Permalink
i'd love to have a formal meeting on 0.7. i think we have a lot of good
changes to make for then (more inline below). unfortunately given our
overlap, the first time i'll have available is next friday (~2 weeks) as i
work this friday :( is that too far out? if so i might be able to swing
something if we can keep it to an hour at a time during the noon hour (or
unless it is over the weekend of course).

next see below for comments on your todo list
* better bongo-manager;
i'm interested to see what you've worked up on this as this is a pretty
critical piece to getting agents running on multiple boxes. we had
originally thought there would be one bongo-manager per server that would
be running and that they would somehow communicate which had the store on
it.
* the queue rework, including some kind of mailq implementation and
much more visibility/control;
the queue changes we've discussed in the past will fix all of the agent
restarting issues as we'll be able to get rid of the callback system for
something like collection watches. this will be a fair amount of work
though as we'll have to re-do portions of all the queue agents currently
in use. also we'll be able to get a sane mailq by just doing a select out
of the db. the only worry i have is running into locking issues, though
the worry might be completely unfounded depending on how we are going to
implement the queue inside the store.
* direct message queues for agents to talk to each other [store?]
this is an interesting thought that we haven't really discussed yet (that
i can remember). agents talking to each other would be nice so that we
could get connmgr type functionality back in the future in a more sane
way. i suppose this could also be used for snmp stats though we had
thought of using bongo-manager for that.
* a better grasp on what works/doesn't work in the TLS area [connio];
tls... i've spent the better part of the last two hours looking at this
and i don't see any reason that this shouldn't work unless there is
something silly going on with the hostnames. i tried to connect to
bongo-test.info via the gnutls test tool and found that it denied me
access because "localhost" wasn't in the cert. i'm sending a separate
email to jur to see if he's got some odd nat going on that might
complicate things, however my tests to the domain he was having issues
with seem to run fine off my self signed cert.
* full-text indexing [store]
no real comment on this, other than i'm sure we need it for conversations
and we lost it when we dropped clucene?
It's not really an awful lot; and aside from that it's basically the web
bits which are the killer.
i totally agree here. if we could get UI functionality like what DF gave
us i think we'd rock as a full up system as we currently stand, but then
again i'm a bit biased :)
Alex Hudson
2009-08-25 07:39:18 UTC
Permalink
Hey Pat,
Post by pfelt-Urct4Ww6//Pz1n+
i'd love to have a formal meeting on 0.7. i think we have a lot of good
changes to make for then (more inline below). unfortunately given our
overlap, the first time i'll have available is next friday (~2 weeks) as i
work this friday :( is that too far out?
No, I think that's fine - I think it's important us two in particular
can make it, and it's not like there isn't plenty to get on with in the
meantime.
Post by pfelt-Urct4Ww6//Pz1n+
* better bongo-manager;
i'm interested to see what you've worked up on this as this is a pretty
critical piece to getting agents running on multiple boxes. we had
originally thought there would be one bongo-manager per server that would
be running and that they would somehow communicate which had the store on
it.
Yeah, and I think that continues to be the plan. What I'll try to do is
write something up more fully which goes into more details about this,
because although I some initial code it can still be changed.
Post by pfelt-Urct4Ww6//Pz1n+
the only worry i have is running into locking issues, though
the worry might be completely unfounded depending on how we are going to
implement the queue inside the store.
Well, originally, I was thinking we'd use the _system store and each
agent would create/be given two collections in there - incoming and
outgoing. Stuff would get moved into incoming, the agent would process
it and move it/rewrite it into outgoing, and the queue agent would take
it from there.

Using separate collections should mean that locking doesn't become an
issue at all, but if there are any problems it shouldn't be too hard to
solve them. I'm planning on adding in the last bit of store locking in
the next rev, but I don't really think it would affect how a queue might
work - the only thing we're missing from store in terms of functionality
at the moment is a store-to-store copy.
Post by pfelt-Urct4Ww6//Pz1n+
* direct message queues for agents to talk to each other [store?]
this is an interesting thought that we haven't really discussed yet (that
i can remember). agents talking to each other would be nice so that we
could get connmgr type functionality back in the future in a more sane
way. i suppose this could also be used for snmp stats though we had
thought of using bongo-manager for that.
Yeah, and this is also crucial for an improved bongo-manager in other
ways - e.g., bongo-manager needs to be able to tell agents to shut down,
and obviously if the agent is on another system you can't signal it :D

Briefly what I had in mind was an extremely simple publish/subscribe
system a little like the current WATCH setup. However, currently we
really don't deal with asynchronous data very well at all :(

The alternative I had in mind was essentially using a jabber-like
transport, which is naturally very good at this kind of thing. However,
I'm not sure I want to go down that road before 1.0 really.
Post by pfelt-Urct4Ww6//Pz1n+
tls... i've spent the better part of the last two hours looking at this
and i don't see any reason that this shouldn't work unless there is
something silly going on with the hostnames. i tried to connect to
bongo-test.info via the gnutls test tool and found that it denied me
access because "localhost" wasn't in the cert. i'm sending a separate
email to jur to see if he's got some odd nat going on that might
complicate things, however my tests to the domain he was having issues
with seem to run fine off my self signed cert.
Potentially I guess there is a gnutls versioning issue here. As an
aside; how did you create your cert - was it just what Bongo created on
install? (I've had suspicions for a while that our self-created cert
isn't quite right)
Post by pfelt-Urct4Ww6//Pz1n+
* full-text indexing [store]
no real comment on this, other than i'm sure we need it for conversations
and we lost it when we dropped clucene?
Actually, it's not for conversations - we basically handle that already
(I think - the code is there, it's extremely untested though). It's
simply for just searching stuff.

Currently we support enough search to do IMAP but it's not very smart
and doesn't index data. IIRC, though, the IMAP model is incompatible
with the full-text model anyway :(
Post by pfelt-Urct4Ww6//Pz1n+
It's not really an awful lot; and aside from that it's basically the web
bits which are the killer.
i totally agree here. if we could get UI functionality like what DF gave
us i think we'd rock as a full up system as we currently stand, but then
again i'm a bit biased :)
Indeed :-)

Cheers,

Alex.
pfelt-Urct4Ww6//Pz1n+
2009-08-25 11:44:34 UTC
Permalink
Post by Alex Hudson
Well, originally, I was thinking we'd use the _system store and each
agent would create/be given two collections in there - incoming and
outgoing. Stuff would get moved into incoming, the agent would process
it and move it/rewrite it into outgoing, and the queue agent would take
it from there.
Using separate collections should mean that locking doesn't become an
issue at all, but if there are any problems it shouldn't be too hard to
solve them. I'm planning on adding in the last bit of store locking in
the next rev, but I don't really think it would affect how a queue might
work - the only thing we're missing from store in terms of functionality
at the moment is a store-to-store copy.
ah yeah. it's coming back to me now yeah. i like this idea
Post by Alex Hudson
Potentially I guess there is a gnutls versioning issue here. As an
aside; how did you create your cert - was it just what Bongo created on
install? (I've had suspicions for a while that our self-created cert
isn't quite right)
it is the default cert generated by bongo-config. it seems to be a valid
cert according to the gnutls tools (openssl complained because it couldn't
find the ca though)

Loading...