Current store work

[Alex Hudson]

Hey everyone,

So, a status update on where we are with the Store.

A while ago I was talking about having a specified Query Language within
the store to replace the de-facto use of CLucene queries, since CLucene
may (probably will) go away at some point and we don't want to have to
parse their stuff because it's hard and not terribly well suited to use.

The code I wrote for the parser has been committed into SVN, and I've
documented the new query language somewhat here:

http://www.bongo-project.org/Documentation/Bongo_Query_Language

It could use some more examples, which I'll try to fill out in the
future, but the meat and bones of it is there - it's very simple. This
query language is what I want to use to drive SQLite behind the scenes a
lot of the time, because it has one really nice property: we can set
queries within Bongo, accept queries from clients, and then join them at
the hip by doing "& <bongoquery> <clientquery>".

The framework to convert this into SQL isn't really complete, but it's
beginning to hit SVN, and I blogged about it here:

http://www.alexhudson.com/blog/2008/08/01/#new-query-builder

This creates SQL along the lines of my discussion with Terry on the list
months ago: it's taken me ages to figure out the "right" way of doing
this, but I'm pretty pleased which how it's coming along at the moment.

I haven't wired this into the store yet - that's the next job - and I
also want to setup a testing data set to run some store commands against
to verify that the results are correct.

I'm hoping to get a good chunk of this done this weekend, by which point
I can test it a bit harder and then think about merging it onto trunk.

Cheers,

Alex.

[Terry Yapt]

Post by Alex Hudson
The framework to convert this into SQL isn't really complete, but it's
http://www.alexhudson.com/blog/2008/08/01/#new-query-builder
This creates SQL along the lines of my discussion with Terry on the list
months ago: it's taken me ages to figure out the "right" way of doing
this, but I'm pretty pleased which how it's coming along at the moment.

Hi Alex....

I have been reading blog and BQL "manual". It seems pretty cool.

I have a question and perhaps a silly question. In advance I must to
say I don't know how the store is organized and what is the
security/privacity level we want/require to access it, but looking at
the easier example on blog, I have a question:

What will be the result of this BQL ?

|! nmap.type 9999|

Who has rights to do this kind of SQL-Statements against the store ?

It seems a bit dangerous ????

Greetings....

PS: Do we have a document with the store organization explained ???

[Alex Hudson]

Hi Terry,

Post by Terry Yapt
I have been reading blog and BQL "manual". It seems pretty cool.

Thanks :)

Post by Terry Yapt
I have a question and perhaps a silly question. In advance I must to
say I don't know how the store is organized and what is the
security/privacity level we want/require to access it, but looking at
What will be the result of this BQL ?
|! nmap.type 9999|

This would just turn into a where clause, WHERE so.type != 9999,
ignoring the | characters.

If your | characters were intentional, then it wouldn't parse - it looks
like the start of WHERE (so.type != 9999) OR ( ... OR ...), but those
last two terms either side of the OR are missing. The parser detects
those failure modes and just won't accept it.

I think the root of your question, though, is "Can I set up BQL
statements which are potentially dangerous in some way?". I hope the
answer to this is roughly, "no".

In terms of the BQL queries themselves, the language is so simple and
the parser so simple that it's really quite safe. You can't have more
than fifty terms in the current implementation, so it's bounded in terms
of memory size, and it's really robust in terms of not accepting things
which aren't valid in terms of BQL before it even tries to turn things
into SQL.

Once you've got valid BQL, is it possible to turn this into unsafe SQL?
Again, it's designed not to be like that. You can't inject random SQL
statements: there are known operators, and everything else is basically
just data, so injection attacks will be impossible as far as I'm concerned.

It might be possible to write BQL which turns into a very hard to
compute SQL query which effectively makes SQLite work very hard. This is
something I'm going to explore when we're much closer to being finished:
I'm not sure if it will happen, and if it does it might be difficult to
guard against. However, anyone running BQL against the store has to be
authenticated anyway, so the only outside denial of service possible is
if we code queries into Bongo itself which have severe performance
problems. This is definitely something that I'm going to be coding for,
not because I think it's a security problem per se, but because I want
to detect where our queries are going slowly (taking longer than a
couple of seconds, really) and finding out how we can improve that.

Post by Terry Yapt
PS: Do we have a document with the store organization explained ???

In terms of the internal design, at the moment not really because I'm in
the process of redesigning bits. However, I am documenting things as I
go along, and I will try to work on the documentation a bit more before
I get any further!

Thanks,

Alex.

[Terry Yapt]

Post by Alex Hudson

Post by Terry Yapt
What will be the result of this BQL ?
|! nmap.type 9999|

This would just turn into a where clause, WHERE so.type != 9999,
ignoring the | characters.

The | characters was a mistake. Copy+Paste... Sorry.. :-(

Post by Alex Hudson
I think the root of your question, though, is "Can I set up BQL
statements which are potentially dangerous in some way?". I hope the
answer to this is roughly, "no".

That was the answer to the first part of the question. The second one
is if we have any way to restrict the access to a client to _only_ a
subselect of the whole mail system. In example: Client X can only
access to items.type 2. No access to the others.. Or Client Z can only
access to all items but not on a particular domain '[e-mail]'.

I mean a better granularity security when a client is accessing to the
mail system and NOT whole access to whole clients.

I hope I had explained my question a bit better.

Post by Alex Hudson
.... However, anyone running BQL against the store has to be
authenticated anyway,.....

Ok.... perhaps this answer the second part of my question...

Greetings....