Currently we 'or' combine all objects of a type (from/to/what/when)
regardless of group, so we only keep a single list of all objects.
Since we want to introduce different logic (and/invert) we want to keep
the configured group structure. This patch does this, without changing
the current matching logic (still all 'or'-ed).