Skipping subtrees

[akirak]

I sometimes need only an overview of tasks in a file. For example, I might need only top-level headings of a file. org-agenda has org-agenda-todo-list-sublevels option (which is actually the default). Would you add an option to skip subtrees in org-ql (and org-agenda-ng)?

[alphapapa]

Yeah, that should be easy enough.

[alphapapa]

There, now you can do this:

(org-ql (org-agenda-files)
  (level 1))

;; or

(org-ql (org-agenda-files)
  (level <= 2))

Does that do it? :)

[akirak]

That is another feature I wanted to request, but what I meant was skipping to the next heading of the same or upper level when you reach a matching heading. For example, you may have a file that contains NEXT headings at multiple levels but you may be interested in only the root headings of those NEXT trees. Is this really that easy in this implementation?

For example, is it possible to retrieve only a, b, and c.1 in the following example?

* NEXT a
** NEXT a.1
** TODO a.2
* NEXT b
* TODO c 
** NEXT c.1

[alphapapa]

Well, I think I understand what you mean now. Although, do you mean that the root heading would also have a NEXT keyword, or that it would have direct child headings with NEXT keywords?

Is this really that easy in this implementation?

I don't think it's easy, no. :) We can probably figure something out...

[akirak]

Please reopen this issue.

[alphapapa]

I guess what's needed to support this is to call a different next-heading function when the predicate matches.

[akirak]

I just wondered if org-ql could be used to scan project as defined in this article. Actually, doing such scanning can be both complex and slow, so not supporting it at all might be a good idea.

I guess what's needed to support this is to call a different next-heading function when the predicate matches.

Yeah, maybe you can add support for a custom skipping function as a filtering predicate, or maybe you don't need to support it.

[alphapapa]

I have a prototype almost working, but due to the way the Org next-heading and end-of-subtree functions work, it's not as simple as it would seem...

[alphapapa]

With that branch, you can do this:

#+BEGIN_SRC elisp
  (org-ql (current-buffer)
    (todo "NEXT")
    :match-next-fn #'org-ql--outline-next-heading-same-level)
#+END_SRC


* NEXT a
** NEXT a.1
** TODO a.2
* NEXT b
* TODO c 
** NEXT c.1

asdf

And get these results:

((headline
  (:raw-value "a" :begin 138 :end 146 :pre-blank 0 :contents-begin 147 :contents-end 147 :level 1 :priority nil :tags nil :todo-keyword
              #("NEXT" 0 4
                (fontified t line-prefix "" wrap-prefix
                           #("* " 0 2
                             (face org-indent))
                           face org-todo))
              :todo-type todo :post-blank 1 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 138 :title
              (#("a" 0 1
                 (:parent #1)))))
 (headline
  (:raw-value "b" :begin 171 :end 179 :pre-blank 0 :contents-begin 180 :contents-end 180 :level 1 :priority nil :tags nil :todo-keyword
              #("NEXT" 0 4
                (fontified t line-prefix "" wrap-prefix
                           #("* " 0 2
                             (face org-indent))
                           face org-todo))
              :todo-type todo :post-blank 1 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 171 :title
              (#("b" 0 1
                 (:parent #1)))))
 (headline
  (:raw-value "c.1" :begin 190 :end 201 :pre-blank 0 :contents-begin 202 :contents-end 202 :level 2 :priority nil :tags nil :todo-keyword
              #("NEXT" 0 4
                (fontified t line-prefix
                           #("*" 0 1
                             (face org-hide))
                           wrap-prefix
                           #("*** " 0 1
                             (face org-indent)
                             1 4
                             (face org-indent))
                           face org-todo))
              :todo-type todo :post-blank 1 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 190 :title
              (#("c.1" 0 3
                 (:parent #1))))))

[alphapapa]

I think that might do it. What do you think?

I'd like to benchmark it before merging it to master, to make sure that it doesn't make normal matching slower. I do want to keep speed up as much as possible, because I have some large Org files that I search with it, and little slowdowns add up.

I also might simplify the API. Maybe it could have a query like:

(without-children (todo "NEXT"))

Or a :skip-children t argument. What do you think?

[akirak]

I think that might do it. What do you think?

It seems to work. Thank you.

I thought of adding an option like :skip-children as an analogy to the option in org-agenda.

Is it possible to add the feature as a predicate like (without-children ...)? org-super-agenda has :children option which has a different meaning, so I think you should avoid confusion with it.

[alphapapa]

Is it possible to add the feature as a predicate like (without-children ...)?

Actually, it may be. I thought of another approach: rather than calling a different next-heading function when a match is found, the predicate could skip the rest of the subtree itself, then back up one character, and then the call to outline-next-heading would go to the heading after the subtree.

org-super-agenda has :children option which has a different meaning, so I think you should avoid confusion with it.

Good point! I'm glad you are so familiar with my other projects. :)

[alphapapa]

Ok, here's what I've come up with for a query API:

(org-ql (current-buffer)
  (skip-children (todo "NEXT")))

However, we could also do it like this, which might be better:

(org-ql (org-agenda-files)
  (when (todo "NEXT")
    (skip-children)))

It's a bit more verbose, but it feels more like elisp. Then again, I almost feel like it implies that it just skips things, rather than returning results. So maybe something like this would be better:

(org-ql (current-buffer)
  (skipping-children (todo "NEXT")))

Or:

(org-ql (current-buffer)
  (without-children (todo "NEXT")))

Or we could just do it as an argument, like:

(org-ql (org-agenda-files)
  (todo "NEXT")
  :skip-children t)

Which do you think is best?

[akirak]

I personally prefer a query API that conforms to set theory (and SQL), so using and and or looks more natural than when. However, Lisp hackers may have a different opinion.

(skipping-children ...) would be more flexible than :skip-children, but I am not sure if it is necessary. I thought :skip-children (actually I thought of :skip-subtrees at first) would be enough, but is it possible to add support for complex expressions like (or (skipping-children ...) ...) without sacrificing performance?

I also had the following ideas about this package, which were inspired by SQL:

• Limit the number of items returned (e.g. at most 5 items matching the predicate), which should be an option of org-ql
• Query at most one item, which should probably be implemented as another function

These features can allow retrieving some (not all) items faster, especially if no sorting is enforced. Sorting is not needed at all if the target file is sorted beforehand. Org-mode has a builtin function for sorting entries, and I saw a user who uses the feature. It would be better if these features were implemented with the same predicate API.

[alphapapa]

I personally prefer a query API that conforms to set theory (and SQL), so using and and or looks more natural than when

Single-clause and and when work the same way in lisp. I'm not proposing to reimplement and or when. So you could use either one.

is it possible to add support for complex expressions like (or (skipping-children ...) ...) without sacrificing performance?

Probably, but I haven't experimented with complex expressions like that.

Limit the number of items returned (e.g. at most 5 items matching the predicate), which should be an option of org-ql

That would be easy, I think.

Query at most one item, which should probably be implemented as another function

Why a separate function, rather than a limit of 1?

These features can allow retrieving some (not all) items faster, especially if no sorting is enforced. Sorting is not needed at all if the target file is sorted beforehand.

Sorting is already optional, of course.

Org-mode has a builtin function for sorting entries, and I saw a user who uses the feature.

org-sort can be used to sort trees, yes.

It would be better if these features were implemented with the same predicate API.

I'm not sure what you mean here.

[akirak]

Single-clause and and when work the same way in lisp. I'm not proposing to reimplement and or when. So you could use either one.

That would be better.

Probably, but I haven't experimented with complex expressions like that.

You probably don't have to do that.

Why a separate function, rather than a limit of 1?

It would be better if these features were implemented with the same predicate API.

I thought of a function that can be used like a generator/iterator. It stops scanning the target file(s) when it reaches a heading that satisfies a given condition. It would be much faster than scanning the whole file but returning only one item. Actually, all I need may be a function that moves the point to a heading matching the predicate.

[alphapapa]

The org-ql--filter-buffer function's loop could easily be extended to return a limited number of items.

[akirak]

Thanks. I'll take a look at it.

[alphapapa]

This is not exactly what is asked for here, but it seems relevant: 479824f

[akirak]

Thanks, it looks useful.

I think the feature I requested at the beginning of this thread can be somehow implemented by filtering the result of org-ql.

As for the last topic, it is probably not a scope of this package. I think I'll implement a generator if I really need it.

Thank you for the discussions above. I'll close this thread.

[alphapapa]

It's okay, I have some ideas and I want to think about this some more. :)

[akirak]

I implemented this feature in #89.