org-ql-block formatting messes up with org-agenda formatting

[yantar92]

org-ql-block introduces it's own formatting of the agenda items.
It may be fine assuming that more customization will arrive in future (I miss buil-in org-agenda-prefix-format), but org-agenda re-applies the formatting once you change the item todo state from agenda view.

The result does not look nice:

  DOING [#A] try this email |- Adam Porter <[email protected]>: Re: Asynchronous org-agenda-redo  2d ago                                                :COMMON:EMAIL:
            	get_started_org_mode:	 -0.9x	DONE [#A] try to play with options by removing DOUG_LEA_MALLOC conds email |- Eli Zaretskii <[email protected]>: Re: bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs) :COMMON::EMAIL:
            	get_started_org_mode:	 -0.9x	DOING [#A] benchmark agenda after 1 day running                                                            :COMMON::

The first line is the default org-ql's formatting, but it is changed to what is in the following two lines once I change the todo state.

[alphapapa]

I think there's nothing that can be done about this without implementing a compatible formatter. Of course, you can rebuild the agenda view after updating an entry, but I think that's the best we can do for now.

[mskorzhinskiy]

Just a thought, but may be one can patch org-mode itself, so it will have some kind of lever\button enabling and disabling this behaviour? I mean re-applying formatting after done.

[alphapapa]

Possibly. If you find a way, let me know. However, I would be reluctant to add such a workaround to this package, because it would likely be fragile.

[yantar92]

Hi. I have an idea how to make the built-in agenda formatting (partially) work with org-ql.
We can utilise the agenda's higher level formatting functions: org-agenda-get-day-entries and org-scan-tags. The implementation below works for my agenda view, though I did not test for border cases.

(defun org-ql-search-block (query)
  "Insert items for QUERY into current buffer.
QUERY should be an `org-ql' query form.  Intended to be used as a
user-defined function in `org-agenda-custom-commands'.  QUERY
corresponds to the `match' item in the custom command form.

Like other agenda block commands, it searches files returned by
function `org-agenda-files'.  Inserts a newline after the block.

If `org-ql-block-header' is non-nil, it is used as the header
string for the block, otherwise a the header is formed
automatically from the query."
  (let (narrow-p old-beg old-end)
    (when-let* ((from (pcase org-agenda-restrict
                        ('nil (org-agenda-files nil 'ifmode))
                        (_ (prog1 org-agenda-restrict
                             (with-current-buffer org-agenda-restrict
			       ;; Narrow the buffer; remember to widen it later.
			       (setf old-beg (point-min) old-end (point-max)
                                     narrow-p t)
			       (narrow-to-region org-agenda-restrict-begin org-agenda-restrict-end))))))
                (items (org-ql-select from query
                         :action 'element-with-markers
                         :narrow narrow-p)))
      (when narrow-p
        ;; Restore buffer's previous restrictions.
        (with-current-buffer from
          (narrow-to-region old-beg old-end)))
      ;; Not sure if calling the prepare function is necessary, but let's follow the pattern.
      (org-agenda-prepare)
      ;; FIXME: `org-agenda--insert-overriding-header' is from an Org version newer than
      ;; I'm using.  Should probably declare it as a minimum Org version after upgrading.
      ;;  (org-agenda--insert-overriding-header (or org-ql-block-header (org-ql-agenda--header-line-format from query)))
      (insert (org-add-props (or org-ql-block-header (org-ql-view--header-line-format
                                                      :buffers-files from :query query))
                  nil 'face 'org-agenda-structure) "\n")


      ;; build a org-agenda-skip-function, which is aware about the
      ;; org-ql query result. With this function, running normal
      ;; agenda processing is extremely fast.
      ;; I use `org-agenda-get-day-entries' + `org-scan-tags' here to
      ;; cover most possible query results.
      ;; However, org-ql-query may return much more complex results
      ;; than what is assumed by available agenda views.
      ;; Extra formatting may still be needed in those cases.
      (let* ((ans)
	     (item-markers (sort (mapcar (apply-partially #'org-element-property :org-hd-marker)
					 items)
				 #'<))
	     (item-buffers (delete-dups (mapcar #'marker-buffer item-markers)))
	     (date (calendar-gregorian-from-absolute (org-today))))
        (dolist (buffer item-buffers)
	  (cl-letf* ((item-markers-in-buffer (-filter (lambda (el) (equal (marker-buffer el) buffer))
						      item-markers))
		     ((symbol-function 'org-agenda-skip) 
		      (eval `(lambda ()
			       (when (or
				      (save-match-data
					(let ((marker (save-excursion
							(save-restriction
							  (org-back-to-heading 'invisible-ok)
							  (point-marker)))))
					  (unless (member marker ',item-markers-in-buffer)
					    (goto-char (1- (or
							    (-first (apply-partially #'< marker) ',item-markers-in-buffer)
							    (point-max)))))))
				      (funcall ,(symbol-function 'org-agenda-skip)))
				 (throw :skip t))))))
	    (setq ans (append (org-agenda-get-day-entries (buffer-file-name buffer) date) ans))))

        (let ((extra-ans (let ((-compare-fn (lambda (a b)
					      (equal (org-element-property :org-hd-marker a)
						     (get-text-property 0 'org-hd-marker b)))))
			   (-difference items ans))))
          (when extra-ans
            (setq item-markers (sort (mapcar (apply-partially #'org-element-property :org-hd-marker)
					     extra-ans)
				     #'<))
	    (setq item-buffers (delete-dups (mapcar #'marker-buffer item-markers)))
            
	    (dolist (buffer item-buffers)
	      (cl-letf* ((item-markers-in-buffer (-filter (lambda (el) (equal (marker-buffer el) buffer))
							  item-markers))
			 ((symbol-function 'org-agenda-skip) 
			  (eval `(lambda ()
				   (when (or
					  (save-match-data
					    (let ((marker (save-excursion
							    (save-restriction
							      (org-back-to-heading 'invisible-ok)
							      (point-marker)))))
					      (unless (member marker ',item-markers-in-buffer)
						(goto-char (1- (or
								(-first (apply-partially #'< marker) ',item-markers-in-buffer)
								(point-max)))))))
					  (funcall ,(symbol-function 'org-agenda-skip)))
				     (throw :skip t))))))
		(setq ans (append (with-current-buffer buffer (org-scan-tags 'agenda ".*" nil)) ans))))
            ;;(setq ans (append (-map #'org-ql-view--format-element extra-ans) ans))
            ))

        (let ((extra-ans (let ((-compare-fn (lambda (a b)
					      (equal (org-element-property :org-hd-marker a)
						     (get-text-property 0 'org-hd-marker b)))))
			   (-difference items ans))))
          (when extra-ans
            (setq ans (append (-map #'org-ql-view--format-element extra-ans) ans))))

	;; Calling `org-agenda-finalize' should be unnecessary, because in a "series" agenda,
	;; `org-agenda-multi' is bound non-nil, in which case `org-agenda-finalize' does nothing.
	;; But we do call `org-agenda-finalize-entries', which allows `org-super-agenda' to work.

	(->> ans
             ;; (-map #'org-ql-view--format-element)
             org-agenda-finalize-entries
             insert)
	)
      (insert "\n"))))

[alphapapa]

@yantar92 Thanks, but the purpose of this package is to not use the existing org-agenda code.

[yantar92]

I understand that your final goal is re-implementing org-agenda, but you are already relying on org-agenda-mode implementation to make org-ql-block work. I do not see why org-agenda's version of formatting cannot be used temporary until there is equivalent org-ql's version. The current version is simply inferior in comparison with org-agenda. Actually, it is one of the major things stopping me from using org-ql.

[yantar92]

FYI, I have recently found a re-implementation of agenda formatter [1]. It seems to be more complete than in org-ql. Though I do not think that it can be directly used, it might be a good resource to study.

[1] https://github.com/weirdNox/dotfiles/blob/master/config/.emacs.d/config.org#renderer

[alphapapa]

I understand that your final goal is re-implementing org-agenda, but you are already relying on org-agenda-mode implementation to make org-ql-block work.

That description puts the cart before the horse. org-ql-block was added as a convenience feature; the implementation is so simple that there seemed no reason not to add it. It does not rely on org-agenda; it enhances it. Of course, the risk of adding such a feature is that someone will think that it is the primary entry point and that everything else is supposed to revolve around it.

I do not see why org-agenda's version of formatting cannot be used temporary until there is equivalent org-ql's version.

This package was originally called org-agenda-ng. It began as an experimental reimplementation of org-agenda features. A reason for the reimplementation is that org-agenda's code is not very modular or reusable. This code you posted demonstrates that problem.

As well, I am learning that my code should rely less on Org code, not more. The more code in Org that this project depends on, the more breakage will happen, and the more compatibility workarounds will be required as Org changes. This is because Org does not make any promises to third-party packages about APIs. So I'm not seeking to make this project more reliant on code in org-agenda.el.

The current version is simply inferior in comparison with org-agenda.

org-agenda is a mature, complex system that has evolved over years from many contributors' work. org-ql is not intended to replace it in every way (perhaps eventually it might, but that goal is a distant one). The plans to enhance the formatter are already discussed in other issues here, like #44.

Actually, it is one of the major things stopping me from using org-ql.

I won't be merging code that doesn't align with the project's goals. If the code you posted works for you, that's great. You're welcome to continue using it. One of the goals of org-ql is to be more modular than org-agenda so you can more easily customize it.

[yantar92]

I won't be merging code that doesn't align with the project's goals. If the code you posted works for you, that's great. You're welcome to continue using it. One of the goals of org-ql is to be more modular than org-agenda so you can more easily customize it.

Got it.

org-ql-block was added as a convenience feature; the implementation is so simple that there seemed no reason not to add it

What do you think about the idea to implement something like org-ql-skip meant to be used as org-agenda-skip-function? The idea for org-ql-skip would be similar to the code above - using the results of org-ql-query to skip processing uninteresting tasks and speed up agenda generation. Then, anyone willing to use agenda's formatting can use org-ql-skip instead of org-ql-block without any need to introduce extra dependencies on org-agenda.

[alphapapa]

That idea and some PoC code is in the notes file: https://github.com/alphapapa/org-ql/blob/master/notes.org#c-org-agenda-skip-function

[yantar92]

That idea and some PoC code is in the notes file: https://github.com/alphapapa/org-ql/blob/master/notes.org#c-org-agenda-skip-function

Thanks! I did not know about it. Do you know about performance of the org-ql-skip-function in comparison with org-ql-query?

[alphapapa]

Did you not see the part that says:

I should benchmark it to see how much difference it makes, because all those fset calls on each heading isn’t free. But if a macro were used to rewrite the built-in predicates to their full versions, all of that could be avoided…

[yantar92]

Did you not see the part that says:

Sorry. My fault. I did not read the part after the code.

P.S. Thanks for your patient replies.

[yantar92]

I should benchmark it to see how much difference it makes, because all those fset calls on each heading isn’t free.

I just tried to benchmark the org-ql-block agenda, org-ql-skip agenda, and my own org-agenda-skip-function. The results are very strange (I measured the execution time two times for each)

org-ql-block: 26.778508461 sec and 19.764141983 sec
org-agenda-skip-function (default): 10.851595299 sec and 8.315862626 sec
org-ql-skip: 8.26962783 sec and 6.398630924 sec

[alphapapa]

Why are they very strange?

Anyway, I recommend using this macro for benchmarking: https://github.com/alphapapa/emacs-package-dev-handbook#bench-multi-lexical

Also, note the FIXME in the comment that explains that the proof-of-concept code probably doesn't work quite correctly as a skip function is expected to.