[Structured Citations] - Part 2 enhancements

[Tribunal33]

Description:

• Include specific context details that have been discovered that surround this developer task.

During testing some improvements were suggested to facilitate a more fluid user experience that were not in scope for the initial feature. It, additionally, addresses some requirements that were unclear.

• [ ] Async functionality with jobs processing. (Currently there is a noticible slow down in response times for other users navigating the site when a large job like this is being processed) Will be addressed here: Queue running via CLI worker does not trigger plugin hooks #9345
• Arxiv does not need URL validation - arXiv:2401.12345 is valid. (Unclear what should be validated for this field)
• Increase the size of the citation input field. Feels too small for the amount of information and spacing between citations.
• Disable Metadata lookup if it is enabled site-wide in settings for author.
• Allow author to manage (add and delete) citations in the workflow publication tab.
• Editing citation should have more informative validation warnings for Type and source type or have auto complete or dropdown list
• Editing citation fields should limit the max number of characters for these free text fields. (Security general best practice is to limit user input) A GitHub discussion should be opened, so that we all discuss it and find a general solution and a way to go forward
  https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html#:~:text=When%20designing%20regular%20expression%2C%20be,Allow%20List%20Regular%20Expression%20Examples
• Should limit to 100 citations at a time (for both security and navigating the site while it is processing is sluggish). Similar to max text length except for separated based on new lines. You shouldn't be able to add more until it is done processing the first batch. The problem is with job processing that will be addressed here: Queue running via CLI worker does not trigger plugin hooks #9345

Steps to test the changes:
These steps can include user interactions, database queries and API requests
Precondidtion: Workflow Settings > Submission > Metadata > References > Enable References > Enable Metadata lookup.

• Not a single set of steps to cover all the extra requested improvements. (I could break it out into more tickets if need be)

1. Async - Process 100+ citations then swap to another browser and navigate the site.
2. After processing the citation select Edit and add the Arxiv reference without a URL.
3. After processing the citation select 3 dots > Edit and there are a few fields which difficult to understand what is expected. Like type and source. Kept getting an warning when I added something wrong.
4. Add whatever the MaxLength +1 of the field is and observe the warning and is won't save the form.

Solution: What the proposed solutions are and additional affected areas
Solutions described above.

What application are you using?
OJS 3.6

Additional information
Please add any screenshots, videos or other information we can use to understand this dev task report.

• SourceType values : (enum: [book series,conference,ebook platform,journal,metadata,other,repository]
  Field references for Type field controlled vocab

Enum type

crossref	openalex	OJS
book	book	book
book-chapter	book-chapter	book-chapter
book-part		book-part
book-section		book-section
book-series		book-series
book-set		book-set
book-track		book-track
component		component
database		database
dataset	dataset	dataset
dissertation	dissertation	dissertation
edited-book		edited-book
	editorial	editorial
	erratum	erratum
grant	grant	grant
journal		journal
journal-article	article	journal-article
journal-issue		journal-issue
journal-volume		journal-volume
	letter	letter
	libguides	libguides
monograph		monograph
other	other	other
	paratext	paratext
peer-review	peer-review	peer-review
posted-content		posted-content
	preprint	preprint
proceedings		proceedings
proceedings-article		proceedings-article
proceedings-series		proceedings-series
reference-book		reference-book
reference-entry	reference-entry	reference-entry
report	report	report
report-component		report-component
report-series		report-series
	retraction	retraction
	review	review
standard	standard	standard
	supplementary-materials	supplementary-materials

---

PRs:

• pkp/pkp-lib#11902 Structured citations improvements #12000
• pkp/pkp-lib#11902 Structured citations improvements ui-library#733
• pkp/pkp-lib#11902 submodules updates ojs#5169 (only submodule update)
• pkp/pkp-lib#11902 submodules updates omp#2161 (only submodule update)
• pkp/pkp-lib#11902 submodules updates ops#1131 (only submodule update)
• I11902 fixes #11915
• I11902 fixes ui-library#718

UI fix, to track the processing status:

• pkp/pkp-lib#11902 consider processingStatus in citation manager ui-library#741
• pkp/pkp-lib#11902 add citation processing statuses constants to PKPDa… #12052
• pkp/pkp-lib#11902 submodules updates ojs#5186 (only submodules updates)
• pkp/pkp-lib#11902 submodules updates omp#2171 (only submodules updates)
• pkp/pkp-lib#11902 submodules updates ops#1139 (only submodules updates)

Fix 2: copy citations on publication versioning, allow citations editing, deleting, reprocessing when publication published

• pkp/pkp-lib#11902 copy citations on publication versioning, allow citations editing, deleting, reprocessing when publication published #12066
• pkp/pkp-lib#11902 submodule update ojs#5197 (only submodules updates)

[bozana]

A comment on the last point:

Should limit to 100 citations at a time (for both security and navigating the site while it is processing is sluggish). Similar to max text length except for separated based on new lines. You shouldn't be able to add more until it is done processing the first batch.

This will not change anything regarding the sluggish processing. The jobs will be there, either immediately or later, and they will need to be processed (making the responses slow now or later).

I think we need to have general policy regarding security and limitations of journal manager, editor, author, etc. input fields: maybe to double check this with infrastructure team and UI team, and also define a strategy so that everybody knows how to proceed, before doing the changes here?

[bozana]

I will note here as well:
If someone wants to run jobs async, it will need to use the worker deamon, s. https://docs.pkp.sfu.ca/admin-guide/en/deploy-jobs. If users are using the default setting -- to run jobs within a request -- they will experience the delay of the response. So this is not quite the problem associated with this issue, with structured citations jobs, but jobs run within a request in general.
Lets see if Touhidur will have a suggestion...

[Tribunal33]

A comment on the last point:

Should limit to 100 citations at a time (for both security and navigating the site while it is processing is sluggish). Similar to max text length except for separated based on new lines. You shouldn't be able to add more until it is done processing the first batch.

This will not change anything regarding the sluggish processing. The jobs will be there, either immediately or later, and they will need to be processed (making the responses slow now or later).

I think we need to have general policy regarding security and limitations of journal manager, editor, author, etc. input fields: maybe to double check this with infrastructure team and UI team, and also define a strategy so that everybody knows how to proceed, before doing the changes here?

My solution is to limit the number of citations to a smaller number (100 for example) or even match the config setting for Job_runner_max_jobs. The Add button would be disabled until it is done processing. Currently when there are less than 50 citations being processed the sluggish behavior doesn't last long (5-10s). There is still some slow downs but briefly. If we restricted it then this would mitigate a prolonged sluggish behavior. Also, note, this is just a suggestion if you don't think it's worth the effort then you can say no thanks. I won't feel bad. I try to represent the user but we can be needy 😄 .

[GaziYucel]

Arxiv does not need URL validation - arXiv:2401.12345 is valid. (Unclear what should be validated for this field)

The correct value will be without the arxiv: prefix, e.g. 2401.12345.

Validation will be with a regex, thus

"arxiv": {
    "type": "string",
    "description": "Arxiv id.",
    "multilingual": false,
    "apiSummary": true,
    "validation": [
        "nullable",
        "regex:/^\d{4}\.\d{4,5}(v\d+)?/i"
    ]
}

Handle also does not need URL validation. Same structure implemented as for arxiv.

The correct value will be without the hdl: prefix, e.g. 11366/1249.

"handle": {
    "type": "string",
    "description": "Handle id.",
    "multilingual": false,
    "apiSummary": true,
    "validation": [
        "nullable",
        "regex:/^\d+\/[a-zA-Z0-9\-_]+/i"
    ]
},

Doi also does not need URL validation, see schema.

"doi": {
    "type": "string",
    "description": "The DOI itself, e.g. 10.1234/5a6b-7c8d.",
    "multilingual": false,
    "apiSummary": true,
    "validation": [
        "nullable",
        "regex:/^10[.][0-9]{4,}[^\\s\"\\/<>]*\\/[^\\s\"<>]+/i"
    ]
},

Editing citation should have more informative validation warnings for Type and source type or have auto complete or dropdown list

Enum and dropdowns implemented for both.

Disable Metadata lookup if it is enabled site-wide in settings for author.

This is currently being discussed in Development channel in Mattermost. It's unclear how this should be implemented.

Pull Request

These are all fixed here #11915

[bozana]

Hi @Tribunal33, I think the following tasks we decided not to tackle here, because they need other and general attention:

• Async functionality with jobs processing (it is a general issue that cannot be solved just here, we will see what is the outcome of our discussion on MM)
• limit the max number of characters (create a discussion to find a generic way how to proceed with it in future)
• Should limit to 100 citations at a time -- the samve as "Async functionality with jobs processing".
  Correct?

[bozana]

Hi @GaziYucel, could we allow the validation of both in the UI, for the user: the URL (https://arxiv.org/abs/1706.03762) as well as name prefix (arXiv:1706.03762) but we would only save the identifier (1706.03762) ?

[bozana]

The job processing problem discovered here will be improved together with this issue: #9345 (s. the discussion in MM).

[GaziYucel]

Hi @GaziYucel, could we allow the validation of both in the UI, for the user: the URL (https://arxiv.org/abs/1706.03762) as well as name prefix (arXiv:1706.03762) but we would only save the identifier (1706.03762) ?

Resolved here: #11915