NeuroWebLab backend
Your portal for collaborative open neuroscience on the web.
Come check out our vision here.
Small functions the apps share, exported from require('neuroweblab') (since 1.8.0):
-
escapeRegex(str): user text matched literally in a MongoDB$regex. -
jsonForScript(value),jsonForAttribute(value): JSON to write into an inline<script>(<,>,&, U+2028, U+2029 escaped), or into a single-quoted attribute ('too). -
toId(id): theObjectIdof a 24-digit hex string in either case, elsenull. -
catchErrors(handler, respond): wraps an async Express handler; an error answers 500{error: 'Server error'}(orrespond(res)) unless headers were sent, and is logged withreq.logorconsole. -
usernameFromCookie(sessionParser, cookie): the username of the session a WebSocket's cookie belongs to, ornull; readsusernameand the configuredusernameField. -
Saving a project (since 1.9.0,
project/save.js):canSaveProject(oldProject, user)(edit on at least one list),keepFieldsWithoutFilesEdit(newProject, oldProject, user, {sanitize})(the project's own fields needfiles: edit),saveRefusal()(the 403 body),ignoredChangesMessage(ignored)(the "Some changes (on …) were ignored" text),filesListIncomplete(newProject, oldProject, dropped)(a files list sent before every file was loaded), and since 1.19.0keepListsWithoutEditRights(newProject, oldProject, user, {rowKey, oldRows})(each list: longer needsadd, shorterremove, any other change, a move included,edit). -
Validating a project (since 1.10.0,
project/rules.js):plainText(str)(markup removed by DOMPurify over jsdom, text kept),sanitize(value)(every string of a JSON value, asplainText),isBlank(v),text(v),listOf(project, type),isBlankRow(blank rows by list),sortRows(type, rows, oldRows, {rowKey, isBlankRow})(stored, new and blank rows;rowKeyis the app's),shortnamePattern(a new shortname: letters, numbers, hyphens),droppedMessage(dropped)(the "Empty rows were not saved" text). -
The validation core (since 1.20.0,
project/validate.js):validateProject(data, {shortname, oldProject, owner, findMissingUsers, rules})({project, dropped, stored}or{error}: parse, sanitise, stored and blank rows, shortname, the three lists, field types, collaborator rows, duplicate layer names,anyone, users; the app'srulesgivecollaboratorUser,fileError,layerError,fieldsError,projectError, and optionallyrowKey,isBlankRow,checkStoredFields,checkStoredKeys),projectRowProblems(project, rules)(every row problem as if all rows were new, the same checks),duplicateLayerNames(list, stored),parseProjectData(data). -
Admin access (since 1.11.0,
admin/):createAdmins({cfgPath, logger})(an app's admins list from its cfg.json, withgetAdmins,setAdmins,isAdmin,requestUsername,requireAdmin(401/403 JSON) andadminPage({title, userPayload, loginPrompt})bound to it),requestUsername(req),sendResult(res, action, {logger})(an admin action's answer, 500 logged),adminLog(db, key, value, {logger, fields})(an entry inlog, returned with its_id; a log line only with alogger). -
Damaged projects (since 1.23.0,
admin/damaged.js,project/versions.js):projectVersion(doc)(a stored version's token: 20 hex digits of the sha256 of its JSON; the admin fixes refuse a project whose token changed since listed, 409) anddamagedProjects({sourcesOf, annotatedFiles, save, lockKeys, logger, logFields})→{findDamagedProjects(db), restoreFiles(db, {shortname, version, save, username}), fixCurrentVersions(db, {shortname, version, username}), isIncompleteSave}(saves that removed files still missing, several current versions; the fixes under the lock, logged with the previous document; the rows put back keep their names),BATCH_LENGTH(100),EXAMPLES(3). -
Embed (since 1.12.0,
embed/,routes/):mintWsTicket(db, scope)(an embed's WebSocket ticket, its scope stored as given, 6 h),findWsTicket(db, token)(the ticket, ornull),ensureTicketIndex(db, {logger})(token_1and the TTL indexexpiryDate_1; never rejects),WS_TICKET_COLLECTION,WS_TICKET_DURATION,projectEmbedRoute(adapter)(GET /project/:projectName/embed: 404s, the app's refusal, or 302 to the first file's embed) andfirstSource(project). -
Unknown URLs and errors (since 1.13.0,
routes/notFound.js):notFound({view})(the 404 catch-all: JSON, the error page or plain text),errorHandler({view, isDevelopment, onError})(the error's status as JSON or the error page; a 5xx message hidden outside development;onError(err, req, res)logs, by defaultconsole.errorfor a 5xx),wantsJSON(req),wantsPage(req),displayPath(req). -
What a visitor may see (since 1.14.0,
project/visible.js,accessControl/):visibleProject(project, user)(a copy: the collaborators other thananyoneandmodifiedBy, the annotations and the files hidden unless the visitor may view them; missing lists empty),isAnyoneRow(row),collaboratorKey(row)(userID,nicknameorusername), andAccessControlService.isLoggedOut(user),AccessControlService.getEffectiveAccessLevel(project, user, accessType)(the higher of the own row andanyone; annotations at mostviewwhen logged out). -
Label sets (since 1.15.0,
routes/labelSets.js):listLabelSets(dir, {first, logger})(the*.jsonfiles of a folder as[{name, source}], by file name withfirstbefore the others; an invalid file skipped and logged) andlabelSetsRoute(dir, {first, logger})(GET /api/getLabelsets, as JSON). -
Project versions (since 1.16.0,
project/versions.js):findCurrentProject(db, shortname, {projection})(the current version, the newest_idof several;nullfor none or a non-string shortname),saveProjectVersion(db, doc, {by, keepModified, afterSave})(inserts a new version, then marks the othersbackup: true;afterSave({db, previous, version})after both writes),CURRENT,NEWEST_FIRST. The collection is the configuredprojectsCollection. -
Paged files and the project listing (since 1.17.0,
routes/projectFiles.js,routes/projectList.js):projectFilesRoute({db, canView, noViewStatus, entries})(GET /project/json/:projectName/files?start&length[&names=true]:start/lengthrequired, clamped to 0 and cut to the list;entries(req, project, slice, {names})maps the raw slice),projectListRoute({db, canView, row, projection})(GET /project/json?page: every current project read, filtered bycanView, thenPROJECTS_PER_PAGE(20) per page asrow(req, project)); errors go tonext. -
The project lock (since 1.18.0,
project/lock.js):withLock(keys, fn)(runsfnonce every earlier holder of each key has finished, FIFO per key; several keys deduplicated and sorted, so no deadlock; a throw releases every key; a bad key is aTypeError),isLocked(key),projectKey(shortname)(project:<shortname>),withProjectLock(shortname, fn). One process, one copy ofneuroweblabper app; no timeout. -
The project routes' access (since 1.21.0,
routes/projectAccess.js):visitor(req)(the username, ornulllogged out; sessions and tokens),noViewStatus(req)(401 logged out, 403 logged in), and five factories, errors tonext:projectPageRoute({db, canView, render})(GET /project/:projectName: 404Project Not Found, the refusalNot authorized to view project),projectSettingsRoute({db, canView, blankProject, render})(an unknown project isblankProject(req),render(req, res, project, {isNew})),projectJsonRoute({db, canView, visible})(404{error: 'No project <name>'},?var=a/bon own properties),projectSaveRoute({lockKeys, save})andprojectDeleteRoute({db, lockKeys, canRemove, remove})(401 before the lock; the delete's 404 and 403 inside it,{success: false, error, message});projection(default{_id: 0}) on the four that read. -
A user's projects and public fields (since 1.22.0,
routes/userProjects.js,user/):userProjectsRoute({db, canView, rows, answer})(GET /user/json/:userName/projects?[start][&length]: the user's current projects, owner or a row byuserID,usernameornickname, filtered bycanView, then sliced;start/lengthoptional, a value that is not digits counts as not given;answer(await rows(req, page)); errors tonext),publicUser(doc, usernameField)(the username field andPUBLIC_USER_FIELDS,name,avatarURL,url,joined, own properties only: an allow-list) andpublicLoggedUser(reqUser, usernameField)(nulllogged out, elsepublicUserplususername).
They live in util/ (and auth/sessionUser.js, project/, admin/, embed/, routes/, user/), each with its spec next to it.
All NAAT repos (NeuroWebLab, nwl-components, BrainBox, microdraw) run the current Node LTS, declared in each repo's .nvmrc. The workflow .github/workflows/yearly-node-upgrade.yml runs on the 1st of every month. Each run re-enables the workflow through the GitHub API, so the schedule isn't disabled after 60 days without commits; outside November that's all it does ("keepalive only"). The issue is still yearly: in November it finds the newest Node LTS on nodejs.org, checks that the unofficial linux-x64-glibc-217 builds exist for it (the production servers run CentOS 7), and, if that LTS is newer than .nvmrc, opens an issue labelled node-upgrade with the checklist for the four repos and both servers (deadline: 1 May). It doesn't open a second issue while one is still open, and it only logs a line when the repos are already on the newest LTS.
- Who gets assigned: set the repository variable
NODE_UPGRADE_ASSIGNEESto a comma-separated list of GitHub usernames (Settings → Secrets and variables → Actions → Variables). Assignees must be collaborators of this repo. If the variable is empty, the issue is opened unassigned and says so. - Run it by hand: Actions → Yearly Node upgrade → Run workflow. A manual run does the Node check in any month ("Run the Node check" is ticked; untick it for a keepalive only). "Dry run" is ticked by default: the log shows the issue it would open, and nothing is created. Untick it to open the issue now.
- Locally:
NODE_CHECK=true DRY_RUN=true node .github/scripts/yearly-node-upgrade.mjs(addNVMRC=<file containing an older major>to see the issue text).
