Skip to content

Conversation

@sjanssen2
Copy link
Contributor

@sjanssen2 sjanssen2 commented Sep 17, 2025

This the accompanying PR to qiita-spots/qiita_client#60

In short: I've extended fetch and push from only files to also directories. In test mode, objects can also be deleted. Depending on the route of API calls (through nginx = 8383 OR directly to tornado = 2117x), file delivery is handled differently to make us of nginx's much faster transfer capability

More elaborate:

  1. Resetting the DB of qiita is done by triggering the /apitest/reset/ endpoint (and also others). However, this only works properly if qiita master is addressed. Tests therefore do not speak against nginx (port 8383) but directly with the master instance (port 21174). However, when sending files, I relied on delivery through nginx. If requests go directly to master, the file content will be empty - and tests fail. With PR Tornado fetch file from central handler #3483 the according endpoint checks if a request was made through nginx (and then uses it's fast mechanism to deliver the file) or directly to the tornado instance (which now can also send the file in a much slower fashion). Should be fast enough for testing.
  2. Similarly, clients can now fetch directories. As I fear that misbehaving clients might trigger fetching the whole QIITA_BASE_DIR, I am limiting this to those directories that are "managed" by Qiita's DB as filetype directory.
  3. I realized that https://github.com/qiita-spots/qtp-sequencing/blob/4a325e7a75ca989eea9d9029acd02be2384a601e/qtp_sequencing/validate.py#L81 is directly deleting files in QIITA_BASE_DIR. Therefore, I added an according function to qiita_client and qiita, but only delete the file when using "filesystem" protocol OR qiita is run in test mode as I am too skeptical about providing an API endpoint to delete arbitrary files.

@coveralls
Copy link

coveralls commented Sep 18, 2025

Coverage Status

coverage: 91.943% (-0.2%) from 92.094%
when pulling dcedd02 on jlab:tornado_FetchFileFromCentralHandler
into 1b8b0c7 on qiita-spots:dev.

@antgonza
Copy link
Member

antgonza commented Dec 4, 2025

Yeah, I think that "hack" in qtp-sequencing was the first time I encountered the issue of running the same tests multiple times in the same environment and having leftover files - I think that's the same issue with my current PR in qtp-job-output-folder but I need to confirm. Now, before putting all these ideas together I actually encountered this issue again in qp-pacbio and fixed like this. I think this deserves a call; I'll email you to coordinate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants