Added fastq manager

[alexiswl]

• Comprises fastq objects and linked fastq objects through fastq sets
• Interacts with filemanager to store ingest ids for fastq objects
• Runs QC analysis on samples with Sequali
• Runs fileCompression information allowing easy transition between ORA and GZIP
• Runs NTSM services for easy comparison within a library and for comparison between libraries

Currently running at https://fastq.dev.umccr.org/schema/swagger-ui

Supersedes #819

Resolves #561

TODO

• Added toolkits for common microservices, upgraded build to use 'uv' #896

[alexiswl]

@mmalenic eventSourceConstruct tests failing at https://github.com/umccr/orcabus/actions/runs/14192658946/job/39760644268?pr=897#step:5:10

Not sure why, I haven't edited any of the event patterns or test

[mmalenic]

I added some tests for pattern matching in this PR: #920.

It's not the nicest-made test as it depends on the order of the buckets declared in the eventSource props in shared.ts. It should probably loop over all the rules and test them rather than hard-coding the index.

To fix it, you can change the index to 3 and 4, rather than 2 and 3, here and here, or add the ntsm bucket rule after all the other rules. I would also add the eventSourcePattern() to the ntsm bucket (like the other buckets) and possibly add it to the test:

orcabus/config/stacks/shared.ts

Lines 148 to 151 in 1eb4e02

	{
	bucket: ntsmBucket[stage],
	eventTypes,
	},

[alexiswl]

@mmalenic I tried putting ntsm at the bottom of the list both in shared.ts and in the filemanger.ts as well (after the if conditions).

In the tests ran a console.log on the event and pattern right before the line
expect(await testEventPattern(event, pattern)).toBe(false);

And got the following (event)

{
        version: '0',
        id: '17793124-05d4-b198-2fde-7ededc63b103',
        'detail-type': 'Object Created',
        source: 'aws.s3',
        account: '111122223333',
        time: '2021-11-12T00:00:00Z',
        region: 'ca-central-1',
        resources: [ 'arn:aws:s3:::org.umccr.data.oncoanalyser' ],
        detail: {
          version: '0',
          bucket: { name: 'org.umccr.data.oncoanalyser' },
          object: {
            key: 'example-key/',
            size: 0,
            etag: 'b1946ac92492d2347c6235b4d2611184',
            'version-id': 'IYV3p45BT0ac8hjHg1houSdS1a.Mro8e',
            sequencer: '617f08299329d189'
          },
          'request-id': 'N4N7GDK58NMKJ12R',
          requester: '123456789012',
          'source-ip-address': '1.2.3.4',
          reason: 'PutObject'
        }
      }

With (pattern)

{
        source: [ 'aws.s3' ],
        'detail-type': [
          'Object Created',
          'Object Deleted',
          'Object Restore Completed',
          'Object Restore Expired',
          'Object Storage Class Changed',
          'Object Access Tier Changed'
        ],
        detail: { bucket: { name: [Array] }, object: { '$or': [Array] } }
      }

Neither of these seem out of the ordinary, but then testEventPattern fails with Region is missing

[mmalenic]

That looks right. Do you have valid AWS credentials in your shell? This test needs access credentials as it uses the TestEventPattern API call on AWS.

There's no "local" way to test the event pattern - although there definitely should be (not sure why this is an API call).

[mmalenic]

If this is something that's annoying I think this test should be ignored by default. I don't think credentials should be required just to run CDK tests.

I guess this is more of an integration test and probably belongs somewhere else.

[alexiswl]

That looks right. Do you have valid AWS credentials in your shell? This test needs access credentials as it uses the TestEventPattern API call on AWS.

Okay very strange, tried on my pi5 with exported AWS creds from my ubuntu tower and I had the same result. Added in the environment variables AWS_REGION (and AWS_DEFAULT_REGION) and it works now! Because the event has the 'region' key I think this made the context of the error more confusing than what it was!