Add support for QNX OS

[dchordia]

Describe the feature

Hi,
I open this issue to facilitate discussion on adding cross-compilation support for QNX OS to aws-c-common library. As @sfod mentions lack of QNX testing infrastructure, can you share more on what kind of testing do you expect to be performed (build testing, unit testing, integration testing, etc.) and at what frequency (eg. on each PR, each tag release version).

Currently PR #1221 is tested on latest free non-commercial version of QNX. Out of 458 tests, 454 pass, 3 fail and 1 skipped. We usually upload tests for all QNX ported OSS packages at oss.qnx.com and are working on updating the dashboard with aws-c-common test results.

Based on merged PRs I see CI testing builds for different OS/architecture/compiler combinations and am open to adding support for QNX build testing there.

Use Case

QNX plans on adding support for popular OSS libraries so general public can directly make use of it. We have ported close to 100 libraries so far and continue adding support for more libraries including aws-c* libraries.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[sfod]

Thanks for the info, I'll bring it up to the team.

As a side question: I found ported aws-crt-cpp in oss qnx, what is its current status? Asking because it depends on aws-c-* libs (aws-c-common among them), so I assume some work still required.

[dchordia]

Sounds great, thanks!

Yes, aws-crt-cpp is ported for an earlier version few months back (v0.30.1), but it never got upstreamed. I am also working on upstreaming latest version (v0.34.1) and since different aws-c* are also it's dependencies, working on their upstreaming.

[sfod]

I am also working on upstreaming latest version (v0.34.1) and since different aws-c* are also it's dependencies, working on their upstreaming

It'll probably be more difficult than you anticipated. The main roadblock is providing event loop functionality on QNX -- various parts of aws-crt-cpp simply won't function without it.
We have PR for that (see awslabs/aws-c-io#669). This new event loop even works, but there are still some issues that need to be addressed and edge cases that need to be properly handled.

[dchordia]

Currently we have an ionotify based epoll on qnx-ports, though it only supports level-triggered epoll. I use this epoll for aws-c-io compilation. I read epoll_event_loop.c and am certain of level triggered epoll being good to use. I ran aws-c-io tests based on this build, but there was some weird behavior (like pipe_readable_event_sent_again_after_all_data_read_2loops being stuck for about an hour) which I am yet to investigate.

That's an interesting PR, kudos on the efforts! What kind of issues do you face on that implementation?

Thanks for the info, I'll bring it up to the team.

Did you get a chance to discuss with the team regarding this issue and PR #1221 ?

[sfod]

Did you get a chance to discuss with the team regarding this issue and PR #1221 ?

We started discussion, but as usual it requires some time to come to some decision. One important thing is that we were certain we have to finish work in awslabs/aws-c-io#669, which is not trivial. But since epoll is available (though we'll have to check if any changes are required to work with level-triggered epoll), this might reduce the required effort. We'll need to discuss this more.

pipe_readable_event_sent_again_after_all_data_read_2loops being stuck for about an hour

Maybe this part of my PR will help.

What kind of issues do you face on that implementation?

The one issue I'd appreciate your input on is described (though probably not very well, so let me know if it doesn't make sense, I'll try to remember more details) in this comment

[dchordia]

Sorry for the delayed response, I have been testing aws-c-io and facing error in different test cases. Some related to pipe, some TLS, some sockets, etc. I was looking into pipe test cases but unable to resolve them, namely the errors occur with one end closed and other end has some activity. I incorporate your PR change in but error persists. For some reason, error_code is never AWS_IO_BROKEN_PIPE in any calls to s_on_readable_events.

I will continue debug more tests and see if a bunch can be resolved with common fix, before raising a PR on aws-c-io

The one issue I'd appreciate your input on is described (though probably not very well, so let me know if it doesn't make sense, I'll try to remember more details) in this comment

Hmm, good question. I will have to investigate the particular behavior, and more about ionotify() with _NOTIFY_COND_EXTEN flag. iomgr.h specifies a bunch of flags for extended conditionals, some familiar some newer, though their use case strange.

[dchordia]

Hi @sfod, regarding your pointer to comment on _NOTIFY_COND_EXTEN, it is not really needed to be set in user-space applications. It is used by kernel internally to handle states. Not sure why it is in docs in first place, will ask to change it.

Regarding aws-c-io v0.21.4 build with QNX epoll, I tested it and out of 224 tests 208 pass now. There are also a few skips, namely ones which access badssl.com. Remaining are pipe related issues. I believe it would be good to open up a PR on aws-c-io for this. (I will take the latest main code, build and test once). What do you think?

[sfod]

We looked into this further, but can't prioritize QNX work right now. But we're still happy to help if you want to lead the effort though.

The main roadblock is we can’t run tests on QNX in our CI. I see that s2n has the same concern in aws/s2n-tls#5496 . You were asking what kind of testing we expect: to properly supports a platform, our CI needs to run tests on every PR on that platform. Without CI, change to CMake configs or even some unrelated changes (e.g., printf(“%s”, NULL); or MIN/MAX macro) could accidentally break the lib on QNX due to platform differences. If you have any suggestions on how to implement QNX CI testing, we’d be open consider them.

One potential compromise is maintaining dedicated QNX branches across all CRT libraries that get periodically rebased to main as part of a qnx-ports release pipeline. This would need some investigation, but it might work.

[dchordia]

Got it. I have raised a discussion internally on how QNX build and test pipelines can be integrated for external repositories.Since this is the first time, there are questions around licensing, cross compile builds and testing, display of results, etc. Legal is also involved (due to licensing) hence I will get back to you in some time.