Connect saucelabs testing environment with travis

[kwonoj]

This PR now connects travis to saucelabs (https://saucelabs.com/) to execute browser test with karma.

• updated karma configuration to include minimal set of browser for initial setup
• karma configuration detects environment, run test under saucelabs if runs under travis
• travis configuration updated to trigger karma end of build

Few notes:

• Saucelab test does not being executed with Pull Request bulid (https://docs.travis-ci.com/user/sauce-connect/)
• ReactiveX/RxJS travis (https://travis-ci.org/ReactiveX/RxJS) preconfigured environment variable for saucelabs authentication prior to this PR. Means, once this PR is merged test will be executed with merged TOT.
• Current testing status can be found at https://saucelabs.com/u/ojkwon, publicly available by using OSS account.

[staltz]

looks ok.

[kwonoj]

Applied minor update into PR to trigger karma test only if

• given build is not pull request
• travis has configured to have saucelabs authentication variables

allows to not to block any travis jobs by browser test.

[benlesh]

given build is not pull request

I'd almost prefer it did run if it's a PR too... We wouldn't want to merge a PR that was going to break a browser we're targeting, right?

Current testing status can be found at https://saucelabs.com/u/ojkwon, publicly available by using OSS account.

What do we need to do to get this working under a more generic account, like a "ReactiveX" account? I've not dealt with Sauce Labs.

@jeffbcross @IgorMinar: Have you guys dealt with this?

[kwonoj]

I'd almost prefer it did run if it's a PR too...

: I hope that too but travis says clearly it's not supported and there won't be any plan for it. If there's possible way to establish this, it should be included.

What do we need to do to get this working under a more generic account, like a "ReactiveX" account? I've not dealt with Sauce Labs.

: Creating new account with right user name 'reactivex' in this case, and ask saucelabs to switch them into OSS would work. For this PR, I've used my own at least to ensure it works. Switching account is fairly straightforward and simple, only requires to replace environment variables in travis configuration. (Means, this PR does not have any reference to my account and does not have constraint to be merged due to account)

[kwonoj]

Creating new account with right user name 'reactivex' in this case, and ask saucelabs to switch them into OSS would work.

: I'd like to do this gladly but creating account requires mail address which should not be mine for official reactivex account, leaving it from my hand.

[benlesh]

@kwonoj does the account just need to be a part of the ReactiveX organization?

[kwonoj]

@Blesh I do see they allow to add 'member' under primary account, but in this case I'd rather suggest to create separate account per repo such as reactives_rxjs, or similar.

Primary reason for those is to allow maximum concurrency between repo, OSS account allows 5 maximum concurrent VM so if creating one single account and use it among different repo, it could easily create bottleneck for test. Course caveat is each repo requires separate account to be maintained..

[IgorMinar]

The current situation is not pretty. I can give you a rundown but what you
are looking for is travis-ci/travis-ci#1946

SauceLabs supposedly developed an JWT Addon but to my knowledge Travis
doesn't support it yet.

On Mon, Dec 14, 2015 at 5:03 PM OJ Kwon [email protected] wrote:

@Blesh https://github.com/blesh I do see they allow to add 'member'
under primary account, but in this case I'd rather suggest to create
separate account per repo such as reactives_rxjs, or similar.

Primary reason for those is to allow maximum concurrency between repo, OSS
account allows 5 maximum concurrent VM so if creating one single account
and use it among different repo, it could easily create bottleneck for
test. Course caveat is each repo requires separate account to be
maintained..

—
Reply to this email directly or view it on GitHub
#1023 (comment).

[kwonoj]

My take on this is let establish testing on master branch first and looking possible solutions for PR also , by given status of testing PR is not immediately available.

[benlesh]

My gut tells me that merging this probably won't hurt. I'm not sure how I can verify if it's going to be okay without merging it, honestly.

So... LGTM? Maybe?

@kwonoj would you like to rebase and merge this and we can see how it works out?

[kwonoj]

how I can verify if it's going to be okay without merging it, honestly.

: Yes, due to current circumstances it's not avail to verify before merge :(

Will do check in myself, verify using my account to see results. Once it says it's ok, then we can create fresh new account for official RxJS to connect it actually. Will update status shortly.

[kwonoj]

Merged with 70b1997. Not closing this yet until completes

• confirm saucelab test results
• connect with official RxJS account of saucelabs instead of personal one

[kwonoj]

Connection's verified successfully and having 1 failed test in browser

https://saucelabs.com/tests/dcade6598d344ae1a6b0d54b28e1031e

Chrome 46.0.2490 (Windows 10 0.0.0) Observable.prototype.throttleTime() should throttle events multiple times FAILED
Expected '2-4' to be undefined.
at SafeSubscriber. (/tmp/6230fa8feccbde9b7b0fc4c03e259b6e.browserify:47050:19)
at SafeSubscriber.tryCatcher as _next
at SafeSubscriber.next (/tmp/6230fa8feccbde9b7b0fc4c03e259b6e.browserify:827:18)
at ThrottleTimeSubscriber._next (/tmp/6230fa8feccbde9b7b0fc4c03e259b6e.browserify:6862:30)

I have experience similar before one of my VM environment, suspect due to async test. I'll try to look address this separately.

Meanwhile, @Blesh - I think you can create saucelabs account for RxJS now.

[kwonoj]

To be clear, test fails only on Chrome 46 while FF passes.

[kwonoj]

PR #1129 is created to avoid async test failure.

[kwonoj]

Now test are successfully connected without failure, account is remaining item to close this.

[kwonoj]

On second thought, I'll close this issue since change itself is already in to prevent any accidental additional merge attempt. Connecting actual account doesn't need codebase changes (can be configured in travis), can be tracked in issue #998.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.