Exit hubot on Unauthorized errors #178
Conversation
|
This might also address #157 At least if the service was dying we would know there was an error instead of having it logged and then continue on like there was no problem, even though the connection failed and the service isn't working. |
|
Thanks @nmaludy |
| robot.logger.error(util.format(error_msg, env.ST2_API_URL, err.message)); | ||
| robot.logger.error(util.format('Failed to retrieve commands from "%s": %s', env.ST2_API_URL, err.message)); | ||
| if (err.status === 401 || err.message.includes('Unauthorized')) { | ||
| throw err; |
There was a problem hiding this comment.
After more testing, not happy with this change either right now in case when st2 username & password used. This throw breaks any attempt of hubot-stackstorm to recover from failure and re-request st2 token as it happens normally (see #157 (comment)).
Here is how it works with current PR:
INFO Requesting a token...
INFO Loading commands....
ERROR Failed to retrieve commands from "https://localhost/api": Unauthorized - Token has expired.
ERROR {"name":"APIError","status":401,"message":"Unauthorized - Token has expired."}
INFO Hubot will shut down ...
INFO Disconnected from Slack RTM
INFO Exiting...
In this case request-token loop clashes in timing with the reload commands loop and so token was requested, but not received yet. At that exact moment reload commands loop happens that fails with 401 Token has expired because we hadn't enough time to receive the updated token.
To address that I think we need to modify the loop in st2client.js here: https://github.com/StackStorm/st2client.js/blob/46562a2b4959308d82998cc309b076f6c42d0ec2/index.js#L183-L201
and re-request token 1-3s earlier in advance. That will give enough timing window.
I think that might be one of the issues behind #157
There was a problem hiding this comment.
StackStorm/st2client.js#69 should fix it which emits st2 client expiry token 10s earlier, giving some time to re-authenticate that should fix some race condition cases.
There was a problem hiding this comment.
@armab I think this is resolved with StackStorm/st2client.js#69 merged.
This allows re-request st2 token in advance, instead of trying to get it when it's _already_ expired. Add here HTTP request/response time to communicate token update and it leads to issues and races like StackStorm/st2#4119 StackStorm/hubot-stackstorm#178 (comment) StackStorm/hubot-stackstorm#157
|
@blag added the tests, per your request. |
Addresses StackStorm/st2chatops#124
Accidentally Closes #157
Accidentally Closes StackStorm/st2#4119
This bugfix solves particular problem when invalid, expired
ST2_API_KEY/ST2_AUTH_TOKENor any otherUnauthorizedresponse from st2 server leads to bot exit.Previously the bot was running forever with no exit, trying to re-auth in a loop every
ST2_COMMANDS_RELOAD_INTERVAL(2mins). Now we deem such errors as fatal.