add retries for ephemeral instance api calls #61

steffyP · 2026-01-27T13:46:45Z

As currently the API for ephemeral instances is known to be unstable, this PR aims to make to retry failing API calls and to capture additional logs in case of failure cases.
This way it will be easier to pinpoint issues in the future.

Example logs:

...
curl command failed while fetching instances. Response: {"error": true, "message": "compute.general_error"}
Command failed with exit code 1. Retrying in 5 seconds... (1/5)
Creating preview environment ...
Created preview environment with endpoint URL: https://ls-4ra30eqlofl88.sandbox.localstack.cloud/
...
Fetching logs for preview-localstack-setup-localstack-61 ...
Instance is not ready yet, waiting for 'Ready.' message in logs...
Command failed with exit code 1. Retrying in 5 seconds... (1/5)
preview-localstack-setup-localstack-61 logs:
Localstack extensions installer: installing extensions defined EXTENSIONS_AUTO_INSTALL
....

changes:

retry failing API call up to 5 times, with a 5 second waiting time in between (no guarantee this is enough, but it was sufficient for our CI and we have the logs for debugging issues)
log details (e.g. retry, the actual response)
added a helper script that will be loaded so that we can reuse the function
added additional check to make sure that logs are successfully retrieved

Suggested future work (once the API is stable):

rather use the localstack CLI than hardcoding the API calls, as any changes to the API will need changes to the action as well
- As @lukqw pointed out the CLI may not have the capabilities of handling ENVs that have been set (regarding autoloading of pods, extensions etc).
add more tests, e.g. some attributes are currently not tested or verified: auto-load-pod, ci-project

github-actions · 2026-01-27T13:47:26Z

The ephemeral instance for the application preview has been shut down

steffyP · 2026-01-29T17:09:50Z

.github/workflows/ephemeral.yml

+  pull_request:
+    paths-ignore:
+      - ./*.md
+      - LICENSE


removing the workflow_dispatch that I added in a previous PR, as it doesn't make sense with the logic that the ephemeral instances are currently following (e.g. searching for the PR number and adding comment to the PR)

steffyP · 2026-01-29T17:10:48Z

ephemeral/shutdown/action.yml

-    description: 'LocalStack API key used to access the platform api'
-    required: true
+    description: 'LocalStack Auth Token used to access the platform api'
+    required: false


the localstack-api-key is not documented, neither is it required or currently used in our own CI test. It is a fallback in case the LOCALSTACK_AUTH_TOKEN and LOCALSTACK_API_KEY are not set

Thanks, that's a good point! That's maybe something to revisit in the future. The term is outdated (we only have auth tokens now) and it seems a bit confusing to configure...

ephemeral/startup/action.yml

alexrashed

Thanks a lot for fixing the usage of ephemeral instances by implementing the retry logic!
From my perspective this is looking great. I think there is potential to move this to the CLI and reuse it here in the future, but that would definitely be out of scope for this PR!

Given that @lukqw is the expert on ephemeral instances and also has contributed to this feature in the action in the past, I think it might be good to maybe also get a quick review from you as well? 🙏🏽

alexrashed · 2026-01-30T09:08:48Z

ephemeral/shutdown/action.yml

-    description: 'LocalStack API key used to access the platform api'
-    required: true
+    description: 'LocalStack Auth Token used to access the platform api'
+    required: false


Thanks, that's a good point! That's maybe something to revisit in the future. The term is outdated (we only have auth tokens now) and it seems a bit confusing to configure...

ephemeral/startup/action.yml

alexrashed · 2026-01-30T09:26:33Z

ephemeral/retry-function.sh

+# retry() function: Retries a given command up to 'retries' times with a 'wait' interval.
+# Usage: retry <command>
+# Example: retry my_api_call_function
+retry() {


@lukqw I think this is where your input would really be valuable. Is this retry mechanism what you would expect a client to do if they get a failure response from the ephemeral instance API?
This effectively would be a classic retry machanism, 5 retries, 5 seconds constant backoff.

test retry loops to create ephemeral instances

ff31d3c

steffyP added 11 commits January 27, 2026 14:53

fixup

3ca264f

fixup

dc0474e

fixup

363f287

set +e inside the retry loop

b7d3d58

add more debug messages

44a4b1e

retry retrieving logs

8d3353d

adding more debug information for logs

89ed704

refactored retry function into separate script

8f70f75

fixup

b3ad5bc

fixup

5f7b169

debug log result

3ba8a36

steffyP force-pushed the fix-ephemeral branch from 2224b31 to 3ba8a36 Compare January 29, 2026 15:54

steffyP added 3 commits January 29, 2026 17:18

more debug messages, fail if the response contains error key

0ff0b96

fixup

dba16e5

fixup

9036a17

steffyP changed the title ~~test retry loops to create ephemeral instances~~ add retries for ephemeral instance api calls Jan 29, 2026

fixup

e88acfc

steffyP commented Jan 29, 2026

View reviewed changes

ephemeral/startup/action.yml Show resolved Hide resolved

steffyP marked this pull request as ready for review January 29, 2026 17:34

steffyP requested a review from alexrashed January 29, 2026 17:34

alexrashed approved these changes Jan 30, 2026

View reviewed changes

alexrashed requested a review from lukqw January 30, 2026 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add retries for ephemeral instance api calls #61

add retries for ephemeral instance api calls #61

steffyP commented Jan 27, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 27, 2026 •

edited

Loading

Uh oh!

steffyP Jan 29, 2026

Uh oh!

steffyP Jan 29, 2026

Uh oh!

alexrashed Jan 30, 2026

Uh oh!

Uh oh!

alexrashed left a comment

Uh oh!

alexrashed Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

alexrashed Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

add retries for ephemeral instance api calls #61

Are you sure you want to change the base?

add retries for ephemeral instance api calls #61

Conversation

steffyP commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steffyP Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

steffyP Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

alexrashed Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

alexrashed Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alexrashed Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

steffyP commented Jan 27, 2026 •

edited

Loading

github-actions bot commented Jan 27, 2026 •

edited

Loading