Refactor FilesExt: centralize hostname routing and simplify session management#1294
Refactor FilesExt: centralize hostname routing and simplify session management#1294parthban-db wants to merge 2 commits intomainfrom
Conversation
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
| return session | ||
| The session is created on first call and cached for reuse. | ||
| """ | ||
| if self._cached_cloud_provider_session is None: |
There was a problem hiding this comment.
The self._cached_cloud_provider_session is cached forever once it's created. I am a bit concerned about this as in rare cases the Session might enter an unrecoverable state and all of the following requests will fail and the entire WorkspaceClient has to be re-created to recover.
Example of such case (though this particular case has already been fixed): link
There was a problem hiding this comment.
There are a few possible ways to avoid this scenario:
- Automatic discard of cache: Maybe we can clear this cache after certain time (say 1h)
- Active discard of cache by caller: We can also add this logic to the
retryfunction to make sure the cache is cleared if all retry has failed.
| def _create_cloud_provider_session(self) -> requests.Session: | ||
| """Creates a separate session which does not inherit auth headers from BaseClient session.""" | ||
| session = requests.Session() | ||
| def _cloud_provider_session(self) -> requests.Session: |
There was a problem hiding this comment.
Can we make this function atomic? Multiple threads may be calling the function at the same time and it's possible to create multiple new Sessions while only one Session is cached and returned.
🥞 Stacked PR
Use this link to review incremental changes.
Why
Today the FilesExt class builds API URLs using hardcoded relative paths (e.g. "/api/2.0/fs/create-upload-part-urls") and creates a new unauthenticated requests.Session on every upload/download operation. This works, but has two limitations:
What changed
Interface changes
None. All changes are to private methods.
Behavioral changes
None. This is a pure refactor — the cloud provider session is now cached and reused instead of re-created per operation, but requests.Session is safe to share across calls.
Internal changes
How is this tested?
NO_CHANGELOG=true