Skip to content

[UX] Allow purging non-existent k8s cluster #4980

New issue

Have a question about this project? No Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

No Sign up for GitHub

By clicking “No Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? No Sign in to your account

Merged
merged 6 commits into from
Mar 25, 2025
Merged

[UX] Allow purging non-existent k8s cluster #4980

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b833003
handle non-existent cluster clean up with purge in post_teardown_cleanup
kyuds Mar 17, 2025
563fa21
style
kyuds Mar 18, 2025
b51b096
remove remove_cluster_config call; move to separate issue
kyuds Mar 22, 2025
368473e
Update sky/backends/cloud_vm_ray_backend.py
kyuds Mar 25, 2025
cf1e943
format
kyuds Mar 25, 2025
323071a
format attempt 2
kyuds Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion sky/backends/cloud_vm_ray_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4356,7 +4356,19 @@ def _detect_abnormal_non_terminated_nodes(
# If cluster_yaml is None, the cluster should ensured to be terminated,
# so we don't need to do the double check.
if handle.cluster_yaml is not None:
_detect_abnormal_non_terminated_nodes(handle)
try:
_detect_abnormal_non_terminated_nodes(handle)
except exceptions.ClusterStatusFetchingError as e:
if purge:
msg = common_utils.format_exception(e, use_bracket=True)
logger.warning(
'Failed abnormal non-terminated nodes cleanup. '
'Skipping and cleaning up as purge is set. '
f'Details: {msg}')
logger.debug(f'Full exception details: {msg}',
exc_info=True)
else:
raise

if not terminate or remove_from_db:
global_user_state.remove_cluster(handle.cluster_name,
Expand Down