CloudFormation troubleshooting lab: reset waits for the group's instances

CHANGELOG.md

@@ -7,6 +7,7 @@
 - Three more Data Engineer labs complete **Level 12: Data Engineer Associate** and Wave E (120 labs): an Airflow pipeline on MWAA (a versioned DAG bucket, a real Airflow 2.10.5 environment, a DAG with retries and an SNS failure alert, a run triggered with a CLI token, a failed task cleared and rerun, workers scaled, and a broken DAG rolled back to the previous S3 version); Kinesis stream processing (an event source mapping that bisects failing batches and sends what still fails to SQS, an idempotent consumer, a poison record read back by sequence number, a replay from a timestamp, enhanced fan-out, a shard split, and an MSK cluster); and data lake governance (SSE-KMS, the lake registered with Lake Formation, LF-Tags and a tag-based grant, PII pseudonymized into a curated zone, an analyst limited to the curated prefix by IAM, a Region guardrail, and the denied read found in CloudTrail data events). Every DEA concept card now links a lab.
 - Coverage doc: a new Data Engineer – Associate section with the guide, task-by-task coverage, a Floci 2.4.0 probe of Redshift Serverless and the Data API, Athena, the Glue Data Catalog, Schema Registry, crawlers, jobs and Data Quality, Lake Formation, Step Functions integrations, MWAA, MSK, Flink, and EMR, and the six-lab Wave E plan.
 - Fixed: on a first run, while Floci downloads the database or Airflow images, `ec2/shared-responsibility` (Create the RDS database) and `mwaa/airflow-pipeline` (Create the environment) failed with a read timeout or "DB Instance already exists": Floci starts those containers inside the create call, which outlasted the dashboard's 5-second timeout, and the automatic retry then hit the half-made resource. Those creates now wait up to 2 and 5 minutes without retrying, as the RDS and ECS labs already did.
+- Fixed: resetting `cloudformation/deployment-troubleshooting` left the `lab-deploy-network` stack in DELETE_FAILED while reporting it gone: the Auto Scaling group's instances were still shutting down when the VPC and subnets were deleted. Reset now waits for those instances, and reports an error if the stack still won't delete.
 - Fixed: on Docker Desktop on Apple silicon, Floci 2.4.0 runs x86 AMIs under emulation, and two labs failed there. In `cloudwatch/fleet-monitoring` the web server process never showed in `ps` (a Perl `$0` rename isn't visible under emulation), so its alarms never cleared; it now runs a copy of `sleep` under the server's name. In `ec2/performance-tuning` the 1,200 MiB report job finished inside the 1 GiB limit because its zero-filled buffer was never really used; it now writes every byte, and is killed as intended.
 - Lab snippets: every lab's Python (`boto3`) and Terraform tabs are generated from the AWS CLI command using the AWS CLI's own parser, so arguments map correctly, and the Terraform tab shows real resources for about 90 create operations (a note otherwise). Every lab's Python snippets were replayed against Floci 2.4.0, which fixed several lab commands (SSM `--document-name`, Route 53 Resolver `--creator-request-id`, quoted Firehose prefixes, and DynamoDB expression values).
 - README: rewritten around a three-command quickstart, with links to the Cloud Explainers YouTube channel and website. The longer setup, configuration, and testing notes moved to `docs/guide.md`.

dashboard/labs/cloudformation_troubleshooting_labs.py

@@ -457,6 +457,16 @@ def _enis(subnet_id: str) -> list[dict[str, Any]]:
         {'Name': 'subnet-id', 'Values': [subnet_id]}])['NetworkInterfaces'], [])
 
 
+def _live_instances(subnet_ids: list[str]) -> list[dict[str, Any]]:
+    if not subnet_ids:
+        return []
+    reservations = _safe_call(lambda: _client('ec2').describe_instances(Filters=[
+        {'Name': 'subnet-id', 'Values': subnet_ids},
+        {'Name': 'instance-state-name', 'Values': ['pending', 'running', 'shutting-down', 'stopping', 'stopped']},
+    ])['Reservations'], [])
+    return [item for reservation in reservations for item in reservation.get('Instances', [])]
+
+
 def _workload_enis(subnet_id: str) -> list[dict[str, Any]]:
     return [item for item in _enis(subnet_id) if item.get('Description') == 'other workload']
 
@@ -915,14 +925,21 @@ def attempt(label: str, call) -> None:
     if _group():
         attempt(f'autoscaling-group:{GROUP}', lambda: _client('autoscaling').delete_auto_scaling_group(
             AutoScalingGroupName=GROUP, ForceDelete=True))
-    subnet_a = _outputs(NETWORK_STACK).get('SubnetA')
+    outputs = _outputs(NETWORK_STACK)
+    # A force-deleted group's instances shut down in the background, and the subnets can't go until they have.
+    _wait(lambda: not _live_instances([outputs[key] for key in ('SubnetA', 'SubnetB') if outputs.get(key)]),
+          GROUP_WAIT_SECONDS, delay=3)
+    subnet_a = outputs.get('SubnetA')
     if subnet_a:
         ec2 = _client('ec2')
         for eni in _enis(subnet_a):
             attempt(f'network-interface:{eni["NetworkInterfaceId"]}',
                     lambda eni=eni: ec2.delete_network_interface(NetworkInterfaceId=eni['NetworkInterfaceId']))
     if _stack(NETWORK_STACK):
         attempt(f'stack:{NETWORK_STACK}', lambda: _delete_stack(NETWORK_STACK))
+        left = _stack(NETWORK_STACK)
+        if left:
+            errors.append(f'stack:{NETWORK_STACK}: still {left.get("StackStatus")} ({left.get("StackStatusReason") or "no reason given"})')
     placeholder = f'{TEMPLATE}-placeholder'
     if _safe_call(lambda: _client('ec2').describe_launch_templates(LaunchTemplateNames=[placeholder])
                   ['LaunchTemplates']):

dashboard/tests_cloudformation_troubleshooting_lab.py

@@ -96,3 +96,34 @@ def test_every_step_says_how_floci_emulates_it(self):
             self.assertIn('How Floci emulates this', step['explanation'], step['key'])
             self.assertIn(step['key'], lab.RUNNERS)
             self.assertIn(step['key'], lab.CHECKS)
+
+
+class ResetTests(SimpleTestCase):
+    def _reset(self, stack_after_delete):
+        stacks = {lab.NETWORK_STACK: [{'StackStatus': 'CREATE_COMPLETE'}, {'StackStatus': 'CREATE_COMPLETE'},
+                                      stack_after_delete]}
+
+        def stack(name):
+            queue = stacks.get(name) or []
+            return queue.pop(0) if queue else None
+
+        live = [[{'InstanceId': 'i-1'}], []]
+        with patch.object(lab, '_client'), patch.object(lab, '_share', return_value=None), \
+                patch.object(lab, '_image', return_value=None), patch.object(lab, '_group', return_value=None), \
+                patch.object(lab, '_stack', side_effect=stack), patch.object(lab, '_delete_stack'), \
+                patch.object(lab, '_outputs', return_value={'SubnetA': 'subnet-a', 'SubnetB': 'subnet-b'}), \
+                patch.object(lab, '_enis', return_value=[]), patch.object(lab, '_safe_call', return_value=None), \
+                patch.object(lab, '_live_instances', side_effect=lambda subnets: live.pop(0) if live else []) as instances, \
+                patch.object(lab.time, 'sleep'):
+            result = lab.reset()
+        return result, instances
+
+    def test_waits_for_the_groups_instances_before_deleting_the_network(self):
+        result, instances = self._reset(None)
+        instances.assert_called_with(['subnet-a', 'subnet-b'])
+        self.assertEqual(instances.call_count, 2)
+        self.assertEqual(result['errors'], [])
+
+    def test_a_network_stack_left_in_delete_failed_is_reported(self):
+        result, _ = self._reset({'StackStatus': 'DELETE_FAILED', 'StackStatusReason': 'subnet has dependencies'})
+        self.assertTrue(any('DELETE_FAILED' in error for error in result['errors']))