Help with reproducing errors in `infrastructure-deployer` #767

vicyap · 2023-09-15T00:18:49Z

vicyap
Sep 15, 2023

I am having an issue reproducing an error when I run infrastructure-deployer. It fails when it runs terragrunt plan-all. I am able to run the same terragrunt command from my local machine without an issue.

I am asking for help on how I can best debug this issue. Any ideas would be greatly appreciated!

Context:

Our infrastructure live code is nested under infrastructure-live/.
We have a _envcommon/services/rabbitmq.hcl file which is used in 4 modules under infrastructure-live/prod/us-west-2/prod/services/{rabbitmq,rabbitmq-3-11,internal-rabbitmq,internal-rabbitmq-3-11}

Issue:

Our CI calls this command:

infrastructure-deployer --aws-region us-west-2 -- terraform-planner infrastructure-deploy-script --ref f7085fa21f2640e7389801e1caa8e785c672c78e --binary terragrunt --command plan-all --command-args --terragrunt-modules-that-include=../_envcommon/services/rabbitmq.hcl --deploy-path infrastructure-live/prod/ --repo xxx --force-https true

Which then calls this command:

terragrunt plan-all --terragrunt-non-interactive -input=false --terragrunt-modules-that-include=../_envcommon/services/rabbitmq.hcl

Here is the output from CI:

Running infrastructure-deployer with args: --aws-region us-west-2 -- terraform-planner infrastructure-deploy-script --ref f7085fa21f2640e7389801e1caa8e785c672c78e --binary terragrunt --command plan-all --command-args --terragrunt-modules-that-include=../_envcommon/services/rabbitmq.hcl --deploy-path infrastructure-live/prod/ --repo xxx --force-https true
[infrastructure-deployer] INFO[2023-09-14T23:26:11Z] Invoking Lambda function ecs-deploy-runner-invoker to trigger deployment.
[infrastructure-deployer] INFO[2023-09-14T23:26:12Z] Successfully invoked Lambda function ecs-deploy-runner-invoker to trigger deployment.
[infrastructure-deployer] INFO[2023-09-14T23:26:12Z] Waiting for ECS task arn:aws:ecs:xxx to start
[infrastructure-deployer] INFO[2023-09-14T23:26:53Z] Deployment ECS Task arn:aws:ecs:xxx has started. Streaming logs.
Updating files: 100% (8923/8923), done.+0000] Updating files:  19% (1742/8923)
[ecs-deploy-runner][2023-09-14T23:27:33+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:33  Successfully cloned xxx to directory /tmp/tmp4802kcvv
[ecs-deploy-runner][2023-09-14T23:27:33+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:33  Checking out f7085fa21f2640e7389801e1caa8e785c672c78e
[ecs-deploy-runner][2023-09-14T23:27:33+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:33  Running command "git checkout f7085fa21f2640e7389801e1caa8e785c672c78e"
[ecs-deploy-runner][2023-09-14T23:27:35+0000] Note: switching to 'f7085fa21f2640e7389801e1caa8e785c672c78e'.
[ecs-deploy-runner][2023-09-14T23:27:35+0000] You are in 'detached HEAD' state. You can look around, make experimental
[ecs-deploy-runner][2023-09-14T23:27:35+0000] changes and commit them, and you can discard any commits you make in this
[ecs-deploy-runner][2023-09-14T23:27:35+0000] state without impacting any branches by switching back to a branch.
[ecs-deploy-runner][2023-09-14T23:27:35+0000] If you want to create a new branch to retain commits you create, you may
[ecs-deploy-runner][2023-09-14T23:27:35+0000] do so (now or later) by using -c with the switch command. Example:
[ecs-deploy-runner][2023-09-14T23:27:35+0000]   git switch -c <new-branch-name>
[ecs-deploy-runner][2023-09-14T23:27:35+0000] Or undo this operation with:
[ecs-deploy-runner][2023-09-14T23:27:35+0000]   git switch -
[ecs-deploy-runner][2023-09-14T23:27:35+0000] Turn off this advice by setting config variable advice.detachedHead to false
[ecs-deploy-runner][2023-09-14T23:27:35+0000] HEAD is now at f7085fa21 Revert more changes
[ecs-deploy-runner][2023-09-14T23:27:35+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:35  Successfully checked out f7085fa21f2640e7389801e1caa8e785c672c78e
[ecs-deploy-runner][2023-09-14T23:27:35+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:35  Checking out f7085fa21f2640e7389801e1caa8e785c672c78e
[ecs-deploy-runner][2023-09-14T23:27:35+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:35  Running command "git checkout f7085fa21f2640e7389801e1caa8e785c672c78e"
[ecs-deploy-runner][2023-09-14T23:27:35+0000] HEAD is now at f7085fa21 Revert more changes
[ecs-deploy-runner][2023-09-14T23:27:35+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:35  Successfully checked out f7085fa21f2640e7389801e1caa8e785c672c78e
[ecs-deploy-runner][2023-09-14T23:27:35+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:27:35  Running command "terragrunt plan-all --terragrunt-non-interactive -input=false --terragrunt-modules-that-include=../_envcommon/services/rabbitmq.hcl"
[ecs-deploy-runner][2023-09-14T23:27:35+0000] time=2023-09-14T23:27:35Z level=warning msg='plan-all' is deprecated. Running 'terragrunt run-all plan -input=false' instead. Please update your workflows to use 'terragrunt run-all plan -input=false', as 'plan-all' may be removed in the future!
[ecs-deploy-runner][2023-09-14T23:28:00+0000] time=2023-09-14T23:28:00Z level=info msg=The stack at /tmp/tmp4802kcvv/infrastructure-live/prod will be processed in the following order for command plan:
[ecs-deploy-runner][2023-09-14T23:28:00+0000] Group 1
[ecs-deploy-runner][2023-09-14T23:28:00+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq
[ecs-deploy-runner][2023-09-14T23:28:00+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11
[ecs-deploy-runner][2023-09-14T23:28:00+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq
[ecs-deploy-runner][2023-09-14T23:28:00+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11
[ecs-deploy-runner][2023-09-14T23:28:03+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq]
[ecs-deploy-runner][2023-09-14T23:28:03+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:03+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:18+0000] Initializing modules...
[ecs-deploy-runner][2023-09-14T23:28:19+0000] Downloading terraform-aws-modules/security-group/aws 4.17.2 for rmq_sg...
[ecs-deploy-runner][2023-09-14T23:28:19+0000] - rmq_sg in .terraform/modules/rmq_sg
[ecs-deploy-runner][2023-09-14T23:28:19+0000] Initializing the backend...
[ecs-deploy-runner][2023-09-14T23:28:20+0000]
[ecs-deploy-runner][2023-09-14T23:28:20+0000] Successfully configured the backend "s3"! Terraform will automatically
[ecs-deploy-runner][2023-09-14T23:28:20+0000] use this backend unless the backend configuration changes.
[ecs-deploy-runner][2023-09-14T23:28:20+0000] Initializing provider plugins...
[ecs-deploy-runner][2023-09-14T23:28:20+0000] - Finding hashicorp/aws versions matching ">= 3.29.0, > 3.75.1, < 5.0.0"...
[ecs-deploy-runner][2023-09-14T23:28:20+0000] - Finding latest version of hashicorp/random...
[ecs-deploy-runner][2023-09-14T23:28:20+0000] - Installing hashicorp/aws v4.67.0...
[ecs-deploy-runner][2023-09-14T23:28:23+0000] - Installed hashicorp/aws v4.67.0 (signed by HashiCorp)
[ecs-deploy-runner][2023-09-14T23:28:23+0000] - Installing hashicorp/random v3.5.1...
[ecs-deploy-runner][2023-09-14T23:28:24+0000] - Installed hashicorp/random v3.5.1 (signed by HashiCorp)
[ecs-deploy-runner][2023-09-14T23:28:24+0000] Terraform has created a lock file .terraform.lock.hcl to record the provider
[ecs-deploy-runner][2023-09-14T23:28:24+0000] selections it made above. Include this file in your version control repository
[ecs-deploy-runner][2023-09-14T23:28:24+0000] so that Terraform can guarantee to make the same selections by default when
[ecs-deploy-runner][2023-09-14T23:28:24+0000] you run "terraform init" in the future.
[ecs-deploy-runner][2023-09-14T23:28:24+0000] Terraform has been successfully initialized!
[ecs-deploy-runner][2023-09-14T23:28:24+0000]
[ecs-deploy-runner][2023-09-14T23:28:24+0000] You may now begin working with Terraform. Try running "terraform plan" to see
[ecs-deploy-runner][2023-09-14T23:28:24+0000] any changes that are required for your infrastructure. All Terraform commands
[ecs-deploy-runner][2023-09-14T23:28:24+0000] should now work.
[ecs-deploy-runner][2023-09-14T23:28:24+0000] If you ever set or change modules or backend configuration for Terraform,
[ecs-deploy-runner][2023-09-14T23:28:24+0000] rerun this command to reinitialize your working directory. If you forget, other
[ecs-deploy-runner][2023-09-14T23:28:24+0000] commands will detect it and remind you to do so if necessary.
[ecs-deploy-runner][2023-09-14T23:28:28+0000] random_password.password: Refreshing state... [id=none]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] random_uuid.id: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] random_pet.username: Refreshing state... [id=oxxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] aws_secretsmanager_secret.username[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] aws_secretsmanager_secret.password[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] aws_secretsmanager_secret_version.password[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] aws_secretsmanager_secret_version.username[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:28+0000] module.rmq_sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] aws_mq_broker.rmq: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[1]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[1]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[0]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[2]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[2]: Refreshing state... [id=xxx]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] No changes. Your infrastructure matches the configuration.
[ecs-deploy-runner][2023-09-14T23:28:29+0000] Terraform has compared your real infrastructure against your configuration
[ecs-deploy-runner][2023-09-14T23:28:29+0000] and found no differences, so no changes are needed.
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:29Z level=error msg=3 errors occurred:
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] [INFO] [infrastructure-deploy-script] 2023-09-14 23:28:29  "terragrunt plan-all" exited with code 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] Error: Could not run "terragrunt plan-all". Error=time=2023-09-14T23:27:35Z level=warning msg='plan-all' is deprecated. Running 'terragrunt run-all plan -input=false' instead. Please update your workflows to use 'terragrunt run-all plan -input=false', as 'plan-all' may be removed in the future!
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:00Z level=info msg=The stack at /tmp/tmp4802kcvv/infrastructure-live/prod will be processed in the following order for command plan:
[ecs-deploy-runner][2023-09-14T23:28:29+0000] Group 1
[ecs-deploy-runner][2023-09-14T23:28:29+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq
[ecs-deploy-runner][2023-09-14T23:28:29+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11
[ecs-deploy-runner][2023-09-14T23:28:29+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq
[ecs-deploy-runner][2023-09-14T23:28:29+0000] - Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:29Z level=error msg=3 errors occurred:
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] exit status 1
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time="2023-09-14T23:28:29Z" level=error msg="exit status 1" binary=deploy-runner-entrypoint version=latest
ERROR: Deployment failed
/usr/bin/xargs: bash: exited with status 255; aborting

Here is the output of the terragrunt plan-all command when I run it locally:

terragrunt plan-all --terragrunt-non-interactive -input=false --terragrunt-modules-that-include=../_envcommon/services/rabbitmq.hcl
WARN[0000] 'plan-all' is deprecated. Running 'terragrunt run-all plan -input=false' instead. Please update your workflows to use 'terragrunt run-all plan -input=false', as 'plan-all' may be removed in the future!
INFO[0007] The stack at /xxx/infrastructure-live/prod will be processed in the following order for command plan:
Group 1
- Module /xxx/infrastructure-live/prod/us-west-2/prod/services/rabbitmq
- Module /xxx/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11
- Module /xxx/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq
- Module /xxx/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11

random_password.password: Refreshing state... [id=none]
random_uuid.id: Refreshing state... [id=xxx]
random_pet.username: Refreshing state... [id=xxx]
random_pet.username: Refreshing state... [id=xxx]
random_password.password: Refreshing state... [id=none]
random_uuid.id: Refreshing state... [id=xxx]
random_password.password: Refreshing state... [id=none]
random_uuid.id: Refreshing state... [id=xxx]
random_pet.username: Refreshing state... [id=xxx]
random_uuid.id: Refreshing state... [id=xxx]
random_pet.username: Refreshing state... [id=xxx]
random_password.password: Refreshing state... [id=none]
aws_secretsmanager_secret.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.password[0]: Refreshing state... [id=axxx]
aws_secretsmanager_secret.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.password[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.username[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.username[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=xxx]
aws_secretsmanager_secret_version.password[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[2]: Refreshing state... [id=xxx]
aws_mq_broker.rmq: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[2]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[2]: Refreshing state... [id=xxx]
aws_mq_broker.rmq: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[2]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[2]: Refreshing state... [id=xxx]
aws_mq_broker.rmq: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[0]: Refreshing state... [id=sxxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[2]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[2]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[2]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_source_security_group_id[1]: Refreshing state... [id=xxx]
aws_mq_broker.rmq: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[1]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_cidr_blocks[0]: Refreshing state... [id=xxx]
module.rmq_sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=xxx]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
Releasing state lock. This may take a few moments...
Releasing state lock. This may take a few moments...
Releasing state lock. This may take a few moments...
Releasing state lock. This may take a few moments...

terragrunt_version=v0.50.16

The gruntwork_installer installs the following versions:

GRUNTWORK_INSTALLER_VERSION: v0.0.38
MODULE_CI_VERSION: v0.41.1
MODULE_SECURITY_VERSION: v0.61.0

The ecs-deploy-runner container has the following versions:

module_ci_tag=v0.52.14
tfenv_version=v3.0.0
terraform_version=1.5.5
terragrunt_version=v0.50.16
packer_version=1.7.4
hcl2json_version=v0.5.0

Tracked in ticket #110445

Answered by denis256

Sep 15, 2023

Hi,
looks like fail deployment of internal-rabbitmq-3-11, error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126

Can be attempted to run infrastructure-deployer with --terragrunt-log-level debug to see what is failing inside

[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq]
[ecs-deploy-runner][…

View full answer

denis256 · 2023-09-15T16:25:27Z

denis256
Sep 15, 2023
Collaborator

Hi,
looks like fail deployment of internal-rabbitmq-3-11, error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126

Can be attempted to run infrastructure-deployer with --terragrunt-log-level debug to see what is failing inside

[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:03Z level=error msg=Module /tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11 has finished with an error: [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126 prefix=[/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11]
[ecs-deploy-runner][2023-09-14T23:28:29+0000] time=2023-09-14T23:28:29Z level=error msg=3 errors occurred:
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] 	* [/tmp/tmp4802kcvv/infrastructure-live/prod/us-west-2/prod/services/internal-rabbitmq-3-11] exit status 126
[ecs-deploy-runner][2023-09-14T23:28:29+0000] exit status 1

3 replies

vicyap Sep 16, 2023
Author

Adding --terragrunt-log-level=debug helped!

I was able to determine that terragrunt was actually using Terraform version: 1.0.11 because that is what was set in my .terraform-version file.

I updated the version in .terraform-version to 1.5.5, verified that is the version being used in the log output, and now my pipeline passes.

But what still does not make sense is that terragrunt worked for me locally even with .terraform-version set to 1.0.11. And I verified that was the version using --terragrunt-log-level=debug. So it's possible there is a bug in terraform that only reproduced when I tested through CI but did not reproduce when I ran locally.

In any case, --terragrunt-log-level=debug helped so I'll mark this as the answer.

Thank You!

hmalikunlearn Oct 17, 2023

Hi I was wondering how you were able to add the --terragrunt-log-level=debug into your infrastructure-deployer arguments. I am getting and error that it is not a supported option

umm0n Apr 22, 2024

I'm having the same error using terraform 1.2.0 in .terraform-version. Locally, it works without issue. @hmalikunlearn did you manage to find what the issue was?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gruntwork

Help with reproducing errors in `infrastructure-deployer` #767

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Gruntwork

Help with reproducing errors in infrastructure-deployer #767

Uh oh!

Uh oh!

vicyap Sep 15, 2023

Replies: 1 comment · 3 replies

Uh oh!

denis256 Sep 15, 2023 Collaborator

Uh oh!

Uh oh!

vicyap Sep 16, 2023 Author

Uh oh!

hmalikunlearn Oct 17, 2023

Uh oh!

umm0n Apr 22, 2024

Help with reproducing errors in `infrastructure-deployer` #767

vicyap
Sep 15, 2023

Replies: 1 comment 3 replies

denis256
Sep 15, 2023
Collaborator

vicyap Sep 16, 2023
Author