ERROR: You must specify a region (after upgrading to sceptre 4.0.1 version)
chenlego opened this issue · comments
Subject of the issue
Previously, I use sceptre version 3.3.0 and everything could work fine. After upgrade sceptre to 4.0.1. I started to encounter the following problem:
botocore.exceptions.NoRegionError: You must specify a region.
Environment
$ sceptre --version
Sceptre, version 4.0.1
$ python3 --version
Python 3.9.8
Steps to reproduce
file structure:
.
|-- config
| |-- config.yaml
| `-- us-west-2
| |-- app
| | `-- myapp.yaml
| `-- config.yaml
`-- templates
`-- myapp.yaml
4 directories, 4 files
file content:
$ find . -type f
./config/us-west-2/config.yaml
./config/us-west-2/app/myapp.yaml
./config/config.yaml
./templates/myapp.yaml
## ./config/config.yaml
$ cat ./config/config.yaml
project_code: aws
## ./config/us-west-2/config.yaml
$ cat ./config/us-west-2/config.yaml
region: us-west-2
## ./config/us-west-2/app/myapp.yaml
$ cat ./config/us-west-2/app/myapp.yaml
stack_name: myapp
template_path: myapp.yaml
parameters:
VpcId: !stack_output_external my-vpc::VPCID
## ./templates/myapp.yaml
$ cat ./templates/myapp.yaml
Parameters:
VpcId:
Type: String
Description: VPC ID
Resources:
NullResource:
Type: AWS::CloudFormation::WaitConditionHandle
Expected behaviour
In sceptre version 3.3.0, the same config could working fine without the errors below.
Actual behaviour
execution result
The problem I can see in the following debugging result are:
- It seems cannot re-use the Boto3 session.
- The first Boto3 session could reterive
regionfrom config.yaml properly. But, the second one cannot.
$ sceptre --debug launch -y us-west-2
[2023-02-14 16:26:53] - Adding yaml constructors for the entry point groups ['sceptre.hooks', 'sceptre.resolvers']
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scheduled_actions
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.hooks.cmd.Cmd'> with node tag !cmd
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.environment_variable.EnvironmentVariable'> with node tag !environment_variable
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.file_contents.FileContents'> with node tag !file_contents
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.no_value.NoValue'> with node tag !no_value
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_attr.StackAttr'> with node tag !stack_attr
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutput'> with node tag !stack_output
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutputExternal'> with node tag !stack_output_external
[2023-02-14 16:26:54] - Added constructor for <class 'resolver.file.File'> with node tag !file
[2023-02-14 16:26:54] - Added constructor for <class 'resolver.rcmd.SceptreResolverCmd'> with node tag !rcmd
[2023-02-14 16:26:54] - Reading in 'us-west-2/app/config.yaml' files...
[2023-02-14 16:26:54] - Config: {'project_path': '/root/reproduce', 'stack_group_path': 'us-west-2/app', 'project_code': 'aws', 'region': 'us-west-2'}
[2023-02-14 16:26:54] - Reading in 'us-west-2/app/myapp.yaml' files...
[2023-02-14 16:26:54] - Config: {'project_path': '/root/reproduce', 'stack_group_path': 'us-west-2/app', 'project_code': 'aws', 'region': 'us-west-2', 'stack_name': 'myapp', 'template_path': 'myapp.yaml', 'parameters': {'VpcId': <sceptre.resolvers.stack_output.StackOutputExternal object at 0x7f8fad0afa00>}}
/usr/local/lib/python3.9/site-packages/sceptre/stack.py:426: DeprecatedWarning: template_path is deprecated as of 4.0.0 and will be removed in 5.0.0. Use the template Stack Config key instead.
setattr(self, deprecated_attribute_name, deprecated_value)
[2023-02-14 16:26:54] - Generate dependencies for stack us-west-2/app/myapp
[2023-02-14 16:26:54] - us-west-2/app/myapp - Launching Stack
[2023-02-14 16:26:54] - No cloudformation client found, creating one...
[2023-02-14 16:26:54] - Getting Boto3 session
('us-west-2', None, None)
[2023-02-14 16:26:54] - No Boto3 session found, creating one...
[2023-02-14 16:26:54] - Using cli credentials...
[2023-02-14 16:26:54] - Using credential set from shared-credentials-file: {'AccessKeyId': '****************DTGX', 'SecretAccessKey': '************************************dZlW', 'Region': 'us-west-2'}
[2023-02-14 16:26:54] - Boto3 session created
[2023-02-14 16:26:55] - us-west-2/app/myapp - Stack is in the PENDING state
[2023-02-14 16:26:55] - us-west-2/app/myapp - Creating Stack
[2023-02-14 16:26:55] - Resolving external Stack output: my-vpc::VPCID
[2023-02-14 16:26:55] - Collecting outputs from 'my-vpc'...
[2023-02-14 16:26:55] - No cloudformation client found, creating one...
[2023-02-14 16:26:55] - Getting Boto3 session
(None, None, None)
[2023-02-14 16:26:55] - No Boto3 session found, creating one...
[2023-02-14 16:26:55] - Using cli credentials...
[2023-02-14 16:26:55] - Using credential set from shared-credentials-file: {'AccessKeyId': '****************DTGX', 'SecretAccessKey': '************************************dZlW', 'Region': None}
[2023-02-14 16:26:55] - Boto3 session created
Traceback (most recent call last):
File "/usr/local/bin/sceptre", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/cli/helpers.py", line 43, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 62, in launch_command
exit_code = launcher.launch(prune)
File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 105, in launch
code = code or self._deploy(deploy_plan)
File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 197, in _deploy
result = deploy_plan.launch()
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 170, in launch
return self._execute(*args)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 29, in wrapped
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 55, in _execute
return executor.execute(*args)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/executor.py", line 52, in execute
stack, status = future.result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/executor.py", line 59, in _execute
result = getattr(actions, self.command)(*args)
File "/usr/local/lib/python3.9/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
response = func(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/actions.py", line 213, in launch
status = self.create()
File "/usr/local/lib/python3.9/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
response = func(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/plan/actions.py", line 73, in create
"Parameters": self._format_parameters(self.stack.parameters),
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 202, in __get__
container = super().__get__(stack, stack_class)
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 94, in __get__
return self.get_resolved_value(stack, stack_class)
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 252, in get_resolved_value
_call_func_on_values(resolve, container, Resolver)
File "/usr/local/lib/python3.9/site-packages/sceptre/helpers.py", line 64, in _call_func_on_values
func_on_instance(key)
File "/usr/local/lib/python3.9/site-packages/sceptre/helpers.py", line 58, in func_on_instance
func(attr, key, value)
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 226, in resolve
result = self.resolve_resolver_value(value)
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 165, in resolve_resolver_value
return resolver.resolve()
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 179, in resolve
return self._get_output_value(
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 42, in _get_output_value
outputs = self._get_stack_outputs(stack_name, profile, region, sceptre_role)
File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 70, in _get_stack_outputs
response = connection_manager.call(
File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 54, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 460, in call
client = self._get_client(service, region, profile, stack_name, sceptre_role)
File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 370, in _get_client
self._clients[key] = self._get_session(
File "/usr/local/lib/python3.9/site-packages/boto3/session.py", line 299, in client
return self._session.create_client(
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 884, in create_client
client = client_creator.create_client(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 102, in create_client
client_args = self._get_client_args(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 384, in _get_client_args
return args_creator.get_client_args(
File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 71, in get_client_args
final_args = self.compute_client_args(
File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 148, in compute_client_args
endpoint_config = self._compute_endpoint_config(
File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 234, in _compute_endpoint_config
return self._resolve_endpoint(**resolve_endpoint_kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 320, in _resolve_endpoint
return endpoint_bridge.resolve(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 465, in resolve
resolved = self.endpoint_resolver.construct_endpoint(
File "/usr/local/lib/python3.9/site-packages/botocore/regions.py", line 196, in construct_endpoint
result = self._endpoint_for_partition(
File "/usr/local/lib/python3.9/site-packages/botocore/regions.py", line 230, in _endpoint_for_partition
raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.
Workaround
export AWS_DEFAULT_REGION="us-west-2"instead of definingregionin config.yaml.- downgrade to 3.3.0
Hey @chenlego! Thanks for posting this issue. Looks like there's some sort of conflict between stack-output-external and the recent changes to the ConnectionManager. I'll need to dive in, which probably won't happen today. If you wanted to do some investigation yourself and post your findings, that might speed things along.
I'm pretty certain the issue relates to the ConnectionManager's call() method and v4's overhaul of how it resolves values when combined with stack_output_external and how it resolves values.
I've figured this issue out and have a fix that appears to work. Our integration tests didn't catch this because our integration tests on !external_stack_output...
- Didn't test using that resolver without specifying explicit profile and region (which you're not doing)
- Apparently to replicate this issue, it looks like either (1) you cannot have a profile specified in your Stack Configs or via environment variable (which I alway do) OR (2) If you have a profile specified, it cannot specify a default region (which I do).
In either case, it somehow made it through. I've added an integration test that should cover this situation in my branch. I should get a fix up today, I think.
Thank you @jfalkenstein. I do very appreciated with your help. I have validated the new version and it could work properly.