stuck at cloudformation
everplays opened this issue · comments
First of all, thanks for the ufo. It makes dealing with fargate a lot easier.
I have tried to deploy my own project to fargate using ufo. However, it got stuck on AWS::ECS::Service Ecs Resource creation Initiated of building the stack using cloudformation. So I thought maybe I am doing something wrong. However, I faced the same issue by following the quick start example (http://ufoships.com/quick-start):
(both cases using ufo v4.1.7)
Building Task Definitions...
Generating Task Definitions:
.ufo/output/demo-web.json
.ufo/output/demo-worker.json
.ufo/output/demo-clock.json
Task Definitions built in .ufo/output
Equivalent aws cli command:
aws ecs register-task-definition --cli-input-json file://.ufo/output/demo-web.json
demo-web task definition registered.
Deploying demo-web...
Ensuring log group for demo-web task definition exists
Log group name: ecs/demo-web
development cluster created.
Creating stack development-demo-web...
Generated template saved at: /tmp/ufo/development-demo-web/stack.yml
Generated parameters saved at: /tmp/ufo/development-demo-web/parameters.yml
08:38:34PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack development-demo-web User Initiated
08:38:38PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup EcsSecurityGroup
08:38:38PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup ElbSecurityGroup
08:38:38PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup
08:38:38PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup Resource creation Initiated
08:38:39PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::TargetGroup TargetGroup
08:38:42PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup EcsSecurityGroup Resource creation Initiated
08:38:42PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroup ElbSecurityGroup Resource creation Initiated
08:38:44PM CREATE_COMPLETE AWS::EC2::SecurityGroup EcsSecurityGroup
08:38:44PM CREATE_COMPLETE AWS::EC2::SecurityGroup ElbSecurityGroup
08:38:46PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::LoadBalancer Elb
08:38:47PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule
08:38:47PM CREATE_IN_PROGRESS AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule Resource creation Initiated
08:38:47PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::LoadBalancer Elb Resource creation Initiated
08:38:48PM CREATE_COMPLETE AWS::EC2::SecurityGroupIngress EcsSecurityGroupRule
08:40:48PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::LoadBalancer Elb
08:40:51PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::Listener Listener
08:40:51PM CREATE_IN_PROGRESS AWS::ElasticLoadBalancingV2::Listener Listener Resource creation Initiated
08:40:51PM CREATE_COMPLETE AWS::ElasticLoadBalancingV2::Listener Listener
08:40:54PM CREATE_IN_PROGRESS AWS::ECS::Service Ecs
08:40:54PM CREATE_IN_PROGRESS AWS::ECS::Service Ecs Resource creation Initiated
It never goes farther than the last line ^.
I thought maybe the generated stack.yml in /tmp can help to pinpoint the issue:
Description: "Ufo ECS stack demo-web"
Parameters:
# required
Vpc:
Description: Existing vpc id
Type: AWS::EC2::VPC::Id
ElbSubnets:
Description: Existing subnet ids for ELB
Type: List<AWS::EC2::Subnet::Id>
EcsSubnets:
Description: Existing subnet ids for ECS
Type: List<AWS::EC2::Subnet::Id>
EcsSecurityGroups:
Description: Existing ecs security group ids
Type: String
Default: ''
ElbSecurityGroups:
Description: Existing elb security group ids. List with commas.
Type: String
Default: ''
ElbTargetGroup:
Description: Existing target group
Type: String
Default: '' # when blank the automatically created TargetGroup is used
CreateElb:
Description: Create elb
Type: String
Default: true
EcsDesiredCount:
Description: Ecs desired count
Type: String
Default: 1
EcsTaskDefinition:
Description: Ecs task definition arn
Type: String
# Using to keep state
ElbEipIds:
Description: ELB EIP Allocation ids to use for network load balancer
Type: String
Default: ''
EcsSchedulingStrategy:
Description: The scheduling strategy to use for the service
Type: String
Default: 'REPLICA'
Conditions:
CreateElbIsTrue: !Equals [ !Ref CreateElb, true ]
ElbTargetGroupIsBlank: !Equals [ !Ref ElbTargetGroup, '' ]
CreateTargetGroupIsTrue: !And
- !Condition CreateElbIsTrue
- !Condition ElbTargetGroupIsBlank
ElbSecurityGroupsIsBlank: !Equals [ !Ref ElbSecurityGroups, '' ]
EcsSecurityGroupsIsBlank: !Equals [ !Ref EcsSecurityGroups, '' ]
EcsDesiredCountIsBlank: !Equals [ !Ref EcsDesiredCount, '' ]
Resources:
Elb:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Condition: CreateElbIsTrue
Properties:
Type: application
Tags:
- Key: Name
Value: demo-web
# Add additional extra security groups if parameters set
SecurityGroups: !Split
- ','
- !If
- ElbSecurityGroupsIsBlank
- !Ref ElbSecurityGroup
- !Join [',', [!Ref ElbSecurityGroups, !Ref ElbSecurityGroup]]
Subnets: !Ref ElbSubnets
Scheme: internet-facing
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Condition: CreateTargetGroupIsTrue
Properties:
VpcId: !Ref Vpc
Tags:
- Key: Name
Value: demo-web
Protocol: HTTP
Port: 80
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 10
Listener:
Type: AWS::ElasticLoadBalancingV2::Listener
Condition: CreateElbIsTrue
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
!If [ElbTargetGroupIsBlank, !Ref TargetGroup, !Ref ElbTargetGroup]
LoadBalancerArn: !Ref Elb
Protocol: HTTP
Port: 80
ElbSecurityGroup:
Type: AWS::EC2::SecurityGroup
Condition: CreateElbIsTrue
Properties:
GroupDescription: Allow http to client host
VpcId: !Ref Vpc
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '80'
ToPort: '80'
CidrIp: 0.0.0.0/0
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: '0'
ToPort: '65535'
CidrIp: 0.0.0.0/0
Tags:
- Key: Name
Value: demo-web-elb
Ecs:
Type: AWS::ECS::Service
DependsOn: Listener
Properties:
Cluster: development
DesiredCount: !If
- EcsDesiredCountIsBlank
- !Ref AWS::NoValue
- !Ref EcsDesiredCount
TaskDefinition: !Ref EcsTaskDefinition
# Default to port 80 to get template to validate. For worker processes
# there is no actual port used.
LoadBalancers: !If
- CreateTargetGroupIsTrue
- - ContainerName: web
ContainerPort: 4567
TargetGroupArn: !Ref TargetGroup
- !If
- ElbTargetGroupIsBlank
- []
- - ContainerName: web
ContainerPort: 4567
TargetGroupArn: !Ref ElbTargetGroup
SchedulingStrategy: !Ref EcsSchedulingStrategy
EcsSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allow http to client host
VpcId: !Ref Vpc
# Outbound access: instance needs access to internet to pull down image
# or else get CannotPullContainerError
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: '0'
ToPort: '65535'
CidrIp: 0.0.0.0/0
Description: outbound traffic
Tags:
- Key: Name
Value: demo-web
# Allow all traffic from ELB SG to ECS SG
EcsSecurityGroupRule:
Type: AWS::EC2::SecurityGroupIngress
Condition: CreateElbIsTrue
Properties:
IpProtocol: tcp
FromPort: '0'
ToPort: '65535'
SourceSecurityGroupId: !GetAtt ElbSecurityGroup.GroupId
GroupId: !GetAtt EcsSecurityGroup.GroupId
Description: application elb access to ecs
Outputs:
ElbDns:
Description: Elb Dns
Condition: CreateElbIsTrue
Value: !GetAtt Elb.DNSNameNot sure if it's relevant but I'm trying this on eu-west-1 (Ireland) region.
Can you share the events on the ECS service tab ? It's probably having issues trying to provision the tasks, so Cloudformation keeps trying until timeout.
@marcossantiago Thanks so much for helping! 😁
@everplays Dug into this. Thanks so much for all the info and providing the generated CloudFormation template. Believe the reason that CloudFormation is stuck on pending is because there are no EC2 servers to provision the ECS tasks onto. Ufo with ECS EC2 assumes that you have set up an ECS cluster already. Without a fleet of EC2 servers ECS cannot place the ECS task and finish creating the ECS service.
Could tell this by comparing your generated CloudFormation template with the one that should be generated when your using ufo with ECS Fargate instead of ECS EC2. Here's the diff:
$ colordiff yours.yml fargate.yml
84a85
> TargetType: ip
135a137,147
> LaunchType: FARGATE
> NetworkConfiguration:
> AwsvpcConfiguration:
> Subnets: !Ref EcsSubnets # required
> SecurityGroups: !Split
> - ','
> - !If
> - EcsSecurityGroupsIsBlank
> - !Ref EcsSecurityGroup
> - !Join [',', [!Ref EcsSecurityGroups, !Ref EcsSecurityGroup]]
> AssignPublicIp: ENABLED # Works with fargate but doesnt seem to work with non-fargate
$ Tested
Just ran through the Fargate guide, which contains a slightly different set of commands. Namely:
ufo init --image tongueroo/demo-ufo --launch-type fargate --execution-role-arn arn:aws:iam::112233445566:role/ecsTaskExecutionRole
Remember to replace 112233445566 with your actual account id and to create the ecsTaskExecutionRole. This is documented in the Fargate guide, so refer to it for details 👍 There's also a video demo.
Tested things both in us-west-2 (Oregon) and eu-west-1 (Ireland) this morning and ECS Fargate works in both.
Also note, if you are testing between regions you should run the ufo init command again before switching regions. The reason is that ufo init generates .ufo/settings/network/default.yml by looking up and using the default VPC for that region. It does this to allow customizations by editing the .ufo/settings/network/default.yml file. Most of this stuff happens tranparently so you don't have to deal with it.
Screenshots from eu-west-1
ECS Tasks dashboard:
CloudFormation Events:
Docs Update
Also updated the Quick Start docs to hopefully make things clearer. There's a "ECS EC2 vs ECS Fargate" section now.
@marcossantiago thanks for the help. Indeed, the issue was what @tongueroo explained in detail. I managed to get the demo running with the mentioned changes.
Thank you both.
I seem to be having this issue as well but I am using the Fargate deployment model in my init command