Problem: Config profile could not be found. I need help using semaphore ui and AWS profiles from the ~/.aws/credentials.

Question

Problem: Config profile could not be found. I need help using semaphore ui and AWS profiles from the ~/.aws/credentials.

kdjomeda opened this issue 3 months ago · comments

kodjo-kuma Djomeda commented 3 months ago

Hello Community,

I have run into issues trying to execute ansible-semaphoreui with AWS profiles. There are 2 issues in one:

Semaphore ui doesn't seem to see the AWS PROFILES from the installed awscli from python with ~/.aws/{credentials,config} configured for the same user that owns and executes semaphores
If the above is fixed, how to pass AWS_PROFILE=blablabla command line environment variable to the ansible-playbook -I inventory blabla.playbook.yml bla bla bla execution ?

Setup used
ubuntu 22.04 LTS on AWS EC2

semaphore@ip-172-18-1-143:~$ whoami
semaphore
semaphore@ip-172-18-1-143:~$ ls
LICENSE  ansible_projects  config.json  semaphore  semaphore_2.9.64_linux_arm64.tar.gz  test
semaphore@ip-172-18-1-143:~$ python --version
Python 3.12.0
semaphore@ip-172-18-1-143:~$ ansible --version
ansible [core 2.16.6]
  config file = None
  configured module search path = ['/home/semaphore/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible
  ansible collection location = /home/semaphore/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/semaphore/.pyenv/versions/3.12.0/bin/ansible
  python version = 3.12.0 (main, Apr 26 2024, 22:47:02) [GCC 13.2.0] (/home/semaphore/.pyenv/versions/3.12.0/bin/python3.12)
  jinja version = 3.1.3
  libyaml = True
semaphore@ip-172-18-1-143:~$ pip freeze | grep -e awscli -e boto
awscli==1.32.93
boto==2.49.0
boto3==1.34.93
botocore==1.34.93
semaphore@ip-172-18-1-143:~$ ls -lh ~/.aws/
total 12K
drwxrwxr-x 3 semaphore semaphore 4.0K May  7 15:02 cli
-rw------- 1 semaphore semaphore  695 Apr 26 23:15 config
-rw------- 1 semaphore semaphore  116 Apr 26 23:15 credentials
semaphore@ip-172-18-1-143:~$ aws s3 ls s3://devops-data-sync-test/ --profile devops --region us-east-1
                           PRE AWS:ComplianceItem/
                           PRE AWS:ComplianceSummary/
                           PRE AWS:InstanceInformation/
                           PRE AWS:Tag/
semaphore@ip-172-18-1-143:~$ ./semaphore version
v2.9.64
semaphore@ip-172-18-1-143:~$ ps aux | grep semaphore
semapho+    2686  0.0  0.3 1258012 29864 ?       Ssl  May07   1:13 /home/semaphore/semaphore server --config=/home/semaphore/config.json
root       68544  0.0  0.0  17852  6272 pts/2    S+   08:40   0:00 sudo -i -u semaphore
root       68560  0.0  0.0  17852  2540 pts/0    Ss   08:40   0:00 sudo -i -u semaphore
semapho+   71159  0.0  0.0   6676  1920 pts/0    S+   09:02   0:00 grep --color=auto semaphore

My task template is as follows

with the CLI args

["-e","global_var_environment=prod","-e", "global_var_profile=prod-uat-example","-vvvv"]

I had to pass the aws profile as a extra argument passed to ansible playbook shown below:

- name: All Fact Gatherer playbook
  hosts: localhost
  connection: local
  gather_facts: true
  vars_files:
    - "playbook_vars/{{ global_var_environment }}/global_vars.yml"
    - "playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault"
  environment:
    AWS_PROFILE: "{{ global_var_profile | default(lookup('env','AWS_PROFILE')) }}"
  tasks:

The default value is to pick it from the command line environment variable if I am able to set one

When I execute a task on it in debug mode the output is this

Task 12 added to queue
2:39:35 PM
Started: 12
2:39:35 PM
Run TaskRunner with template: Fact_Gatherer_Playbook_Test
2:39:35 PM
Preparing: 12
2:39:35 PM
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
2:39:36 PM
From github.com:companyA/ansible_project_project_a
2:39:36 PM
   9ef3a5d..8d380a5  semaphore  -> origin/semaphore
2:39:36 PM
Updating Repository git@github.com:companyA/ansible_project_project_a.git
2:39:36 PM
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
2:39:36 PM
From github.com:companyA/ansible_project_project_a
2:39:36 PM
 * branch            semaphore  -> FETCH_HEAD
2:39:36 PM
Updating 9ef3a5d..8d380a5
2:39:36 PM
Fast-forward
2:39:36 PM
 fact_gatherer_playbook.yml | 2 +-
2:39:36 PM
 1 file changed, 1 insertion(+), 1 deletion(-)
2:39:36 PM
No collections/requirements.yml file found. Skip galaxy install process.
2:39:36 PM
roles/requirements.yml has no changes. Skip galaxy install process.
2:39:36 PM
ansible-playbook [core 2.16.6]
2:39:36 PM
  config file = /home/semaphore/ansible_projects/repository_1_1/ansible.cfg
2:39:36 PM
  configured module search path = ['/home/semaphore/ansible_projects/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
2:39:36 PM
  ansible python module location = /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible
2:39:36 PM
  ansible collection location = /home/semaphore/ansible_projects/repository_1_1/collections
2:39:36 PM
  executable location = /home/semaphore/.pyenv/versions/3.12.0/bin/ansible-playbook
2:39:36 PM
  python version = 3.12.0 (main, Apr 26 2024, 22:47:02) [GCC 13.2.0] (/home/semaphore/.pyenv/versions/3.12.0/bin/python3.12)
2:39:36 PM
  jinja version = 3.1.3
2:39:36 PM
  libyaml = True
2:39:36 PM
Using /home/semaphore/ansible_projects/repository_1_1/ansible.cfg as config file
2:39:36 PM
[DEPRECATION WARNING]: [defaults]collections_paths option, does not fit var 
2:39:36 PM
naming standard, use the singular form collections_path instead. This feature 
2:39:36 PM
will be removed from ansible-core in version 2.19. Deprecation warnings can be 
2:39:36 PM
disabled by setting deprecation_warnings=False in ansible.cfg.
2:39:36 PM
setting up inventory plugins
2:39:37 PM
Loading collection ansible.builtin from 
2:39:37 PM
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
2:39:37 PM
Loading collection amazon.aws from /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws
2:39:39 PM
Parsed /home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_ec2.yml inventory source with ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin
2:39:39 PM
setting up inventory plugins
2:39:39 PM
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
2:39:39 PM
ansible_collections.amazon.aws.plugins.inventory.aws_ec2 declined parsing /home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml as it did not pass its verify_file() method
2:39:39 PM
host_list declined parsing /home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml as it did not pass its verify_file() method
2:39:39 PM
script declined parsing /home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml as it did not pass its verify_file() method
2:39:39 PM
Using inventory plugin 'ansible_collections.amazon.aws.plugins.inventory.aws_rds' to process inventory source '/home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml'
2:39:39 PM
toml declined parsing /home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml as it did not pass its verify_file() method
2:39:39 PM
[WARNING]:  * Failed to parse
2:39:39 PM
/home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml with
2:39:39 PM
auto plugin: The config profile (prod-uat-example) could not be found
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/inventory/manager.py", line 293, in parse_source
2:39:39 PM
    plugin.parse(self._inventory, self._loader, source, cache=cache)
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
2:39:39 PM
    plugin.parse(inventory, loader, path, cache=cache)
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/inventory/aws_rds.py", line 282, in parse
2:39:39 PM
    results = self._get_all_db_hosts(
2:39:39 PM
              ^^^^^^^^^^^^^^^^^^^^^^^
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/inventory/aws_rds.py", line 253, in _get_all_db_hosts
2:39:39 PM
    for connection, _region in self.all_clients("rds"):
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/inventory.py", line 178, in all_clients
2:39:39 PM
    connection = self.client(service, region=region)
2:39:39 PM
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/inventory.py", line 84, in client
2:39:39 PM
    return super().client(*args, **kw_args)
2:39:39 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/base.py", line 41, in client
2:39:39 PM
    conn = boto3_conn(self, conn_type="client", resource=service, **kw_args)
2:39:39 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/botocore.py", line 39, in boto3_conn
2:39:39 PM
    plugin.fail_aws(to_native(e))
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/base.py", line 34, in fail_aws
2:39:39 PM
    self._do_fail(to_native(message))
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/plugin_utils/base.py", line 28, in _do_fail
2:39:39 PM
    raise AnsibleError(message)
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/inventory/manager.py", line 293, in parse_source
2:39:39 PM
    plugin.parse(self._inventory, self._loader, source, cache=cache)
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/plugins/inventory/yaml.py", line 114, in parse
2:39:39 PM
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/inventory/manager.py", line 293, in parse_source
2:39:39 PM
    plugin.parse(self._inventory, self._loader, source, cache=cache)
2:39:39 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/plugins/inventory/ini.py", line 138, in parse
2:39:39 PM
    raise AnsibleParserError(e)
2:39:39 PM
[WARNING]:  * Failed to parse
2:39:39 PM
/home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml with
2:39:39 PM
yaml plugin: Plugin configuration YAML file, not YAML inventory
2:39:39 PM
[WARNING]:  * Failed to parse
2:39:39 PM
/home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml with
2:39:39 PM
ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not
2:39:39 PM
allowed, this character is reserved to provide a port.
2:39:39 PM
[WARNING]: Unable to parse
2:39:39 PM
/home/semaphore/ansible_projects/repository_1_1/inventory/bb.aws_rds.yml as an
2:39:39 PM
inventory source
2:39:39 PM
[WARNING]: provided hosts list is empty, only localhost is available. Note that
2:39:39 PM
the implicit localhost does not match 'all'
2:39:39 PM
redirecting (type: modules) ansible.builtin.aws_caller_info to amazon.aws.aws_caller_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
Loading collection community.aws from /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/community/aws
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_instance_info to community.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_instance_info to amazon.aws.ec2_instance_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_net_info to amazon.aws.ec2_vpc_net_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_subnet_info to amazon.aws.ec2_vpc_subnet_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_subnet_info to amazon.aws.ec2_vpc_subnet_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_subnet_info to amazon.aws.ec2_vpc_subnet_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_subnet_info to amazon.aws.ec2_vpc_subnet_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.elb_application_lb_info to community.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) community.aws.elb_application_lb_info to amazon.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.iam_role_info to community.aws.iam_role_info
2:39:39 PM
redirecting (type: modules) community.aws.iam_role_info to amazon.aws.iam_role_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.elb_application_lb_info to community.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) community.aws.elb_application_lb_info to amazon.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.elb_application_lb_info to community.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) community.aws.elb_application_lb_info to amazon.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.elb_application_lb_info to community.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) community.aws.elb_application_lb_info to amazon.aws.elb_application_lb_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.cloudwatchlogs_log_group_info to community.aws.cloudwatchlogs_log_group_info
2:39:39 PM
redirecting (type: modules) community.aws.cloudwatchlogs_log_group_info to amazon.aws.cloudwatchlogs_log_group_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_route_table_info to community.aws.ec2_vpc_route_table_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_vpc_route_table_info to amazon.aws.ec2_vpc_route_table_info
2:39:39 PM
redirecting (type: modules) ansible.builtin.ec2_vpc_route_table_info to community.aws.ec2_vpc_route_table_info
2:39:39 PM
redirecting (type: modules) community.aws.ec2_vpc_route_table_info to amazon.aws.ec2_vpc_route_table_info
2:39:39 PM
Loading callback plugin default of type stdout, v2.0 from /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/plugins/callback/default.py
2:39:39 PM
Skipping callback 'default', as we already have a stdout callback.
2:39:39 PM
Skipping callback 'minimal', as we already have a stdout callback.
2:39:39 PM
Skipping callback 'oneline', as we already have a stdout callback.
2:39:39 PM
2:39:39 PM
PLAYBOOK: fact_gatherer_playbook.yml *******************************************
2:39:39 PM
Positional arguments: fact_gatherer_playbook.yml
2:39:39 PM
verbosity: 4
2:39:39 PM
connection: ssh
2:39:39 PM
become_method: sudo
2:39:39 PM
tags: ('all',)
2:39:39 PM
inventory: ('/home/semaphore/ansible_projects/repository_1_1/inventory',)
2:39:39 PM
extra_vars: ('{"semaphore_vars":{"task_details":{"id":12,"username":"joseph@companyA.app"}}}', 'global_var_environment=prod', 'global_var_profile=prod-uat-example')
2:39:39 PM
vault_password_files: ('/home/semaphore/ansible_projects/access_key_1152535',)
2:39:39 PM
forks: 5
2:39:39 PM
1 plays in fact_gatherer_playbook.yml
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:39 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:39 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:39 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:39 PM
2:39:39 PM
PLAY [All Fact Gatherer playbook] **********************************************
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:39 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:39 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:39 PM
2:39:39 PM
TASK [Gathering Facts] *********************************************************
2:39:39 PM
task path: /home/semaphore/ansible_projects/repository_1_1/fact_gatherer_playbook.yml:1
2:39:39 PM
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
2:39:39 PM
<127.0.0.1> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
2:39:39 PM
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968 `" && echo ansible-tmp-1715092779.3082068-10327-3425972425968="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968 `" ) && sleep 0'
2:39:40 PM
Using module file /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible/modules/setup.py
2:39:40 PM
<127.0.0.1> PUT /home/semaphore/ansible_projects/.ansible/tmp/ansible-local-10275oprb0_on/tmp7271e671 TO /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968/AnsiballZ_setup.py
2:39:40 PM
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968/ /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968/AnsiballZ_setup.py && sleep 0'
2:39:40 PM
<127.0.0.1> EXEC /bin/sh -c 'AWS_PROFILE=prod-uat-example /home/semaphore/.pyenv/versions/3.12.0/bin/python3.12 /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968/AnsiballZ_setup.py && sleep 0'
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1715092779.3082068-10327-3425972425968/ > /dev/null 2>&1 && sleep 0'
2:39:41 PM
ok: [localhost]
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:41 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:41 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/global_vars.yml'
2:39:41 PM
Trying secret FileVaultSecret(filename='/home/semaphore/ansible_projects/access_key_1152535') for vault_id=default
2:39:41 PM
Read vars_file 'playbook_vars/{{ global_var_environment }}/vault/{{ global_var_environment }}_variables.vault'
2:39:41 PM
2:39:41 PM
TASK [Obtaining AWS Account Facts] *********************************************
2:39:41 PM
task path: /home/semaphore/ansible_projects/repository_1_1/fact_gatherer_playbook.yml:11
2:39:41 PM
redirecting (type: modules) ansible.builtin.aws_caller_info to amazon.aws.aws_caller_info
2:39:41 PM
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935 `" && echo ansible-tmp-1715092780.7218232-10425-79984230110935="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935 `" ) && sleep 0'
2:39:41 PM
redirecting (type: modules) ansible.builtin.aws_caller_info to amazon.aws.aws_caller_info
2:39:41 PM
Using module file /home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/ansible_collections/amazon/aws/plugins/modules/aws_caller_info.py
2:39:41 PM
<127.0.0.1> PUT /home/semaphore/ansible_projects/.ansible/tmp/ansible-local-10275oprb0_on/tmp5zi4g4ij TO /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935/AnsiballZ_aws_caller_info.py
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935/ /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935/AnsiballZ_aws_caller_info.py && sleep 0'
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c 'AWS_PROFILE=prod-uat-example /home/semaphore/.pyenv/versions/3.12.0/bin/python3.12 /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935/AnsiballZ_aws_caller_info.py && sleep 0'
2:39:41 PM
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1715092780.7218232-10425-79984230110935/ > /dev/null 2>&1 && sleep 0'
2:39:41 PM
The full traceback is:
2:39:41 PM
  File "/tmp/ansible_aws_caller_info_payload_9m7tag3n/ansible_aws_caller_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/botocore.py", line 92, in boto3_conn
2:39:41 PM
    return _boto3_conn(conn_type=conn_type, resource=resource, region=region, endpoint=endpoint, **params)
2:39:41 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/tmp/ansible_aws_caller_info_payload_9m7tag3n/ansible_aws_caller_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/botocore.py", line 148, in _boto3_conn
2:39:41 PM
    session = boto3.session.Session(
2:39:41 PM
              ^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/boto3/session.py", line 90, in __init__
2:39:41 PM
    self._setup_loader()
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/boto3/session.py", line 131, in _setup_loader
2:39:41 PM
    self._loader = self._session.get_component('data_loader')
2:39:41 PM
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/session.py", line 802, in get_component
2:39:41 PM
    return self._components.get_component(name)
2:39:41 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/session.py", line 1140, in get_component
2:39:41 PM
    self._components[name] = factory()
2:39:41 PM
                             ^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/session.py", line 199, in 
2:39:41 PM
    lambda: create_loader(self.get_config_variable('data_path')),
2:39:41 PM
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/session.py", line 323, in get_config_variable
2:39:41 PM
    return self.get_component('config_store').get_config_variable(
2:39:41 PM
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/configprovider.py", line 465, in get_config_variable
2:39:41 PM
    return provider.provide()
2:39:41 PM
           ^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/configprovider.py", line 671, in provide
2:39:41 PM
    value = provider.provide()
2:39:41 PM
            ^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/configprovider.py", line 761, in provide
2:39:41 PM
    scoped_config = self._session.get_scoped_config()
2:39:41 PM
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2:39:41 PM
  File "/home/semaphore/.pyenv/versions/3.12.0/lib/python3.12/site-packages/botocore/session.py", line 422, in get_scoped_config
2:39:41 PM
    raise ProfileNotFound(profile=profile_name)
2:39:41 PM
fatal: [localhost]: FAILED! => {
2:39:41 PM
    "changed": false,
2:39:41 PM
    "invocation": {
2:39:41 PM
        "module_args": {
2:39:41 PM
            "access_key": null,
2:39:41 PM
            "aws_ca_bundle": null,
2:39:41 PM
            "aws_config": null,
2:39:41 PM
            "debug_botocore_endpoint_logs": false,
2:39:41 PM
            "endpoint_url": null,
2:39:41 PM
            "profile": "prod-uat-example",
2:39:41 PM
            "region": null,
2:39:41 PM
            "secret_key": null,
2:39:41 PM
            "session_token": null,
2:39:41 PM
            "validate_certs": true
2:39:41 PM
        }
2:39:41 PM
    },
2:39:41 PM
    "msg": "The config profile (prod-uat-example) could not be found"
2:39:41 PM
}
2:39:41 PM
2:39:41 PM
PLAY RECAP *********************************************************************
2:39:41 PM
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
2:39:41 PM
2:39:42 PM
Running playbook failed: exit status 2

Grateful if anyone can share their wisdom on this. Been blocked at this since 24th April 2024. I think I have run out of options and wanted to check with the community if there is something I needed to change or some understanding I don't have yet. Thanks in advance

Denis Gukov · Answer 1 · Tue May 21 2024 18:44:47 GMT+0800 (China Standard Time)

Hi @kdjomeda does it work by using ansible-playbook? It is not looks like Semaphore issue because python was found correctly.

Denis Gukov · Answer 2 · Tue May 21 2024 19:08:37 GMT+0800 (China Standard Time)

@kdjomeda in your setup description you use AWS profile devops but in Ansible you use profile prod-uat-example. Is it correct? Can you try global_var_profile=devops ?

kodjo-kuma Djomeda · Answer 3 · Tue May 21 2024 20:02:32 GMT+0800 (China Standard Time)

Sorry that was a way of showing that profile was set up and operational. There are multiple profiles in there

I am going to give it a try. I edited the profile to be prod-uat-example because it will reveal the product name. it is originally product-. I also used devops in the example so that it doesn't show critical names. But I have made a copy and we now have a profile called product-example. I will keep all consistent, try again and revert

kodjo-kuma Djomeda · Answer 4 · Tue May 21 2024 20:35:28 GMT+0800 (China Standard Time)

@fiftin I did run it directly on the device and it run but not on the semaphoreui

Denis Gukov · Answer 5 · Wed May 22 2024 06:15:39 GMT+0800 (China Standard Time)

Hm, it is really strange. @tboerger, @andreas-marschke do you have any idea?

Thomas Boerger · Answer 6 · Thu May 23 2024 21:26:20 GMT+0800 (China Standard Time)

I don't have an idea really... I never used profiles.

Denis Gukov · Answer 7 · Sat May 25 2024 00:13:21 GMT+0800 (China Standard Time)

@kdjomeda Looks like I understood why it can't find your profile. Because Semaphore override HOME environment variable.
Please try following config which I got from ChatGPT:

- name: Launch EC2 instance
  hosts: localhost
  gather_facts: no
  environment:
    AWS_PROFILE: your-custom-profile-name
    AWS_CONFIG_FILE: /path/to/your/custom/config
    AWS_SHARED_CREDENTIALS_FILE: /path/to/your/custom/credentials
  tasks:
    - name: Create an EC2 instance
      ec2:
        key_name: my-key
        instance_type: t2.micro
        image: ami-12345678
        region: us-east-1
        wait: yes
      register: ec2
    - debug:
        var: ec2

kodjo-kuma Djomeda · Answer 8 · Mon Jul 15 2024 13:57:14 GMT+0800 (China Standard Time)

Sorry for the late revert. The passing of the AWS_PROFILE, the AWS_CONFIG, and the credentials files have been done but it didn't change much on the outcome . The profile still remained not found. Besides that, this customisation or feeding in of the various environment variables inside the playbooks a is a little bit of a turn off for us. This will cause a lot of rewrite . 90% of our workload runs on AWS. So the profile is a key feature for us to use the semaphore ui. @fiftin