DataRepositoryAssociations reports sucess but files don't show up

[ses-dan]

If you have an active AWS support contract, please open a case with AWS Premium Support team using the below documentation to report the issue:
https://docs.aws.amazon.com/awssupport/latest/user/case-management.html

Before submitting a new issue, please search through open GitHub Issues and check out the troubleshooting documentation.

Please make sure to add the following data in order to facilitate the root cause detection.

Required Info:

• AWS ParallelCluster version [e.g. 3.1.1]: 3.8.0
• Full cluster configuration without any credentials or personal data.
  config_180124.txt
• Cluster name: pingu
• Output of pcluster describe-cluster command.
  {
  "creationTime": "2024-01-24T08:15:03.919Z",
  "headNode": {
  "launchTime": "2024-01-24T08:24:43.000Z",
  "instanceId": "i-0d4908c5ce11c0d3d",
  "publicIpAddress": "16.171.226.150",
  "instanceType": "c5n.large",
  "state": "running",
  "privateIpAddress": "10.0.0.228"
  },
  "version": "3.8.0",
  "clusterConfiguration": {
  "url": "https://parallelcluster-21a2b8a6db2e042c-v1-do-not-delete.s3.eu-north-1.amazonaws.com/parallelcluster/3.8.0/clusters/pingu-7l9fj77yzjl1b3wm/configs/cluster-config.yaml?versionId=9PiFICNIiOq.Na8c1DRP5lB6adTQOCIO&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARP4E2QSTIOX7FEXY%2F20240124%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240124T091625Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2c0831e5d780b6e5113b4f61c99421ffab265b7105ce00a7a5fb4fe9db706562"
  },
  "tags": [
  {
  "value": "hpc-cluster",
  "key": "CFD"
  },
  {
  "value": "3.8.0",
  "key": "parallelcluster:version"
  },
  {
  "value": "pingu",
  "key": "parallelcluster:cluster-name"
  }
  ],
  "cloudFormationStackStatus": "CREATE_COMPLETE",
  "clusterName": "pingu",
  "computeFleetStatus": "RUNNING",
  "cloudformationStackArn": "arn:aws:cloudformation:eu-north-1:102820906150:stack/pingu/ace48b20-ba90-11ee-8eef-0ec568337136",
  "lastUpdatedTime": "2024-01-24T08:15:03.919Z",
  "region": "eu-north-1",
  "clusterStatus": "CREATE_COMPLETE",
  "scheduler": {
  "type": "slurm"
  }
  }
• [Optional] Arn of the cluster CloudFormation main stack: See output above

Bug description and how to reproduce:
We are building a cluster through the config and use the DataRepositoryAssociations to set up links to s3 buckets. The creation seem to report no errors, but we have issues with files not showing up for one of the DRAs.

Currently we have 2 DRAs in the config. The first specified executes the Data repository task specified properly and the files are available when logging in. The second (CFD-setup) does report success for the Data repository task but the folder is empty on the instance. When creating a new Data repository task it executes successfully and updates the folder content - thus it seems like the issue is not with the configuration of the DRA but some process not fully completed during setup.

Please be sure to attach the following logs:

• From Head node: /var/log/cloud-init.log, /var/log/cfn-init.log and /var/log/chef-client.log
  cfn-init.log
  chef-client.log
  cloud-init.log
• From Compute node: /var/log/cloud-init-output.log.

Additional context:
Any other context about the problem. E.g.:

• CLI logs: ~/.parallelcluster/pcluster-cli.log
  pcluster-cli.log

• Custom bootstrap scripts, if any

• Screenshots, if useful.
  

[ses-dan]

After a bit more testing it seems like the post-install scripts that tries to access the linked S3 files are able to execute before the sync is done. When not doing roll-back the script fails on OnNodeConfigured, but when logging in to the node after some time the folders are synced the scripts finds the resources and can be run with success passed the failed step. Is this a bug or is it a user error not doing proper checks and wait?