Need to use parameters without cloudformation using the cli
geobennz opened this issue · comments
AWS customer would like to use parameters similar to cfn to avoid hardcoding SG and other settings in his yaml file using Pcluster cli
You can use an AWS ParallelCluster CloudFormation custom resource in an AWS CloudFormation stack.
Here is the documentation on how to do so: https://docs.aws.amazon.com/parallelcluster/latest/ug/cloudformation-v3.html
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.
Hi,
We also currently have pclusters version 3.7.1 deployed using cli command pcluster create-cluster --cluster-configuration pcluster/slurm.yaml --cluster-name name
We are looking to pass few inputs like security groups, which is copy pasted multiple time manually in the template using cli option like --extra-parameters flag in v2 (https://aws-parallelcluster.readthedocs.io/en/latest/commands.html)
Recreating this with AWS CloudFormation stack would be difficult considering the current configurations we have with slurm.
Hi @mharini ,
did you evaluate the usage of an intermediary framework like Jinja to have a parametric template for the cluster config?
This is an example of Jinja cluster config template we're using in our integration test framework.
The configuration is rendered with the parameters passed to the test framework cli, before calling the pcluster create-cluster command.
Thank you @enrico-usai for sharing this suggestion.
We definitely have this workaround in consideration, but having a direct option to pass via parameter would be much more efficient. It would simplify the configuration, making it easier to understand and implement.
Could we consider this as a feature request?
Hi @mharini,
I'm not sure it's something we can consider as feature enhancement. Rather than passing many parameters to the CLI, as design decision we choose to give the users the flexibility to pass a configuration file instead.
In your case, you need to pass Security Groups, but this kind of mechanism should be general.
The key point is that every parameter can be mapped to different configuration sections, (e.g. SG in the HeadNode vs SG in the queues) so the extra parameters should be mapped to the right section of the config file and to the right queue (and compute resource).
Then, in any case, a configuration file must be passed to the CLI. This means we would need to implement a way to have a "placeholder" in the configuration file. This is what Jinja for example already provides.
The basic idea is to avoid to introduce a templating mechanism in ParallelCluster CLI code, and instead rely on existing frameworks that can be used on top of the cluster configuration file, to give the flexibility to the users to select the framework they prefer or that they already can have in place.
Another mechanism we can suggest is the usage of ParallelCluster as CloudFormation Custom Resource. By using this existing feature you can consider Pcluster's cluster as a building block, as a resource of your CloudFormation template, so you can pass Security Groups or any parameter as input of the cluster configuration created by your "external" CFN template.