Add PyHPS CLI #3091

germa89 · 2024-05-14T15:41:26Z

As the title.

The idea is you can have:

$ pymapdl submit main_file.py 
$ pymapdl submit --name="my project" main_file.py --requirements_file=requirements.txt

The help shows:

(venv) pymapdl $ pymapdl submit --help
Usage: pymapdl submit [OPTIONS] MAIN_FILE

  Submit jobs to an HPC cluster using PyHPS package.

Options:
  --name TEXT                 Name of the PyHPS project to be created.
  --url TEXT                  URL where the HPS cluster is deployed. For
                              example: "https://myserver:3000/hps"
  --user TEXT                 Username to login into the HPC cluster.
  --password TEXT             Password used to login into the HPC cluster.
  --python TEXT               Set python version to be used to create the
                              virtual environment and run the python file. By
                              default it uses python3 (default in cluster).
  --output_files TEXT         Set the output files to be monitored.
  --shell_file TEXT           If desired, you can provide a shell script to
                              execute instead of the python file. You can call
                              your python file from it if you wish. By
                              default, it is not used.
  --requirements_file TEXT    If provided, the created virtual environment is
                              installed with the libraries specified in this
                              file. If not, the activated virtual environment
                              is cloned through a temporary 'pip list' file.
                              If you are using editable package, it is
                              recommended you attach your own requirement file
                              using ``pip freeze``
  --extra_files TEXT          To upload extra files which can be called from
                              your main python file (or from the shell file).
  --config_file TEXT          To load job configuration from a file.
  --num_cores TEXT            Set the amount of CPU cores reserved for the
                              job. By default it is 1 CPU.
  --memory TEXT               Set the amount of memory RAM in MB reserved for
                              the job. By default it is 100 MB.
  --disk_space TEXT           Set the amount of hard drive space in MB
                              reserved for the job. By default it is 100 MB.
  --exclusive TEXT            Set the job to run in a machine exclusively,
                              without sharing it with other jobs. By default
                              it is False
  --max_execution_time TEXT   Set the maximum execution time for the job. By
                              default it is zero (unlimited).
  --wait TEXT                 Set the terminal to wait for job completion
                              before return the control to the user.
  --save_config_file BOOLEAN  Writes the configuration to the config file,
                              after successfully submit the job. It overwrites
                              the configuration file. The configuration file
                              path is given using ``config_file`` argument.
  --debug BOOLEAN             Set PyMAPDL to prints the debug logging to the
                              console output.
  --help                      Show this message and exit.

It works

ansys-reviewer-bot · 2024-05-14T15:41:33Z

Thanks for opening a Pull Request. If you want to perform a review write a comment saying:

@ansys-reviewer-bot review

codecov-commenter · 2024-05-14T15:46:46Z

Codecov Report

Attention: Patch coverage is 6.29371% with 536 lines in your changes missing coverage. Please review.

Project coverage is 79.74%. Comparing base (3dcb798) to head (a7adf0b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3091      +/-   ##
==========================================
- Coverage   86.61%   79.74%   -6.88%     
==========================================
  Files          52       54       +2     
  Lines        9528    10100     +572     
==========================================
- Hits         8253     8054     -199     
- Misses       1275     2046     +771

koubaa · 2024-05-14T16:34:53Z

src/ansys/mapdl/core/cli/hpc.py

+        )
+
+    if config_file is None:
+        config_file = os.path.join(os.getcwd(), "hps_config.json")


@germa89 should the config be in appdirs instead?

MMhhh... if we use a file in the root directory, we could point to different files.... and hence multiple configurations.

But maybe you are right... maybe it is better to use appdirs.... buttt... it might be a bit hidden for the user...

And we have many options to be keep in the file:

{ "url": "https://10.231.106.91:3000/hps", "user": "repuser", "password": "repuser", "python": "3.9", "name": "my job", "num_cores": 1, "memory": 100, "disk_space": 100, "exclusive": false, "max_execution_time": 1000 }

keeping so many important options hidden in a appdir path might bring some complications.

@germa89 use keyring for user/password. I don't think its secure enough to put that in plaintext like this. For everything else I would agree

koubaa · 2024-05-14T16:35:41Z

src/ansys/mapdl/core/cli/hpc.py

+        max_execution_time=max_execution_time,
+    )
+
+    if save_config_file:


@germa89 ideally there could be multiple configurations that can be saved so that users can switch between clusters easily.

You can easily change the file path in config_file to point to other configurations

src/ansys/mapdl/core/hpc/pyhps.py

koubaa · 2024-05-14T16:36:55Z

src/ansys/mapdl/core/hpc/pyhps.py

+    max_execution_time: int = None,
+):
+
+    if python not in [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12]:


why not 3.8+?

you mean 3.8.X ??

In theory, only major and minor needs to be specified in the bash command (python3.12 -m venv...), hence we do not need to use the "patch" number.

@germa89 no, I mean, why bother supporting python < 3.8

Some hpc clusters have old python versions. So i dont want to limitate the compatibility. But probably, HPS might force you to some python version minimal requirement.

koubaa · 2024-05-14T16:37:39Z

src/ansys/mapdl/core/hpc/pyhps.py

+        requirements_file = create_tmp_file("requirements.txt", content)
+        logger.debug(f"Requirements file in: {requirements_file}")
+
+    if not shell_file:


Shell file is needed because pyhps doesn't support python jobs with a venv. I think we should see if that feature can be added to pyhps itself. Maybe using pip-run

I agree we need to support venv within pyhps. Using pip-run seems interesting. For flexibility I would keep an optional shell file though.

koubaa · 2024-05-14T16:38:04Z

src/ansys/mapdl/core/cli/hpc.py

+    requirements_file: str = None,
+    extra_files: Optional[Union[str, list]] = None,
+    config_file: str = None,
+    num_cores: int = None,


option to download the result files?

I am not aware if we can do that from the PyHPS library... i will check it out.

Yes, It can be done.

https://hps.docs.pyansys.com/version/stable/api/_autosummary/ansys.hps.client.jms.api.ProjectApi.download_file.html#download-file

src/ansys/mapdl/core/cli/__init__.py

src/ansys/mapdl/core/cli/hpc.py

src/ansys/mapdl/core/hpc/pyhps.py

Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com>

src/ansys/mapdl/core/cli/hpc.py

Adding project url print

germa89 · 2024-05-21T08:47:14Z

Follow up PRs:
- Adding inputs and outputs #3112
- Adding APDL jobs support #3111

(To be merged from top to bottom to main)

* Adding inputs and outputs * Changing arguments order * Apply suggestions from code review Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com> * Update src/ansys/mapdl/core/cli/hpc.py --------- Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com>

* Adding inputs and outputs * Supporting pure APDL jobs * Changing arguments order * Allowing force mode * Apply suggestions from code review Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com> --------- Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com>

* Adding inputs and outputs * Supporting pure APDL jobs * Allowing force mode * Initial implementation. Making task object detachable * Apply suggestions from code review Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com> * Added job definition bypasser * Adapting CLI * Adding option for output and wait * fixing python and cpu issues * Using dict for taskdefinition Allowing outputs in apdl mode * Adding `close_client` method --------- Co-authored-by: Kathy Pippert <84872299+PipKat@users.noreply.github.com>

PipKat · 2024-06-05T18:23:06Z

src/ansys/mapdl/core/cli/__init__.py

+
+pymapdl convert input_file.inp -o output_file.out ...
+
+For more information please visit: https://mapdl.docs.pyansys.com/version/dev/user_guide/cli.html


Suggested change

For more information please visit: https://mapdl.docs.pyansys.com/version/dev/user_guide/cli.html

For more information, see `PyMAPDL command line interface <https://mapdl.docs.pyansys.com/version/dev/user_guide/cli.html>`_.

Is there a reason, you don't use a :ref: role for a link?

PipKat · 2024-06-05T18:23:49Z

src/ansys/mapdl/core/cli/hpc.py

+    help="""URL where the HPS cluster is deployed. For example: "https://myserver:3000/hps" """,
+)
+@click.option(
+    "--user", default=None, type=str, help="Username to login into the HPC cluster."


Suggested change

"--user", default=None, type=str, help="Username to login into the HPC cluster."

"--user", default=None, type=str, help="Username for logging into the HPC cluster."

PipKat · 2024-06-05T18:24:55Z

src/ansys/mapdl/core/cli/hpc.py

+Input arguments for the simulation. You can specify several arguments by
+joining them with commas. Thus, strings defined in this way cannot contain
+commas. Only integers, floats and strings are allowed.


Suggested change

Input arguments for the simulation. You can specify several arguments by

joining them with commas. Thus, strings defined in this way cannot contain

commas. Only integers, floats and strings are allowed.

Input arguments for the simulation. Because you can specify several arguments by

joining them with commas, strings defined in this way cannot contain

commas. Only integers, floats, and strings are allowed.

PipKat · 2024-06-05T18:25:41Z

src/ansys/mapdl/core/cli/hpc.py

+    help="""Output files to monitor. Because you use commas to separate
+the file names, the names cannot contain commas. For example,


Suggested change

help="""Output files to monitor. Because you use commas to separate

the file names, the names cannot contain commas. For example,

help="""Output files to monitor. Because you use commas to separate

filenames, the names cannot contain commas. For example,

PipKat · 2024-06-05T18:26:14Z

src/ansys/mapdl/core/cli/hpc.py

+    help="""Optional created virtual environment to install with the
+libraries specified in this requirements file. If not, the activated virtual environment is
+cloned through a temporary ``pip list`` file. If you are using an editable package,
+you should attach your own requirement file using ``pip freeze`` """,


Suggested change

you should attach your own requirement file using ``pip freeze`` """,

you should attach your own requirement file using ``pip freeze``.""",

PipKat · 2024-06-05T18:27:37Z

src/ansys/mapdl/core/cli/hpc.py

+    type=str,
+    is_flag=False,
+    flag_value=True,
+    help="""Print the output values to the terminal as json. It requires to use ``--wait`` value too. """,


Not sure I understand this: It requires to use --wait value too.

PipKat · 2024-06-05T18:28:03Z

src/ansys/mapdl/core/cli/hpc.py

+    type=bool,
+    is_flag=False,
+    flag_value=True,
+    help="""Whether PyMAPDL is to print debug logging to the console output.""",


Suggested change

help="""Whether PyMAPDL is to print debug logging to the console output.""",

help="""Whether PyMAPDL is to print debug logging to the console.""",

It doesn't seem output is needed.

PipKat · 2024-06-05T18:28:35Z

src/ansys/mapdl/core/cli/hpc.py

+        }
+
+        logger.debug(
+            f"Saving the following configuration to the config file ({config_file}):\n{config}"


Suggested change

f"Saving the following configuration to the config file ({config_file}):\n{config}"

f"Saving the following configuration to the config file ({config_file}):\n{config}."

PipKat · 2024-06-05T18:29:38Z

src/ansys/mapdl/core/cli/hpc.py

+    proj = job.project
+    if not output_to_json:
+        print(
+            f"You can check your project by visiting: {url}/projects#/projects/{proj.id}/jobs"


Messages should conclude with punctuation, but I don't enforce this when a colon is used to provide a URL or filename

germa89 added 2 commits May 13, 2024 22:43

First iteration

85d371e

Adding logging

1da9ca4

It works

germa89 added the New Feature Request or proposal for a new feature label May 14, 2024

germa89 requested a review from koubaa May 14, 2024 15:41

germa89 self-assigned this May 14, 2024

github-actions bot added the Enhancement Improve any current implemented feature label May 14, 2024

germa89 and others added 3 commits May 14, 2024 17:47

fixes to the CLI help

4f32315

Fix codacy issues

08d2055

Merge branch 'main' into feat/pyhps-support

08731de

koubaa reviewed May 14, 2024

View reviewed changes

src/ansys/mapdl/core/hpc/pyhps.py Outdated Show resolved Hide resolved

koubaa reviewed May 14, 2024

View reviewed changes