Figures

In order to generate reports, the Feel++ benchmarking framework requires a figure description to specify what the website report page should contain.

These descriptions should be provided either with a specific JSON file with the structure containing uniquely

{
    "plots":[]
}

Or by specifying the plots field on the benchmark configuration JSON file.

Each figure description should contain the following fields

{
    "title": "The figure title",
    "plot_types": [], //List of figure types
    "transformation": "", //Transformation type
    "variables":[], // List of variables to consider
    "names":[], //Respective labels for variables
    "yaxis":{},
    "xaxis":{},
    "color_axis":{}, //Default: performance variables
    "secondary_axis":{}
}

Figures will appear in the same order as they appear on the list.

Users can provide multiple plot_types in the same description field.

Only performance variables specified under the variables list will be considered. If the list is empty, ALL variables inside the ReFrame report will be taken into account.

1. Axis

Each axis (with the exception of the yaxis) take a parameter and a label field. The yaxis will always contain the performance values, therefore only the label key should be specified.

The parameter field of each axis should correspond to either a single dimension parameter specified on the benchmark configuration. In the case of subparameters, the syntax should be the following: parameter.subparameter.

By default, the color axis will contain the performance variables, but this can be customized.

2. Transformations

The ReFrame report will be used to create a Master DataFrame, which will contain all performance variables and their respective values, as well as all parameters and environments.

To explain how transformation and plot types work, we can consider the following example.

import json
report = json.loads("""
    { "session_info": { "cmdline": "/Users/cladellash/Documents/Repos/benchmarking/.venv/bin/reframe -C /Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/config/machineConfigs/local.py -c /Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py -S report_dir_path=/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28 --system=local --exec-policy=async --prefix=/Users/cladellash/Documents/Repos/benchmarking/build/reframe --report-file=/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28/reframe_report.json -J '#SBATCH --time=0-0:5:0' --perflogdir=/Users/cladellash/Documents/Repos/benchmarking/build/reframe/logs -v -r", "config_files": [ "<builtin>", "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/config/machineConfigs/local.py" ], "data_version": "3.1", "hostname": "irma-dhcp-2.math.unistra.fr", "log_files": [ "/var/folders/pd/r8v9chs90wb1bj6pm0x147jr0000gp/T/rfm-qml5j4xz.log" ], "prefix_output": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output", "prefix_stage": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage", "user": "cladellash", "version": "4.6.3", "workdir": "/Users/cladellash/Documents/Repos/benchmarking", "time_start": "2024-12-02T14:46:29+0100", "time_end": "2024-12-02T14:46:40+0100", "time_elapsed": 11.50057601928711, "num_cases": 12, "num_failures": 0 }, "runs": [ { "num_cases": 12, "num_failures": 0, "num_aborted": 0, "num_skipped": 0, "runid": 0, "testcases": [ { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=1000000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "aa223ed0", "jobid": "99178", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=1000000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_aa223ed0", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 5.32933 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.052584 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 4.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_aa223ed0", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.006192207336425781, "time_performance": 0.008784055709838867, "time_run": 9.518624067306519, "time_sanity": 0.007528066635131836, "time_setup": 0.00566411018371582, "time_total": 9.696507215499878, "unique_name": "RegressionTest_11", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "1000000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/aa223ed0" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 4, "num_tasks_per_node": 4, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 4, "exclusive_access": true }, "elements": 1000000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=700000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "6a58f265", "jobid": "99179", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=700000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_6a58f265", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 4.56971 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.005627 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 4.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_6a58f265", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.007662057876586914, "time_performance": 0.038092851638793945, "time_run": 8.129333019256592, "time_sanity": 0.03859901428222656, "time_setup": 0.009199142456054688, "time_total": 8.30247712135315, "unique_name": "RegressionTest_10", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "700000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/6a58f265" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 4, "num_tasks_per_node": 4, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 4, "exclusive_access": true }, "elements": 700000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=400000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "e8955ef5", "jobid": "99180", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=400000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_e8955ef5", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 3.85582 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.180105 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 4.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_e8955ef5", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.008015155792236328, "time_performance": 0.03525996208190918, "time_run": 7.037631988525391, "time_sanity": 0.03978276252746582, "time_setup": 0.01161503791809082, "time_total": 7.199726104736328, "unique_name": "RegressionTest_09", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "400000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/e8955ef5" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 4, "num_tasks_per_node": 4, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 4, "exclusive_access": true }, "elements": 400000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=100000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "9886e190", "jobid": "99181", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 4, 'exclusive_access': True} %elements=100000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_9886e190", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.062597 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.0161 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 4.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_9886e190", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.006299018859863281, "time_performance": 0.017982006072998047, "time_run": 1.7802720069885254, "time_sanity": 0.014677762985229492, "time_setup": 0.005388975143432617, "time_total": 1.927483081817627, "unique_name": "RegressionTest_08", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "100000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/9886e190" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 4, "num_tasks_per_node": 4, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 4, "exclusive_access": true }, "elements": 100000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=1000000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "86092ceb", "jobid": "99182", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=1000000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_86092ceb", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 5.13016 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.000502 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 2.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_86092ceb", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.00612187385559082, "time_performance": 0.002651214599609375, "time_run": 10.523810148239136, "time_sanity": 0.0021507740020751953, "time_setup": 0.005548954010009766, "time_total": 10.664539098739624, "unique_name": "RegressionTest_07", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "1000000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/86092ceb" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 2, "num_tasks_per_node": 2, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 2, "exclusive_access": true }, "elements": 1000000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=700000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "b22a7385", "jobid": "99183", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=700000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_b22a7385", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 5.06941 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.03559 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 2.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_b22a7385", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.00615692138671875, "time_performance": 0.015397310256958008, "time_run": 9.810746192932129, "time_sanity": 0.025763988494873047, "time_setup": 0.006819963455200195, "time_total": 9.945525169372559, "unique_name": "RegressionTest_06", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "700000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/b22a7385" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 2, "num_tasks_per_node": 2, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 2, "exclusive_access": true }, "elements": 700000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=400000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "7dd0d8fb", "jobid": "99184", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=400000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_7dd0d8fb", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 4.70458 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.129757 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 2.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_7dd0d8fb", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.006106138229370117, "time_performance": 0.00861215591430664, "time_run": 8.266666889190674, "time_sanity": 0.006582021713256836, "time_setup": 0.005933046340942383, "time_total": 8.394183158874512, "unique_name": "RegressionTest_05", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "400000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/7dd0d8fb" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 2, "num_tasks_per_node": 2, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 2, "exclusive_access": true }, "elements": 400000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=100000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "e75954a0", "jobid": "99185", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 2, 'exclusive_access': True} %elements=100000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_e75954a0", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.183174 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.003264 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 2.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_e75954a0", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.00605320930480957, "time_performance": 0.019010066986083984, "time_run": 1.770622968673706, "time_sanity": 0.0030341148376464844, "time_setup": 0.00586390495300293, "time_total": 1.8920440673828125, "unique_name": "RegressionTest_04", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "100000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/e75954a0" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 2, "num_tasks_per_node": 2, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 2, "exclusive_access": true }, "elements": 100000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=1000000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "cbfe221b", "jobid": "99225", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=1000000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_cbfe221b", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 4.8334 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 2.1e-05 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1000000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_cbfe221b", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.005920886993408203, "time_performance": 0.0037949085235595703, "time_run": 9.820374727249146, "time_sanity": 0.003256082534790039, "time_setup": 0.005385160446166992, "time_total": 11.37114691734314, "unique_name": "RegressionTest_03", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "1000000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/cbfe221b" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 1, "num_tasks_per_node": 1, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 1, "exclusive_access": true }, "elements": 1000000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=700000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "be4af6da", "jobid": "99226", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=700000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_be4af6da", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 4.90371 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 2.7e-05 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 700000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_be4af6da", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.005860805511474609, "time_performance": 0.0030002593994140625, "time_run": 9.275190114974976, "time_sanity": 0.002341032028198242, "time_setup": 0.005341053009033203, "time_total": 10.844825267791748, "unique_name": "RegressionTest_02", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "700000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/be4af6da" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 1, "num_tasks_per_node": 1, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 1, "exclusive_access": true }, "elements": 700000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=400000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "e8e66601", "jobid": "99278", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=400000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_e8e66601", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 2.24316 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 3.2e-05 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 400000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_e8e66601", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.0059511661529541016, "time_performance": 0.002692699432373047, "time_run": 4.065032005310059, "time_sanity": 0.002173185348510742, "time_setup": 0.005150794982910156, "time_total": 10.587581872940063, "unique_name": "RegressionTest_01", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "400000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/e8e66601" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 1, "num_tasks_per_node": 1, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 1, "exclusive_access": true }, "elements": 400000000.0 } }, { "build_stderr": null, "build_stdout": null, "dependencies_actual": [], "dependencies_conceptual": [], "description": "", "display_name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=100000000.0", "environment": "default", "fail_phase": null, "fail_reason": null, "filename": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe/regression.py", "fixture": false, "hash": "da2298ca", "jobid": "99283", "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "maintainers": [], "name": "RegressionTest %nb_tasks={'tasks': 1, 'exclusive_access': True} %elements=100000000.0", "nodelist": [ "irma-dhcp-2.math.unistra.fr" ], "outputdir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/output/local/default/default/RegressionTest_da2298ca", "perfvars": [ { "name": "computation_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 0.622329 }, { "name": "communication_time", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "s", "value": 3.2e-05 }, { "name": "N", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "sum", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 100000000.0 }, { "name": "num_process", "reference": 0, "thres_lower": null, "thres_upper": null, "unit": "", "value": 1.0 } ], "prefix": "/Users/cladellash/Documents/Repos/benchmarking/.venv/lib/python3.8/site-packages/feelpp/benchmarking/reframe", "result": "success", "stagedir": "/Users/cladellash/Documents/Repos/benchmarking/build/reframe/stage/local/default/default/RegressionTest_da2298ca", "scheduler": "local", "system": "local:default", "tags": [ "async" ], "time_compile": 0.005639791488647461, "time_performance": 0.008454084396362305, "time_run": 1.732982873916626, "time_sanity": 0.0396878719329834, "time_setup": 0.005467891693115234, "time_total": 9.557442903518677, "unique_name": "RegressionTest_00", "check_vars": { "valid_prog_environs": [ "default" ], "valid_systems": [ "local:default" ], "descr": "", "sourcepath": "", "sourcesdir": null, "prebuild_cmds": [], "postbuild_cmds": [], "executable": "/Users/cladellash/Documents/Repos/benchmarking//tests/data/parallelSum", "executable_opts": [ "100000000.0", "/Users/cladellash/Documents/Repos/benchmarking/tests/data/outputs/parallelSum/da2298ca" ], "prerun_cmds": [], "postrun_cmds": [], "keep_files": [], "readonly_files": [], "tags": [ "async" ], "maintainers": [], "strict_check": true, "num_tasks": 1, "num_tasks_per_node": 1, "num_gpus_per_node": null, "num_cpus_per_task": 1, "num_tasks_per_core": null, "num_tasks_per_socket": null, "use_multithreading": null, "max_pending_time": null, "exclusive_access": true, "local": false, "modules": [], "env_vars": {}, "variables": {}, "time_limit": null, "build_time_limit": null, "extra_resources": {}, "build_locally": true, "report_dir_path": "/Users/cladellash/Documents/Repos/benchmarking/reports/parallelSum/parallel_sum/local/2024_12_02T14_46_28", "use_case": "parallel_sum", "platform": "builtin" }, "check_params": { "nb_tasks": { "tasks": 1, "exclusive_access": true }, "elements": 100000000.0 } } ] } ], "restored_cases": [] }
""")

from feelpp.benchmarking.report.atomicReports.model import AtomicReportModel

model = AtomicReportModel(report["runs"])
model.master_df = model.master_df[model.master_df["performance_variable"].isin(["computation_time","communication_time"])].loc[:,["performance_variable","value","unit",	"testcase_time_run","environment","platform","nb_tasks.tasks","nb_tasks.exclusive_access","elements"]]

Out[1]:
   performance_variable     value  ... nb_tasks.exclusive_access      elements
0      computation_time  5.329330  ...                      True  1.000000e+09
1    communication_time  0.052584  ...                      True  1.000000e+09
5      computation_time  4.569710  ...                      True  7.000000e+08
6    communication_time  0.005627  ...                      True  7.000000e+08
10     computation_time  3.855820  ...                      True  4.000000e+08
11   communication_time  0.180105  ...                      True  4.000000e+08
15     computation_time  0.062597  ...                      True  1.000000e+08
16   communication_time  0.016100  ...                      True  1.000000e+08
20     computation_time  5.130160  ...                      True  1.000000e+09
21   communication_time  0.000502  ...                      True  1.000000e+09
25     computation_time  5.069410  ...                      True  7.000000e+08
26   communication_time  0.035590  ...                      True  7.000000e+08
30     computation_time  4.704580  ...                      True  4.000000e+08
31   communication_time  0.129757  ...                      True  4.000000e+08
35     computation_time  0.183174  ...                      True  1.000000e+08
36   communication_time  0.003264  ...                      True  1.000000e+08
40     computation_time  4.833400  ...                      True  1.000000e+09
41   communication_time  0.000021  ...                      True  1.000000e+09
45     computation_time  4.903710  ...                      True  7.000000e+08
46   communication_time  0.000027  ...                      True  7.000000e+08
50     computation_time  2.243160  ...                      True  4.000000e+08
51   communication_time  0.000032  ...                      True  4.000000e+08
55     computation_time  0.622329  ...                      True  1.000000e+08
56   communication_time  0.000032  ...                      True  1.000000e+08

[24 rows x 9 columns]

We can see that this dataframe contains the parameters: - environment - platform - nb_tasks.tasks - nb_tasks.exclusive_access - elements - performance_variable

By having this common structure, we can make use of transformation strategies to manipulate values depending on the desired output.

Strategies will depend on the figure axis. All strategies will create a pivot dataframe that will contain the parameter specified as color_axis as columns, xaxis as first level index and secondary_axis as second level index. Values of the dataframe will always be the values of the master dataframe.

As an example, we will consider the following axis:

"xaxis":{
    "parameter":"nb_tasks.tasks",
    "label":"Number of tasks"
},
"yaxis":{
    "label":"Execution time (s)"
},
"secondary_axis":{
    "parameter":"elements",
    "label":"N"
},
"color_axis":{
    "parameter":"performance_variable",
    "label":"Performance variable"
}

Available strategies are:

performance

This strategy should be seen as the "base" strategy. No transformation, other that a pivot, is done. For the given example, it produces the following dataframe

from feelpp.benchmarking.report.transformationFactory import TransformationStrategyFactory
from feelpp.benchmarking.reframe.config.configPlots import Plot
plot_config = Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    }
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(model.master_df)
print(df)

performance_variable         communication_time  computation_time
elements     nb_tasks.tasks
1.000000e+08 1                         0.000032          0.622329
             2                         0.003264          0.183174
             4                         0.016100          0.062597
4.000000e+08 1                         0.000032          2.243160
             2                         0.129757          4.704580
             4                         0.180105          3.855820
7.000000e+08 1                         0.000027          4.903710
             2                         0.035590          5.069410
             4                         0.005627          4.569710
1.000000e+09 1                         0.000021          4.833400
             2                         0.000502          5.130160
             4                         0.052584          5.329330

relative_performance

The relative performance strategy computes the proportion of the time that a a color_axis variable takes with regards of the total.

from feelpp.benchmarking.report.transformationFactory import TransformationStrategyFactory
from feelpp.benchmarking.reframe.config.configPlots import Plot
plot_config = Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "transformation": "relative_performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    }
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(model.master_df)
print(df)

performance_variable         communication_time  computation_time
elements     nb_tasks.tasks
1.000000e+08 1                         0.005142         99.994858
             2                         1.750716         98.249284
             4                        20.458213         79.541787
4.000000e+08 1                         0.001427         99.998573
             2                         2.684070         97.315930
             4                         4.462546         95.537454
7.000000e+08 1                         0.000551         99.999449
             2                         0.697160         99.302840
             4                         0.122985         99.877015
1.000000e+09 1                         0.000434         99.999566
             2                         0.009784         99.990216
             4                         0.977050         99.022950

The sum along the column axis will always be equal to 1.

speedup

The speedup strategy computes the speedup of the color_axis variables. The minimum of the xaxis values is taken as the base of the speedup. For the example, this strategy will produce the following.

from feelpp.benchmarking.report.transformationFactory import TransformationStrategyFactory
from feelpp.benchmarking.reframe.config.configPlots import Plot
plot_config = Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "transformation": "speedup",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    }
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(model.master_df)
print(df)

performance_variable         communication_time  ...  half-optimal
elements     nb_tasks.tasks                      ...
1.000000e+08 1                         1.000000  ...           1.0
             2                         0.009804  ...           1.5
             4                         0.001988  ...           2.5
4.000000e+08 1                         1.000000  ...           1.0
             2                         0.000247  ...           1.5
             4                         0.000178  ...           2.5
7.000000e+08 1                         1.000000  ...           1.0
             2                         0.000759  ...           1.5
             4                         0.004798  ...           2.5
1.000000e+09 1                         1.000000  ...           1.0
             2                         0.041833  ...           1.5
             4                         0.000399  ...           2.5

[12 rows x 4 columns]

3. Plot types

Considering the same example axis as above, the software can generate the following figures:

scatter

from feelpp.benchmarking.report.figures.figureFactory import FigureFactory
figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "scatter" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "color_axis":{
    "parameter":"performance_variable",
    "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

marked_scatter

The marked scatter plot type supports from 2 to 4 dimensions. The symbol/marks axis will correspond to the secondary_axis parameter. This plot type will behave as follows:

If 1 or 2 dimensions are specified (x-axis and optionally color-axis), then this plot type will be equivalent to scatter.
If 3 dimensions are specified (x-axis, color-axis and secondary-axis), then the secondary_axis will correspond to the symbol/marks axis.
If 4 dimensions are specified (x-axis, color-axis, secondary-axis and one extra-axis), then the first element of the extra_axes list will correspond to the symbol/marks axis, and the secondary_axis will correspond to the slider of the returned animation.

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "marked_scatter" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
for f in figures:
    fig = f.createFigure(model.master_df)
    fig.show()

stacked_bar

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "color_axis":{
    "parameter":"performance_variable",
    "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

grouped_bar

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "grouped_bar" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "color_axis":{
    "parameter":"performance_variable",
    "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

heatmap

For this case, we will consider the elements (N) as color_axis and performance_variable for secondary axis (slider).

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "heatmap" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

table

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "table" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation","Comunication"],
    "color_axis":{
    "parameter":"performance_variable",
    "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

sunburst

This figure considers the color_axis parameter as the outer-most ring. Users can supply an extra_axes field, containing a list of additional parameters. Values for these parameters whill be shown on the rings that follow the color_axis ring, in the order they are provided. The secondary_axis and xaxis parameter are present respectively on the inner-most and second inner-most rings.

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "sunburst" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
fig = figures[0].createFigure(model.master_df)
fig.show()

parallelcoordinates

Axes will be shown on the following order: secondary_axis, xaxis, all additional extra_axes, color_axis. The yaxis will be shown in the line color.

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "parallelcoordinates" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    },
    "extra_axes":[
        {
            "parameter":"dim3",
            "label":"Dim3"
        }
    ]
}))
for f in figures:
    fig = f.createFigure(model.master_df)
    fig.show()

3.1. 3D Plots

3D plots are also supported, and then can show up to 4 dimensions. At least 3 parameters must be provided (xaxis,color_axis and secondary_axis ). Axes correspondance is as follows:

x-axis of the 3D plot: xaxis
y-axis of the 3D plot: secondary_axis if no extra axes are provided, else, the first element of the extra_axes list.
z-axis of the 3D plot: yaxis (contains the measured values)
color of the 3D traces: color_axis
Slider: secondary_axis if extra axes are provided.
scatter3d

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "scatter3d" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
for f in figures:
    fig = f.createFigure(model.master_df)
    fig.show()

surface3d

figures = FigureFactory.create(Plot(**{
    "title": "Absolute performance",
    "plot_types": [ "surface3d" ],
    "transformation": "performance",
    "variables": [ "computation_time","communication_time" ],
    "names": ["Computation", "Communication"],
    "color_axis":{
        "parameter":"performance_variable",
        "label":"Performance variable"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{
        "parameter":"elements",
        "label":"N"
    },
    "xaxis":{
        "parameter":"nb_tasks.tasks",
        "label":"Number of tasks"
    }
}))
for f in figures:
    fig = f.createFigure(model.master_df)
    fig.show()

4. Aggregations

Depending on the dashboard level that we are located at, it might be necessary to aggregate the data on the master dataframe. For example, if we have all use cases, applications and machines on the dataframe, and we want to see how a certain use case performs on different machines, we can make use of the aggregations field to group the data accordingly.

"aggregations":[
    {"column":"date","agg":"max"},
    {"column":"applications","agg":"filter:my_app"},
    {"column":"use_cases","agg":"filter:my_use_case"},
    {"column":"performance_variable","agg":"sum"}
]

The previous example will first get only the latest benchmarks (by getting the maximum date), then it will filter the application and the use case to find applications and use cases that correspond to "my_app" and "my_use_case". And finally it will compute the sum of all performance variables for the remaining rows.

Users must provide a column and an aggregation function as a string.

Available aggregations are:

mean : Computes the mean of the column
mean : Computes the sum of the column
max : Computes the maximum of the column
min : Computes the minimum of the column
filter:value: Filters the column by value.

The order of the aggregations list is important.

5. Custom layouts

By providing the layout_modifiers field, users can pass custom layout options for rendering the figures. These options correspond to the accepted layout reference for Plotly: Plotly layout reference It accepts a nested dictionnary just as Plotly does.

For example, we could customize a figure to have have its x-axis on a logscale.

"layout_modifiers":{
    "xaxis":{
        "type":"log"
    }
}