The Post-mortem Processing & Visualization Layer
================================================

TOPSAIL post-mortem visualization relies on the `MatrixBenchmarking
<https://github.com/openshift-psap/matrix-benchmarking>`_.

MatrixBenchmarking consists of multiple components:

- the ``benchmark`` component is in charge of running various test
  configurations. MatrixBenchmarking/benchmark is configured with a
  set of settings, with one or multiple values. The execution engine
  will go through each of the possible configurations and execute it
  to capture its performance.
- the ``visualize`` component is in charge of the generation of plots
  and reports, based on the Dash and Plotly
  packages. MatrixBenchmarking/visualize is launched either against a
  single result directory, or against a directory with multiple
  results. The result directories can have been generated by TOPSAIL,
  which directly writes the relevant files (often the case there's
  only one test executed, or when the test list is a simple iteration
  over a list of configurations), or via MatrixBenchmarking/benchmark
  (when the test list has to iterate over various, dynamically defined
  settings). This component is further described below.
- the ``download`` component is in charge of downloading artifacts
  from S3, OpenShift CI or the Middleware Jenkins. Using this
  component instead of a simple scrapper allows downloading only the
  files important for the post-processing, or even only the cache
  file. This component is used when "re-plotting", that is, when
  regenerating the visualization in the CI without re-running the
  tests.
- the ``upload_lts`` component is used to upload the LTS (long term
  storage) payload and KPIs (key performance indicators) to
  OpenSearch. It is triggered at the end of a gating test.
- the ``download_lts`` component is used to download the historical
  LTS payloads and KPIs from OpenSearch. It is used in gating test
  before running the regression analyze.
- the ``analyze_lts`` component is used to check the results of a test
  against "similar" historical results. "similar" here means that the
  test results should have been executed with the same settings,
  except the so-called "comparison settings" (eg, the RHOAI version,
  the OCP version, etc). The regression analyze is done with the help
  of the `datastax-labs/hunter
  <https://github.com/datastax-labs/hunter>`_ package.

  In this document, we'll focus on the ``visualize`` component, which
  is a key part of TOPSAIL test pipelines. (So are ``analyze_lts``,
  ``download_lts`` and ``upload_lts`` for continuous performance
  testing, but they don't require much per-project customization.)

TOPSAIL/MatrixBenchmarking visualization modules are split into
two main components: the parsers (in ``store`` module) and plotters
(in ``plotting`` module). In addition to that, the continuous
performance testing (CPT) requires two extra components: the models
(in the ``models`` module) and the regression analyze preps (in the
``analyze`` module).