--- description: > Inspecting Prometheus metrics of e2e runs. --- # Inspecting Prometheus Metrics for e2e Runs ## Metrics Gathering During e2e Runs kcp internally exposes the same metrics as kube api-server. These are being gathered during e2e runs if the `PROMETHEUS_URL` variable is set. Here, any e2e spawned kcp server will publish a scraping configuration to prometheus for the given Prometheus URL environment variable. For e2e tests, the above environment variable is respected by the following test servers: - when a shared kcp server is spawned using `SharedKcpServer` or `PrivateKcpServer` in the `kcp.test.e2e.framework` package. - `cmd/sharded-test-server` - `cmd/test-server/kcp` ## Inspecting Metrics of e2e Runs ### GitHub To inspect metrics from GitHub e2e test runs, download the metrics from the GitHub summary page: ![GitHub summary page](github_summary.png) And execute Prometheus locally to inspect metrics: ```shell $ mkdir -p metrics && unzip -p e2e-sharded-metrics.zip | tar xvzf - -C metrics $ prometheus --storage.tsdb.path=metrics --config.file /dev/null ``` ### Prow To inspect e2e metrics from prow runs, just copy-paste the prow link from the pull request: ![PR prow link](prow_link.png) And paste it into [PromeCIeus](https://promecieus.dptools.openshift.org/): ![promecieus](promecieus.png) Once the Prometheus pod is ready, the provisioned link can be called to inspect gathered metrics. Alternatively, download the prometheus tarball from the gcsweb frontend locally and start a Prometheus instance as described above. ## Collecting Metrics Locally To collect metrics locally, a convenience script is available to download and execute a Prometheus instance. Calling `... ./hack/run-in-prometheus make ...` will download, start, collect metrics, and stop Prometheus during the test run. Example: ```shell $ WHAT=./test/e2e/virtual/apiexport TEST_ARGS="-v -run TestAPIExportAuthorizers" ./hack/run-with-prometheus.sh make test-e2e ``` A local Prometheus can then be started to inspect gathered metrics: ```shell $ ./hack/tools/prometheus --storage.tsdb.path=.prometheus_data --config.file /dev/null ```