> ## Documentation Index
> Fetch the complete documentation index at: https://cantonfoundation-feature-integrate-google-analytics.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Splice Metrics Overview

> Prometheus metrics exposed by Canton Network components and how to scrape them

export const networkData = {
  mainnet: {
    name: 'MainNet',
    description: 'Production network for live applications. Data is permanent and never reset. Upgrades follow DevNet and TestNet validation.',
    color: '#22c55e',
    versions: {
      splice: '0.5.18',
      damlSdk: '3.4.12',
      pqs: '3.4.6',
      tokenStandard: '1.0.0',
      walletSdk: '1.1.0',
      dappSdk: '1.1.0',
      walletGateway: '1.2.0'
    },
    advanced: {
      minProtocolVersion: '6',
      migrationId: '4',
      darVersions: [{
        name: 'splice-amulet',
        version: '0.1.14'
      }, {
        name: 'splice-wallet',
        version: '0.1.14'
      }, {
        name: 'splice-dso-governance',
        version: '0.1.20'
      }],
      releaseUrl: 'https://github.com/canton-network/splice/releases/tag/0.5.18'
    },
    endpoint: 'scan.sv-1.global.canton.network.sync.global',
    substitutions: {
      splice_cluster: 'main',
      da_hostname: 'global.canton.network.digitalasset.com',
      gsf_sv_url: 'https://sv.sv-1.global.canton.network.sync.global',
      generic_sv_url: 'https://sv.sv-1.global.canton.network.YOUR_SV_SPONSOR',
      gsf_scan_url: 'https://scan.sv-1.global.canton.network.sync.global',
      generic_scan_url: 'https://scan.sv-1.global.canton.network.YOUR_SV_SPONSOR',
      gsf_sequencer_url: 'https://sequencer-MIGRATION_ID.sv-1.global.canton.network.sync.global',
      version: '0.5.18',
      version_literal: '0.5.18',
      chart_version_literal: '0.5.18',
      chart_version_set: 'export CHART_VERSION=0.5.18',
      image_tag_set: 'export IMAGE_TAG=0.5.18',
      image_tag_set_plain: 'export IMAGE_TAG=0.5.18',
      bundle_download_link: {
        label: 'Download Bundle',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.5.18/0.5.18_splice-node.tar.gz'
      },
      openapi_download_link: {
        label: 'Download OpenAPI specs',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.5.18/0.5.18_openapi.tar.gz'
      },
      helm_repo_prefix: 'oci://ghcr.io/digital-asset/decentralized-canton-sync/helm',
      docker_repo_prefix: 'ghcr.io/digital-asset/decentralized-canton-sync/docker'
    }
  },
  testnet: {
    name: 'TestNet',
    description: 'Pre-production environment for final validation. Requires MainNet approval to join. May be reset periodically.',
    color: '#eab308',
    versions: {
      splice: '0.6.2',
      damlSdk: '3.5.1',
      pqs: '3.5.1-rc6',
      tokenStandard: '1.0.0',
      walletSdk: '1.1.0',
      dappSdk: '1.1.0',
      walletGateway: '1.2.0'
    },
    advanced: {
      minProtocolVersion: '6',
      migrationId: '1',
      darVersions: [{
        name: 'splice-amulet',
        version: '0.1.15'
      }, {
        name: 'splice-wallet',
        version: '0.1.15'
      }, {
        name: 'splice-dso-governance',
        version: '0.1.21'
      }],
      releaseUrl: 'https://github.com/canton-network/splice/releases/tag/0.6.2'
    },
    endpoint: 'scan.sv-1.test.global.canton.network.sync.global',
    substitutions: {
      splice_cluster: 'test',
      da_hostname: 'test.global.canton.network.digitalasset.com',
      gsf_sv_url: 'https://sv.sv-1.test.global.canton.network.sync.global',
      generic_sv_url: 'https://sv.sv-1.test.global.canton.network.YOUR_SV_SPONSOR',
      gsf_scan_url: 'https://scan.sv-1.test.global.canton.network.sync.global',
      generic_scan_url: 'https://scan.sv-1.test.global.canton.network.YOUR_SV_SPONSOR',
      gsf_sequencer_url: 'https://sequencer-MIGRATION_ID.sv-1.test.global.canton.network.sync.global',
      version: '0.6.2',
      version_literal: '0.6.2',
      chart_version_literal: '0.6.2',
      chart_version_set: 'export CHART_VERSION=0.6.2',
      image_tag_set: 'export IMAGE_TAG=0.6.2',
      image_tag_set_plain: 'export IMAGE_TAG=0.6.2',
      bundle_download_link: {
        label: 'Download Bundle',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.6.2/0.6.2_splice-node.tar.gz'
      },
      openapi_download_link: {
        label: 'Download OpenAPI specs',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.6.2/0.6.2_openapi.tar.gz'
      },
      helm_repo_prefix: 'oci://ghcr.io/digital-asset/decentralized-canton-sync/helm',
      docker_repo_prefix: 'ghcr.io/digital-asset/decentralized-canton-sync/docker'
    }
  },
  devnet: {
    name: 'DevNet',
    description: 'Development environment with latest features. Open to any validator (IP allowlist required). Reset every 3 months. Best for testing upgrades.',
    color: '#a78bfa',
    versions: {
      splice: '0.6.4',
      damlSdk: '3.5.1',
      pqs: '3.5.1-rc6',
      tokenStandard: '1.0.0',
      walletSdk: '1.1.0',
      dappSdk: '1.1.0',
      walletGateway: '1.2.0'
    },
    advanced: {
      minProtocolVersion: '6',
      migrationId: '1',
      darVersions: [{
        name: 'splice-amulet',
        version: '0.1.15'
      }, {
        name: 'splice-wallet',
        version: '0.1.15'
      }, {
        name: 'splice-dso-governance',
        version: '0.1.21'
      }],
      releaseUrl: 'https://github.com/canton-network/splice/releases/tag/0.6.4'
    },
    endpoint: 'scan.sv-1.dev.global.canton.network.sync.global',
    substitutions: {
      splice_cluster: 'dev',
      da_hostname: 'dev.global.canton.network.digitalasset.com',
      gsf_sv_url: 'https://sv.sv-1.dev.global.canton.network.sync.global',
      generic_sv_url: 'https://sv.sv-1.dev.global.canton.network.YOUR_SV_SPONSOR',
      gsf_scan_url: 'https://scan.sv-1.dev.global.canton.network.sync.global',
      generic_scan_url: 'https://scan.sv-1.dev.global.canton.network.YOUR_SV_SPONSOR',
      gsf_sequencer_url: 'https://sequencer-MIGRATION_ID.sv-1.dev.global.canton.network.sync.global',
      version: '0.6.4',
      version_literal: '0.6.4',
      chart_version_literal: '0.6.4',
      chart_version_set: 'export CHART_VERSION=0.6.4',
      image_tag_set: 'export IMAGE_TAG=0.6.4',
      image_tag_set_plain: 'export IMAGE_TAG=0.6.4',
      bundle_download_link: {
        label: 'Download Bundle',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.6.4/0.6.4_splice-node.tar.gz'
      },
      openapi_download_link: {
        label: 'Download OpenAPI specs',
        href: 'https://github.com/digital-asset/decentralized-canton-sync/releases/download/v0.6.4/0.6.4_openapi.tar.gz'
      },
      helm_repo_prefix: 'oci://ghcr.io/digital-asset/decentralized-canton-sync/helm',
      docker_repo_prefix: 'ghcr.io/digital-asset/decentralized-canton-sync/docker'
    }
  }
};

export const NetworkVariables = ({children, title = null, description, defaultNetwork = 'devnet', networkData = {}}) => {
  const deploymentNetworkOrder = ['devnet', 'testnet', 'mainnet'];
  const tokenPattern = /\|([A-Za-z0-9_]+)\|/g;
  const tokenTextValue = value => {
    if (value && typeof value === 'object' && ('href' in value)) {
      return value.href;
    }
    return value ?? null;
  };
  const replacementForToken = (token, substitutions) => tokenTextValue(substitutions?.[token]);
  const replaceStringProp = (value, substitutions) => {
    if (typeof value !== 'string') {
      return value;
    }
    return value.replace(tokenPattern, (match, token) => replacementForToken(token, substitutions) ?? match);
  };
  const replaceTextNode = (value, substitutions, keyPrefix = 'token') => {
    if (typeof value !== 'string') {
      return value;
    }
    const pieces = [];
    let lastIndex = 0;
    let match;
    tokenPattern.lastIndex = 0;
    while ((match = tokenPattern.exec(value)) !== null) {
      const [rawToken, token] = match;
      const replacement = substitutions?.[token];
      if (!replacement) {
        continue;
      }
      if (match.index > lastIndex) {
        pieces.push(value.slice(lastIndex, match.index));
      }
      if (replacement && typeof replacement === 'object' && replacement.href) {
        const linkLabel = replacement.label ?? replacement.href;
        const networkLabel = activeData?.versions?.splice ? `${activeData.name ?? activeNetwork} ${activeData.versions.splice}` : activeData?.name ?? activeNetwork;
        pieces.push(<a key={`${keyPrefix}-${pieces.length}`} href={replacement.href}>
            {networkLabel ? `${linkLabel} (${networkLabel})` : linkLabel}
          </a>);
      } else {
        pieces.push(String(replacement));
      }
      lastIndex = match.index + rawToken.length;
    }
    if (!pieces.length) {
      return value;
    }
    if (lastIndex < value.length) {
      pieces.push(value.slice(lastIndex));
    }
    return pieces;
  };
  const replaceNode = (node, substitutions, keyPrefix = 'node') => {
    if (typeof node === 'string') {
      return replaceTextNode(node, substitutions, keyPrefix);
    }
    if (Array.isArray(node)) {
      return node.map((child, index) => replaceNode(child, substitutions, `${keyPrefix}-${index}`));
    }
    if (!React.isValidElement(node)) {
      return node;
    }
    const networkOnly = node.props?.['data-network-only'];
    if (typeof networkOnly === 'string') {
      const allowedNetworks = networkOnly.split(',').map(network => network.trim()).filter(Boolean);
      if (!allowedNetworks.includes(activeNetwork)) {
        return null;
      }
    }
    const props = {};
    for (const [name, value] of Object.entries(node.props ?? ({}))) {
      if (name === 'children') {
        continue;
      }
      props[name] = typeof value === 'string' ? replaceStringProp(value, substitutions) : value;
    }
    return React.cloneElement(node, props, React.Children.map(node.props.children, (child, index) => replaceNode(child, substitutions, `${keyPrefix}-${index}`)));
  };
  const options = deploymentNetworkOrder.filter(networkKey => networkData[networkKey]?.substitutions);
  const defaultIndex = Math.max(0, options.indexOf(defaultNetwork));
  const [activeIndex, setActiveIndex] = React.useState(defaultIndex);
  const activeNetwork = options[activeIndex] ?? options[0];
  const activeData = networkData[activeNetwork];
  const panelRef = React.useRef(null);
  React.useEffect(() => {
    if (!activeData?.substitutions || !panelRef.current) {
      return;
    }
    const replaceTokenText = value => value.replace(/\|([A-Za-z0-9_]+)\|/g, (match, token) => replacementForToken(token, activeData.substitutions) ?? match);
    const replaceRenderedCode = () => {
      for (const codeRoot of panelRef.current.querySelectorAll('pre, code')) {
        if (codeRoot.tagName === 'PRE' && codeRoot.querySelector('code')) {
          continue;
        }
        codeRoot.textContent = replaceTokenText(codeRoot.textContent ?? '');
      }
    };
    let animationFrame;
    const scheduleReplaceRenderedCode = () => {
      if (animationFrame) {
        window.cancelAnimationFrame(animationFrame);
      }
      animationFrame = window.requestAnimationFrame(() => {
        animationFrame = null;
        replaceRenderedCode();
      });
    };
    const observer = new MutationObserver(scheduleReplaceRenderedCode);
    const timeouts = [100, 500, 1000].map(delay => window.setTimeout(replaceRenderedCode, delay));
    replaceRenderedCode();
    observer.observe(panelRef.current, {
      childList: true,
      characterData: true,
      subtree: true
    });
    return () => {
      observer.disconnect();
      if (animationFrame) {
        window.cancelAnimationFrame(animationFrame);
      }
      for (const timeout of timeouts) {
        window.clearTimeout(timeout);
      }
    };
  }, [activeNetwork, activeData]);
  if (!activeData) {
    return null;
  }
  const label = networkKey => {
    const network = networkData[networkKey];
    const version = network?.versions?.splice;
    return version ? `${network?.name ?? networkKey} (${version})` : network?.name ?? networkKey;
  };
  return <div data-component="network-variables" className="my-6 overflow-hidden rounded-lg border border-gray-200 bg-white shadow-sm dark:border-gray-800 dark:bg-gray-950">
      {(title || description) && <div className="border-b border-gray-200 px-4 py-3 dark:border-gray-800">
          {title && <div className="text-sm font-semibold text-gray-950 dark:text-gray-100">{title}</div>}
          {description && <div className="mt-1 text-sm leading-6 text-gray-600 dark:text-gray-400">{description}</div>}
        </div>}
      <div role="tablist" className="flex gap-1 border-b border-gray-200 bg-gray-50 p-1 dark:border-gray-800 dark:bg-gray-900">
        {options.map((networkKey, index) => {
    const isActive = index === activeIndex;
    return <button key={networkKey} type="button" role="tab" aria-selected={isActive} onClick={() => setActiveIndex(index)} className={`min-h-9 px-3 text-sm font-medium transition-colors ${isActive ? 'rounded-md bg-white text-gray-950 shadow-sm dark:bg-gray-800 dark:text-gray-100' : 'rounded-md text-gray-600 hover:bg-white/80 hover:text-gray-950 dark:text-gray-400 dark:hover:bg-gray-800/70 dark:hover:text-gray-100'}`}>
              {label(networkKey)}
            </button>;
  })}
      </div>
      <div key={activeNetwork} ref={panelRef} role="tabpanel" className="space-y-4 px-4 py-4 text-sm leading-6 text-gray-800 dark:text-gray-200">
        {React.Children.map(children, (child, index) => {
    if (typeof child === 'string' && !child.trim()) {
      return null;
    }
    const content = replaceNode(child, activeData.substitutions, `network-${activeNetwork}-${index}`);
    if (content === null || content === false) {
      return null;
    }
    return <div className="min-w-0">
              {content}
            </div>;
  })}
      </div>
    </div>;
};

Every node exposes a set of metrics on the port 10013 under the path `/metrics`. These metrics can be used to monitor the health of the node, and to diagnose issues.

<Note>
  Apps expose metrics on the port 10013, under the path `/metrics` in the Prometheus format.
</Note>

For a validator the following components expose metrics:

* The validator app
* The participant

For a Super Validator, in addition, we have the following components:

* The SV app
* The scan app

<div className="todo">
  link to network or component diagram to improve understanding
</div>

## Scraping the metrics

We use OpenTelemetry to build the metrics and they are exposed in the Prometheus format, and can be scraped by a Prometheus server. For a reference of the existing metrics please check the metrics reference.

### Histograms

We expose the histograms as [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) which are converted to [prometheus native histograms](https://prometheus.io/docs/specs/native_histograms/).

<Note>
  Prometheus support must be enabled using the flag `-enable-feature=native-histograms`.

  Native histograms are available only on the protobuf format so Prometheus will switch to the protobuf collection format.
</Note>

You can switch back to regular histograms node by adding the following environment variable to a node: `ADDITIONAL_CONFIG_DISABLE_NATIVE_HISTOGRAMS="canton.monitoring.metrics.histograms=\[\]"`

### Enabling metrics

#### Configuring a helm deployment to enable metrics

To enable metrics in a helm deployment, set the `metrics.enable` value to `true` (default `false`) in the helm values. This will create a `ServiceMonitor` kubernetes custom resource. For this to work it would require that the Prometheus operator is installed in the cluster.

Alternatively you can add prometheus scrape annotations to the charts that are configured to scrape port 10013.

#### Configuring a docker compose deployment to enable metrics

<Note>
  Validator nodes only
</Note>

When using docker compose for the deployment, the metrics are enabled by default. These can be accessed at `http://validator.localhost/metrics` for the validator app and at `http://participant.localhost/metrics` for the participant.

#### Enabling extra metric triggers

The validator app can be configured to run a trigger that polls the topology state and exports metrics summarizing that state. These metrics have the prefix `splice.synchronizer-topology`. See the `validator-metrics-reference` for the concrete set of metrics.

This trigger is disabled by default. As per the information in Adding ad-hoc configuration, add an environment variable `ADDITIONAL_CONFIG_TOPOLOGY_METRICS_EXPORT=canton.validator-apps.validator_backend.automation.topology-metrics-polling-interval = 5m` to enable the trigger with a polling interval of 5 minutes.

## Grafana Dashboards

<NetworkVariables networkData={networkData}>
  The release bundle (|bundle\_download\_link|) contains a set of Grafana dashboards that are built based on the metrics above. These dashboards can be imported into a Grafana instance. The dashboards are built assuming a K8s deployment, and may need to be modified for other deployment types. The dashboards can be found under the `grafana-dashboards` folder in the release bundle.

  <Note>
    The dashboards are built using queries specific for Prometheus native histograms.
  </Note>
</NetworkVariables>