---
title: "HELP genlayer_node_cpu_usage_percent Current CPU usage percentage (0-100)"
description: "Exposes Prometheus-compatible metrics for monitoring node performance and health."
source: https://docs.genlayer.com/api-references/genlayer-node/ops/metrics
last_updated: 2026-10-19
---

### metrics

Exposes Prometheus-compatible metrics for monitoring node performance and health.

**Method:** `GET /metrics`

**Parameters:** None

**Returns:** Prometheus-format text response with all collected metrics

**Example Request:**

```bash
curl -X GET http://localhost:9153/metrics
```

**Example Response:**

```
# HELP genlayer_node_cpu_usage_percent Current CPU usage percentage (0-100)
# TYPE genlayer_node_cpu_usage_percent gauge
genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3

# HELP genlayer_node_memory_rss_bytes Resident Set Size (RSS) memory usage in bytes
# TYPE genlayer_node_memory_rss_bytes gauge
genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648

# HELP genlayer_node_network_rx_bytes_total Total bytes received across all network interfaces
# TYPE genlayer_node_network_rx_bytes_total counter
genlayer_node_network_rx_bytes_total{component="node"} 1073741824
```

#### Available Metrics

The node collects metrics for three main components:
- **Node** - The main node process
- **GenVM** - The GenVM modules (LLM and Web)
- **WebDriver** - The WebDriver container

**Operator Condition Metrics**

- `genlayer_node_operator_condition_state{condition,state}`: one-hot gauge for the current catalog state. All state series are prewarmed at process start.
- `genlayer_node_operator_condition_occurrences_total{condition,reason}`: accepted failed observations by stable reason.
- `genlayer_node_operator_condition_transitions_total{condition,from,to}`: lifecycle transitions, including recovery.
- `genlayer_node_operator_condition_last_success_timestamp_seconds{condition}`: Unix timestamp of the last successful observation, or zero before one occurs.

`condition`, `state`, `reason`, `from`, and `to` come from the fixed startup catalog; upstream error text is never used as a label.

`occurrences_total` counts differently per condition kind, so the counter stays comparable: an *observed* condition (folded from duty traffic) counts every failed observation, while a *polled* condition counts only entry into a failing state. Counting every failed poll would scale the counter with the poll interval rather than the fault — a three-day ban at a ten-second poll would read as roughly 26,000 events.

**CPU Metrics**

**genlayer_node_cpu_usage_percent**

Current CPU usage percentage (0-100).

**Type:** Gauge

**Labels:**
- `component`: One of `node`, `genvm-llm`, `genvm-web`, `webdriver`

**Example:**
```
genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3
```

**Memory Metrics**

**genlayer_node_memory_rss_bytes**

Resident Set Size (RSS) memory usage in bytes.

**Type:** Gauge

**Labels:**
- `component`: One of `node`, `genvm-llm`, `genvm-web`, `webdriver`

**Example:**
```
genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648
```

**genlayer_node_memory_vms_bytes**

Virtual Memory Size (VMS) in bytes.

**Type:** Gauge

**Labels:**
- `component`: One of `node`, `genvm-llm`, `genvm-web`, `webdriver`

**Note:** Only available for node and GenVM components.

**genlayer_node_memory_usage_bytes**

Current memory usage in bytes (WebDriver only).

**Type:** Gauge

**Labels:**
- `component`: `webdriver`

**genlayer_node_memory_limit_bytes**

Memory limit in bytes (WebDriver only).

**Type:** Gauge

**Labels:**
- `component`: `webdriver`

**genlayer_node_memory_percent**

Memory usage percentage.

**Type:** Gauge

**Labels:**
- `component`: One of `genvm-llm`, `genvm-web`, `node`, `webdriver`
- `node`: Node address (hex format)

**genlayer_node_memory_peak_bytes**

Peak memory usage in bytes (GenVM components only).

**Type:** Gauge

**Labels:**
- `component`: One of `genvm-llm`, `genvm-web`
- `node`: Node address (hex format)

**Network Metrics**

**genlayer_node_network_rx_bytes_total**

Total bytes received across all network interfaces.

**Type:** Counter

**Labels:**
- `component`: One of `node`, `genvm-llm`, `genvm-web`, `webdriver`
- `node`: Node address (hex format)

**Example:**
```
genlayer_node_network_rx_bytes_total{component="node",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 96879942
genlayer_node_network_rx_bytes_total{component="webdriver",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 84
```

**genlayer_node_network_tx_bytes_total**

Total bytes transmitted across all network interfaces.

**Type:** Counter

**Labels:**
- `component`: One of `node`, `genvm-llm`, `genvm-web`, `webdriver`
- `node`: Node address (hex format)

**Disk Metrics**

**genlayer_node_disk_free_bytes**

Available free disk space in bytes.

**Type:** Gauge

**Labels:**
- `component`: `node`
- `directory`: One of `genlayer.db`, `keystore`, `logs`
- `node`: Node address (hex format)

**genlayer_node_disk_total_bytes**

Total disk space in bytes.

**Type:** Gauge

**Labels:**
- `component`: `node`
- `directory`: One of `genlayer.db`, `keystore`, `logs`
- `node`: Node address (hex format)

**genlayer_node_disk_usage_bytes**

Used disk space in bytes.

**Type:** Gauge

**Labels:**
- `component`: `node`
- `directory`: One of `genlayer.db`, `keystore`, `logs`
- `node`: Node address (hex format)

**genlayer_node_disk_usage_percent**

Used disk space as percentage.

**Type:** Gauge

**Labels:**
- `component`: `node`
- `directory`: One of `genlayer.db`, `keystore`, `logs`
- `node`: Node address (hex format)

**Node Status Metrics**

**genlayer_node_blocks_behind**

Number of blocks behind the latest block.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_latest_block**

Latest block number in the network.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_processing_block**

Currently processing block number.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_synced_block**

Last synced block number.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_synced**

Node synchronization status (1 = synced, 0 = not synced).

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_prime_blocked_insufficient_funds**

Whether a mandatory validator prime is currently blocked by an unfunded operator wallet (1 = blocked, 0 = not blocked). While blocked the validator cannot pay for consensus duty transactions and is flagged idle each round — fund the operator wallet to recover.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_uptime_seconds**

Node uptime in seconds.

**Type:** Gauge

**Labels:**
- `component`: `node`
- `node`: Node address (hex format)

**Transaction Metrics**

**genlayer_node_transactions_accepted_synced_total**

Total number of accepted and synced transactions.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_transactions_activated_total**

Total number of activated transactions.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_transactions_leader_proposed_total**

Total number of transactions proposed as leader.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_transactions_leader_revealed_total**

Total number of transactions revealed as leader.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_transactions_validator_commit_total**

Total number of validator commit votes.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**genlayer_node_transactions_validator_reveal_total**

Total number of validator reveal votes.

**Type:** Gauge

**Labels:**
- `node`: Node address (hex format)

**Gas Metrics (duty transactions)**

Emitted inline from the shared consensus gas estimator and transactor, so they cover every duty method (`finalizeTransaction`, `commitVote`, `revealVote`, `proposeReceiptWithFees`, `topUpAndSubmitAppeal`, `activateTransaction`, `validatorPrime`, `epochAdvance`, ...). All are true counters.

**genlayer_node_gas_estimation_total**

Outcome of every duty gas estimation.

**Type:** Counter

**Labels:**
- `method`: Contract method name
- `outcome`: One of `success` (real RPC estimate), `fallback_block_limit` (estimation failed; submitted at full block gas limit), `fallback_cache` / `fallback_cache_scaled` (estimation failed; reused gas from the shared calldata-length-keyed cache, possibly learned from a different method)

**genlayer_node_gas_estimation_capped_total**

Estimation succeeded but the safety-multiplied estimate (or per-call reserve) exceeded the block gas limit and was silently capped — the safety margin is gone while everything still works.

**Type:** Counter

**Labels:**
- `method`: Contract method name

**genlayer_node_gas_shortfall_total**

Confirmed on-chain out-of-gas: the transaction landed but ran out of gas mid-execution. Self-heals — the node retries with a larger gas limit — so a single occurrence is not actionable; a **sustained rate** signals systematic under-estimation (treat as a warning, not a page).

**Type:** Counter

**Labels:**
- `method`: Contract method name
- `reason`: One of `typed_shortfall` (`InsufficientGasForInternalMessageCreation` revert with exact gas requirement), `out_of_gas` (explicit `OutOfGas` revert), `heuristic` (undecoded revert that consumed gas above the exhaustion threshold)

**genlayer_node_gas_block_limit_exhausted_total**

Gas shortfall while the gas limit was already capped at the block limit: the work cannot fit in one block and retrying cannot help. Non-self-healing — **any occurrence is actionable** (page-worthy).

**Type:** Counter

**Labels:**
- `method`: Contract method name

**genlayer_node_gas_spent_wei_total**

Cumulative wei actually billed for gas, from each mined duty receipt (`gasUsed × effectiveGasPrice`). Reverted attempts are billed too and are accounted per attempt; submissions that never land cost nothing. The `reverted` slice is the exact cost of failed work, and `sum(genlayer_node_gas_spent_wei_total)` is the operator's total billed duty gas.

**Type:** Counter

**Labels:**
- `method`: Contract method name
- `outcome`: One of `success`, `reverted`
- `price_source`: `effective` (receipt's billed price) or `fee_cap` (submitted fee cap used as an upper bound when the RPC omits `effectiveGasPrice`)

**Consensus Submission Metrics**

**genlayer_node_consensus_submit_total**

Terminal outcome of every duty submission.

**Type:** Counter

**Labels:**
- `method`: Contract method name
- `outcome`: One of `success` (landed and executed), `reverted` (landed, contract rejected — gas still billed), `failed` (never landed: RPC/nonce/funds/authorization)

**genlayer_node_consensus_revert_total**

Decoded revert causes, one increment per observed revert (attempt-level).

**Type:** Counter

**Labels:**
- `method`: Contract method name
- `error`: Decoded custom error from the fee/gas allowlist (`InsufficientFeesForRound`, `InsufficientFees`, `FailedFeeTransfer`, `FeeSettlementNotCompleted`, `MaxPriceExceeded`, `ExecutionBudgetExceeded`, `MessageBudgetExceeded`, `BudgetTooLow`, `RollupBudgetBelowFloor`, `InsufficientBondCustody`, `InvalidAppealBond`, `AppealBondTooLow`, `InsufficientGasForInternalMessageCreation`, `OutOfGas`); anything else collapses to `other`

**Fee Flow Metrics (v0.6 ConsensusMainWithFees)**

Note: there are no dedicated appeal-bond metrics. No node runtime path submits appeals — appellants are external clients funding bonds from their own wallets. If a validator appeal duty ever ships, the method-labeled gas/submission metrics above cover it under `method="topUpAndSubmitAppeal"` with no code change.

**genlayer_node_fees_operator_insufficient_funds_total**

Duty submissions rejected because the operator account cannot cover value + gas. Complements the `genlayer_node_prime_blocked_insufficient_funds` gauge by leaving a trace of dips that recover between scrapes. A sustained increase means the operator wallet needs funding (page-worthy).

**Type:** Counter

**genlayer_node_fees_settlement_events_total**

Fee-settlement events decoded from this node's own mined finalize receipts. Finalize runs under a staggered lease, so each finalize is observed by exactly one node — aggregate fleet-wide with `sum by (network)`. Use the `severity` label to alert on failures without enumerating event names: `genlayer_node_fees_settlement_events_total{severity="alarm"}` covers all three failure modes and any future one.

**Type:** Counter

**Labels:**
- `event`: One of `UnifiedFeesDistributed` (fees distributed at settlement), `FeesRefunded` (surplus returned to the sender), `FeeSettlementFailed` (finalize succeeded but settlement failed — validators unpaid until `retryFeeSettlement`), `CommitFinalFeesFailed` (tx finalized without fee settlement), `ValueWithdrawalFailed` (a best-effort transfer — refund/appealer payout — failed; funds stay custodied pending recovery)
- `severity`: `alarm` for the three failure events (unpaid or stuck-funds modes needing operator action) or `info` for the normal `UnifiedFeesDistributed` / `FeesRefunded` happy path

**genlayer_node_fees_distributed_wei_total**

Total wei moved at fee settlement across **all** recipients — committee payouts **plus** sender/contributor refunds **plus** appealer rewards — summed from `UnifiedFeesDistributed` amounts in this node's finalize receipts. It **excludes** amounts retained in staking custody (`amountToStaking`). This is a network **"value settled"** liveness signal, **not** per-node earnings — for a validator's own income use `genlayer_node_fees_earned_wei_total`.

**Type:** Counter

**genlayer_node_fees_earned_wei_total**

This node's validator earnings credited at prime, read from the `ValidatorPrime` event when a prime is effective. This is the accurate **per-node** income (unlike `fees_distributed_wei_total`, which is whole-committee plus refunds). Earnings settle into stake per epoch, so this series lags settlement by up to one prime interval. Pair `type="fee_rewards"` with `genlayer_node_gas_spent_wei_total` for operator net P&L (`earned − gas_spent`).

**Type:** Counter

**Labels:**
- `type`: One of `fee_rewards` (fee-flow income), `validator_rewards`, `delegator_rewards` (staking-reward components)

**genlayer_node_fees_penalties_wei_total**

This node's fee penalties deducted at prime, read from the same `ValidatorPrime` event. Kept separate from `fees_earned_wei_total` so earnings stay gross income; subtract this for net.

**Type:** Counter

**genlayer_node_external_message_delivery_total**

External messages this node's own delivery calls (`flushExternalMessages`, `flushInternalMessages`, `finalizeTransaction`) resolved or paused, read from their mined receipts. A paused message was not called and not charged; the node retries it with a backoff (30 s doubling to 10 min) until the chain reports it issued. A `paused_*` series that keeps rising for one network means a message nobody can deliver.

**Type:** Counter

**Labels:**
- `outcome`: `delivered` (ran and succeeded), `call_failed` (ran and failed; final), `paused_gas` (no room to forward its gas envelope), `paused_value` (its value was not yet available), `cancelled_unfunded` (its open-bucket envelope was never funded; value refunded), `cancelled_ceiling` (its gas limit is above the current ceiling; value refunded)

**genlayer_node_external_message_reimbursed_wei_total**

Wei the chain reimbursed this node for running external messages: gas used times the transaction's locked gas price. Pair it with `gas_spent_wei_total` for the net cost of delivery.

**Type:** Counter

**Go Runtime Metrics**

The endpoint also exposes standard Go runtime metrics including:

- `go_gc_duration_seconds` - Garbage collection duration summary
- `go_goroutines` - Number of active goroutines
- `go_memstats_*` - Go memory statistics (heap, stack, GC)
- `go_threads` - Number of OS threads

**Process Metrics**

Standard process metrics are also available:

- `process_cpu_seconds_total` - Total CPU time
- `process_resident_memory_bytes` - Resident memory size
- `process_virtual_memory_bytes` - Virtual memory size
- `process_open_fds` - Number of open file descriptors
- `process_network_receive_bytes_total` - Network bytes received
- `process_network_transmit_bytes_total` - Network bytes transmitted

**Prometheus Handler Metrics**

Metrics about the metrics endpoint itself:

- `promhttp_metric_handler_requests_total` - Total scrape requests by status code
- `promhttp_metric_handler_requests_in_flight` - Current scrapes being served

#### Configuration

Metrics collection can be configured in the node configuration file:

```yaml
metrics:
  interval: "15s"        # Default interval for all collectors
  collectors:
    node:
      enabled: true      # Enable/disable node metrics
      interval: "30s"    # Optional: Override default interval
    genvm:
      enabled: true      # Enable/disable GenVM metrics
      # Uses default interval (15s)
    webdriver:
      enabled: true      # Enable/disable WebDriver metrics
      interval: "60s"    # Optional: Override default interval
```

**Default Values**
- `metrics.interval`: 15s (applies to all collectors by default)
- `enabled`: true (for each collector)
- `interval`: Uses metrics.interval if not specified per collector

**Configuration Priority**
1. Collector-specific `interval` (if specified)
2. Global `metrics.interval` (if specified)
3. System default (15s)

#### Troubleshooting

**No Metrics for GenVM**

GenVM processes are ephemeral and only run when executing smart contracts. If you don't see GenVM metrics, it's likely that no contracts are currently being executed.

**WebDriver Metrics Missing**

Ensure the WebDriver container is running:
```bash
docker ps | grep genlayer-node-webdriver
```

If not running, start it:
```bash
task docker:webdriver:docker-compose:up:detach
```

**Metrics Not Updating**

Check if metrics collection is enabled in your configuration and that the node has been restarted after configuration changes.

#### Performance Considerations

- Collection intervals can be increased to reduce overhead further
- Each collector runs independently and won't block others if one fails