API References
GenLayer Node API
Ops Methods
metrics

metrics

Exposes Prometheus-compatible metrics for monitoring node performance and health.

Method: GET /metrics

Parameters: None

Returns: Prometheus-format text response with all collected metrics

Example Request:

curl -X GET http://localhost:9153/metrics

Example Response:

# HELP genlayer_node_cpu_usage_percent Current CPU usage percentage (0-100)
# TYPE genlayer_node_cpu_usage_percent gauge
genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3

# HELP genlayer_node_memory_rss_bytes Resident Set Size (RSS) memory usage in bytes
# TYPE genlayer_node_memory_rss_bytes gauge
genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648

# HELP genlayer_node_network_rx_bytes_total Total bytes received across all network interfaces
# TYPE genlayer_node_network_rx_bytes_total counter
genlayer_node_network_rx_bytes_total{component="node"} 1073741824

Available Metrics

The node collects metrics for three main components:

  • Node - The main node process
  • GenVM - The GenVM modules (LLM and Web)
  • WebDriver - The WebDriver container

Operator Condition Metrics

  • genlayer_node_operator_condition_state{condition,state}: one-hot gauge for the current catalog state. All state series are prewarmed at process start.
  • genlayer_node_operator_condition_occurrences_total{condition,reason}: accepted failed observations by stable reason.
  • genlayer_node_operator_condition_transitions_total{condition,from,to}: lifecycle transitions, including recovery.
  • genlayer_node_operator_condition_last_success_timestamp_seconds{condition}: Unix timestamp of the last successful observation, or zero before one occurs.

condition, state, reason, from, and to come from the fixed startup catalog; upstream error text is never used as a label.

occurrences_total counts differently per condition kind, so the counter stays comparable: an observed condition (folded from duty traffic) counts every failed observation, while a polled condition counts only entry into a failing state. Counting every failed poll would scale the counter with the poll interval rather than the fault — a three-day ban at a ten-second poll would read as roughly 26,000 events.

CPU Metrics

genlayer_node_cpu_usage_percent

Current CPU usage percentage (0-100).

Type: Gauge

Labels:

  • component: One of node, genvm-llm, genvm-web, webdriver

Example:

genlayer_node_cpu_usage_percent{component="node"} 15.5
genlayer_node_cpu_usage_percent{component="genvm-llm"} 45.2
genlayer_node_cpu_usage_percent{component="genvm-web"} 12.8
genlayer_node_cpu_usage_percent{component="webdriver"} 5.3

Memory Metrics

genlayer_node_memory_rss_bytes

Resident Set Size (RSS) memory usage in bytes.

Type: Gauge

Labels:

  • component: One of node, genvm-llm, genvm-web, webdriver

Example:

genlayer_node_memory_rss_bytes{component="node"} 524288000
genlayer_node_memory_rss_bytes{component="genvm-llm"} 2147483648

genlayer_node_memory_vms_bytes

Virtual Memory Size (VMS) in bytes.

Type: Gauge

Labels:

  • component: One of node, genvm-llm, genvm-web, webdriver

Note: Only available for node and GenVM components.

genlayer_node_memory_usage_bytes

Current memory usage in bytes (WebDriver only).

Type: Gauge

Labels:

  • component: webdriver

genlayer_node_memory_limit_bytes

Memory limit in bytes (WebDriver only).

Type: Gauge

Labels:

  • component: webdriver

genlayer_node_memory_percent

Memory usage percentage.

Type: Gauge

Labels:

  • component: One of genvm-llm, genvm-web, node, webdriver
  • node: Node address (hex format)

genlayer_node_memory_peak_bytes

Peak memory usage in bytes (GenVM components only).

Type: Gauge

Labels:

  • component: One of genvm-llm, genvm-web
  • node: Node address (hex format)

Network Metrics

genlayer_node_network_rx_bytes_total

Total bytes received across all network interfaces.

Type: Counter

Labels:

  • component: One of node, genvm-llm, genvm-web, webdriver
  • node: Node address (hex format)

Example:

genlayer_node_network_rx_bytes_total{component="node",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 96879942
genlayer_node_network_rx_bytes_total{component="webdriver",node="0x84b6cbd007511352d3fea26834c5c39a440903c4"} 84

genlayer_node_network_tx_bytes_total

Total bytes transmitted across all network interfaces.

Type: Counter

Labels:

  • component: One of node, genvm-llm, genvm-web, webdriver
  • node: Node address (hex format)

Disk Metrics

genlayer_node_disk_free_bytes

Available free disk space in bytes.

Type: Gauge

Labels:

  • component: node
  • directory: One of genlayer.db, keystore, logs
  • node: Node address (hex format)

genlayer_node_disk_total_bytes

Total disk space in bytes.

Type: Gauge

Labels:

  • component: node
  • directory: One of genlayer.db, keystore, logs
  • node: Node address (hex format)

genlayer_node_disk_usage_bytes

Used disk space in bytes.

Type: Gauge

Labels:

  • component: node
  • directory: One of genlayer.db, keystore, logs
  • node: Node address (hex format)

genlayer_node_disk_usage_percent

Used disk space as percentage.

Type: Gauge

Labels:

  • component: node
  • directory: One of genlayer.db, keystore, logs
  • node: Node address (hex format)

Node Status Metrics

genlayer_node_blocks_behind

Number of blocks behind the latest block.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_latest_block

Latest block number in the network.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_processing_block

Currently processing block number.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_synced_block

Last synced block number.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_synced

Node synchronization status (1 = synced, 0 = not synced).

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_prime_blocked_insufficient_funds

Whether a mandatory validator prime is currently blocked by an unfunded operator wallet (1 = blocked, 0 = not blocked). While blocked the validator cannot pay for consensus duty transactions and is flagged idle each round — fund the operator wallet to recover.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_uptime_seconds

Node uptime in seconds.

Type: Gauge

Labels:

  • component: node
  • node: Node address (hex format)

Transaction Metrics

genlayer_node_transactions_accepted_synced_total

Total number of accepted and synced transactions.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_transactions_activated_total

Total number of activated transactions.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_transactions_leader_proposed_total

Total number of transactions proposed as leader.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_transactions_leader_revealed_total

Total number of transactions revealed as leader.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_transactions_validator_commit_total

Total number of validator commit votes.

Type: Gauge

Labels:

  • node: Node address (hex format)

genlayer_node_transactions_validator_reveal_total

Total number of validator reveal votes.

Type: Gauge

Labels:

  • node: Node address (hex format)

Gas Metrics (duty transactions)

Emitted inline from the shared consensus gas estimator and transactor, so they cover every duty method (finalizeTransaction, commitVote, revealVote, proposeReceiptWithFees, topUpAndSubmitAppeal, activateTransaction, validatorPrime, epochAdvance, ...). All are true counters.

genlayer_node_gas_estimation_total

Outcome of every duty gas estimation.

Type: Counter

Labels:

  • method: Contract method name
  • outcome: One of success (real RPC estimate), fallback_block_limit (estimation failed; submitted at full block gas limit), fallback_cache / fallback_cache_scaled (estimation failed; reused gas from the shared calldata-length-keyed cache, possibly learned from a different method)

genlayer_node_gas_estimation_capped_total

Estimation succeeded but the safety-multiplied estimate (or per-call reserve) exceeded the block gas limit and was silently capped — the safety margin is gone while everything still works.

Type: Counter

Labels:

  • method: Contract method name

genlayer_node_gas_shortfall_total

Confirmed on-chain out-of-gas: the transaction landed but ran out of gas mid-execution. Self-heals — the node retries with a larger gas limit — so a single occurrence is not actionable; a sustained rate signals systematic under-estimation (treat as a warning, not a page).

Type: Counter

Labels:

  • method: Contract method name
  • reason: One of typed_shortfall (InsufficientGasForInternalMessageCreation revert with exact gas requirement), out_of_gas (explicit OutOfGas revert), heuristic (undecoded revert that consumed gas above the exhaustion threshold)

genlayer_node_gas_block_limit_exhausted_total

Gas shortfall while the gas limit was already capped at the block limit: the work cannot fit in one block and retrying cannot help. Non-self-healing — any occurrence is actionable (page-worthy).

Type: Counter

Labels:

  • method: Contract method name

genlayer_node_gas_spent_wei_total

Cumulative wei actually billed for gas, from each mined duty receipt (gasUsed × effectiveGasPrice). Reverted attempts are billed too and are accounted per attempt; submissions that never land cost nothing. The reverted slice is the exact cost of failed work, and sum(genlayer_node_gas_spent_wei_total) is the operator's total billed duty gas.

Type: Counter

Labels:

  • method: Contract method name
  • outcome: One of success, reverted
  • price_source: effective (receipt's billed price) or fee_cap (submitted fee cap used as an upper bound when the RPC omits effectiveGasPrice)

Consensus Submission Metrics

genlayer_node_consensus_submit_total

Terminal outcome of every duty submission.

Type: Counter

Labels:

  • method: Contract method name
  • outcome: One of success (landed and executed), reverted (landed, contract rejected — gas still billed), failed (never landed: RPC/nonce/funds/authorization)

genlayer_node_consensus_revert_total

Decoded revert causes, one increment per observed revert (attempt-level).

Type: Counter

Labels:

  • method: Contract method name
  • error: Decoded custom error from the fee/gas allowlist (InsufficientFeesForRound, InsufficientFees, FailedFeeTransfer, FeeSettlementNotCompleted, MaxPriceExceeded, ExecutionBudgetExceeded, MessageBudgetExceeded, BudgetTooLow, RollupBudgetBelowFloor, InsufficientBondCustody, InvalidAppealBond, AppealBondTooLow, InsufficientGasForInternalMessageCreation, OutOfGas); anything else collapses to other

Fee Flow Metrics (v0.6 ConsensusMainWithFees)

Note: there are no dedicated appeal-bond metrics. No node runtime path submits appeals — appellants are external clients funding bonds from their own wallets. If a validator appeal duty ever ships, the method-labeled gas/submission metrics above cover it under method="topUpAndSubmitAppeal" with no code change.

genlayer_node_fees_operator_insufficient_funds_total

Duty submissions rejected because the operator account cannot cover value + gas. Complements the genlayer_node_prime_blocked_insufficient_funds gauge by leaving a trace of dips that recover between scrapes. A sustained increase means the operator wallet needs funding (page-worthy).

Type: Counter

genlayer_node_fees_settlement_events_total

Fee-settlement events decoded from this node's own mined finalize receipts. Finalize runs under a staggered lease, so each finalize is observed by exactly one node — aggregate fleet-wide with sum by (network). Use the severity label to alert on failures without enumerating event names: genlayer_node_fees_settlement_events_total{severity="alarm"} covers all three failure modes and any future one.

Type: Counter

Labels:

  • event: One of UnifiedFeesDistributed (fees distributed at settlement), FeesRefunded (surplus returned to the sender), FeeSettlementFailed (finalize succeeded but settlement failed — validators unpaid until retryFeeSettlement), CommitFinalFeesFailed (tx finalized without fee settlement), ValueWithdrawalFailed (a best-effort transfer — refund/appealer payout — failed; funds stay custodied pending recovery)
  • severity: alarm for the three failure events (unpaid or stuck-funds modes needing operator action) or info for the normal UnifiedFeesDistributed / FeesRefunded happy path

genlayer_node_fees_distributed_wei_total

Total wei moved at fee settlement across all recipients — committee payouts plus sender/contributor refunds plus appealer rewards — summed from UnifiedFeesDistributed amounts in this node's finalize receipts. It excludes amounts retained in staking custody (amountToStaking). This is a network "value settled" liveness signal, not per-node earnings — for a validator's own income use genlayer_node_fees_earned_wei_total.

Type: Counter

genlayer_node_fees_earned_wei_total

This node's validator earnings credited at prime, read from the ValidatorPrime event when a prime is effective. This is the accurate per-node income (unlike fees_distributed_wei_total, which is whole-committee plus refunds). Earnings settle into stake per epoch, so this series lags settlement by up to one prime interval. Pair type="fee_rewards" with genlayer_node_gas_spent_wei_total for operator net P&L (earned − gas_spent).

Type: Counter

Labels:

  • type: One of fee_rewards (fee-flow income), validator_rewards, delegator_rewards (staking-reward components)

genlayer_node_fees_penalties_wei_total

This node's fee penalties deducted at prime, read from the same ValidatorPrime event. Kept separate from fees_earned_wei_total so earnings stay gross income; subtract this for net.

Type: Counter

genlayer_node_external_message_delivery_total

External messages this node's own delivery calls (flushExternalMessages, flushInternalMessages, finalizeTransaction) resolved or paused, read from their mined receipts. A paused message was not called and not charged; the node retries it with a backoff (30 s doubling to 10 min) until the chain reports it issued. A paused_* series that keeps rising for one network means a message nobody can deliver.

Type: Counter

Labels:

  • outcome: delivered (ran and succeeded), call_failed (ran and failed; final), paused_gas (no room to forward its gas envelope), paused_value (its value was not yet available), cancelled_unfunded (its open-bucket envelope was never funded; value refunded), cancelled_ceiling (its gas limit is above the current ceiling; value refunded)

genlayer_node_external_message_reimbursed_wei_total

Wei the chain reimbursed this node for running external messages: gas used times the transaction's locked gas price. Pair it with gas_spent_wei_total for the net cost of delivery.

Type: Counter

Go Runtime Metrics

The endpoint also exposes standard Go runtime metrics including:

  • go_gc_duration_seconds - Garbage collection duration summary
  • go_goroutines - Number of active goroutines
  • go_memstats_* - Go memory statistics (heap, stack, GC)
  • go_threads - Number of OS threads

Process Metrics

Standard process metrics are also available:

  • process_cpu_seconds_total - Total CPU time
  • process_resident_memory_bytes - Resident memory size
  • process_virtual_memory_bytes - Virtual memory size
  • process_open_fds - Number of open file descriptors
  • process_network_receive_bytes_total - Network bytes received
  • process_network_transmit_bytes_total - Network bytes transmitted

Prometheus Handler Metrics

Metrics about the metrics endpoint itself:

  • promhttp_metric_handler_requests_total - Total scrape requests by status code
  • promhttp_metric_handler_requests_in_flight - Current scrapes being served

Configuration

Metrics collection can be configured in the node configuration file:

metrics:
  interval: "15s"        # Default interval for all collectors
  collectors:
    node:
      enabled: true      # Enable/disable node metrics
      interval: "30s"    # Optional: Override default interval
    genvm:
      enabled: true      # Enable/disable GenVM metrics
      # Uses default interval (15s)
    webdriver:
      enabled: true      # Enable/disable WebDriver metrics
      interval: "60s"    # Optional: Override default interval

Default Values

  • metrics.interval: 15s (applies to all collectors by default)
  • enabled: true (for each collector)
  • interval: Uses metrics.interval if not specified per collector

Configuration Priority

  1. Collector-specific interval (if specified)
  2. Global metrics.interval (if specified)
  3. System default (15s)

Troubleshooting

No Metrics for GenVM

GenVM processes are ephemeral and only run when executing smart contracts. If you don't see GenVM metrics, it's likely that no contracts are currently being executed.

WebDriver Metrics Missing

Ensure the WebDriver container is running:

docker ps | grep genlayer-node-webdriver

If not running, start it:

task docker:webdriver:docker-compose:up:detach

Metrics Not Updating

Check if metrics collection is enabled in your configuration and that the node has been restarted after configuration changes.

Performance Considerations

  • Collection intervals can be increased to reduce overhead further
  • Each collector runs independently and won't block others if one fails
healthsnapshot