feat(coprocessor): add ZK proof gauges to txn-sender

These cover both txn sending by `transaction-sender` and
`zkproof-worker` operation.

Also make `--host-chain-url` a long-only argument for compatibility with
`-h` for help.

Author: dartdart26

coprocessor/fhevm-engine/transaction-sender/src/bin/transaction_sender.rs

@@ -50,7 +50,7 @@ struct Conf {
     #[arg(short, long)]
     gateway_url: Url,
 
-    #[arg(short, long)]
+    #[arg(long)]
     host_chain_url: Url,
 
     #[arg(short, long, value_enum, default_value = "private-key")]

coprocessor/fhevm-engine/transaction-sender/src/metrics.rs

@@ -94,6 +94,22 @@ pub(crate) static ADD_CIPHERTEXT_MATERIAL_UNSENT: LazyLock<IntGauge> = LazyLock:
     .unwrap()
 });
 
+pub(crate) static VERIFY_PROOF_RESP_UNSENT_TXN: LazyLock<IntGauge> = LazyLock::new(|| {
+    register_int_gauge!(
+        "coprocessor_verify_proof_resp_unsent_txn_gauge",
+        "Number of unsent verify proof response transactions"
+    )
+    .unwrap()
+});
+
+pub(crate) static VERIFY_PROOF_PENDING: LazyLock<IntGauge> = LazyLock::new(|| {
+    register_int_gauge!(
+        "coprocessor_verify_proof_pending_gauge",
+        "Number of pending verify proof requests"
+    )
+    .unwrap()
+});
+
 pub fn spawn_gauge_update_routine(period: std::time::Duration, db_pool: PgPool) -> JoinHandle<()> {
     tokio::spawn(async move {
         loop {
@@ -127,6 +143,34 @@ pub fn spawn_gauge_update_routine(period: std::time::Duration, db_pool: PgPool)
                 }
             }
 
+            match sqlx::query_scalar("SELECT COUNT(*) FROM verify_proofs WHERE verified IS NULL")
+                .fetch_one(&db_pool)
+                .await
+            {
+                Ok(count) => {
+                    info!(verify_proof_pending = %count, "Fetched pending verify proofs count");
+                    VERIFY_PROOF_PENDING.set(count);
+                }
+                Err(e) => {
+                    error!(error = %e, "Failed to fetch pending verify proofs count");
+                }
+            }
+
+            match sqlx::query_scalar(
+                "SELECT COUNT(*) FROM verify_proofs WHERE verified IS NOT NULL",
+            )
+            .fetch_one(&db_pool)
+            .await
+            {
+                Ok(count) => {
+                    info!(verify_proof_resp_unsent_txn = %count, "Fetched unsent verify proof response count");
+                    VERIFY_PROOF_RESP_UNSENT_TXN.set(count);
+                }
+                Err(e) => {
+                    error!(error = %e, "Failed to fetch unsent verify proof response count");
+                }
+            }
+
             sleep(period).await;
         }
     })

docs/metrics/metrics.md

@@ -58,6 +58,18 @@ Note that recommendations assume a smoke test that runs transactions/requests at
  - **Alarm**: If the gauge value exceeds a predefined threshold.
     - **Recommendation**: more than 100 unsent over 2 minutes, i.e. `min_over_time(gauge[2m]) > 100`.
 
+#### Metric Name: `coprocessor_verify_proof_resp_unsent_txn_gauge`
+ - **Type**: Gauge
+ - **Description**: Tracks the number of unsent verify proof response transactions in the transaction-sender.
+ - **Alarm**: If the gauge value exceeds a predefined threshold.
+    - **Recommendation**: more than 100 unsent over 2 minutes, i.e. `min_over_time(gauge[2m]) > 100`.
+
+#### Metric Name: `coprocessor_verify_proof_pending_gauge`
+ - **Type**: Gauge
+ - **Description**: Tracks the number of pending verify proofs (pending on the zkproof-worker).
+ - **Alarm**: If the gauge value exceeds a predefined threshold.
+    - **Recommendation**: more than 100 pending over 2 minutes, i.e. `min_over_time(gauge[2m]) > 100`.
+
 ### gw-listener
 
 #### Metric Name: `coprocessor_gw_listener_verify_proof_success_counter`
@@ -125,7 +137,7 @@ Note that recommendations assume a smoke test that runs transactions/requests at
 
 ### zkproof-worker
 
-Metrics for zkproof-worker are to be added in future releases.
+Metrics for zkproof-worker are to be added in future releases, if/when needed. Currently, the transaction-sender handles ZK proof related metrics, please see its section.
 
 ### sns-worker