# Running an Arbitrum One RPC Node on Kubernetes

We needed an Arbitrum One RPC node in our own infrastructure. The setup looked straightforward: deploy Nitro v3.9.8 as a GKE StatefulSet using a Helm chart, point it at our internal Geth for L1, configure init.latest: "pruned" to pull the latest snapshot, and let it do its thing. Chain ID 42161, non-validator, standard pruned sync. Nothing exotic.

The snapshot download ran for about 15 hours. Five parts, roughly 500 GiB each (the last one 163 GiB), totalling ~2.1 TiB. We watched the logs tick through `pruned.tar.part0000` through `pruned.tar.part0004` and waited. Then the pod crashed mid-extraction.

On the next restart, the node refused to initialize. The error was this:

```plaintext
ERROR error initializing database  err="found 1 unexpected files in database directory, including: tmp"
```

That was it. No further detail. `l2chaindata` was never created - the database directory contained only the `LOCK` file and the `tmp/` staging directory left behind from the download. Every restart produced the same error. Every restart ended in the same crash loop. There was no flag to skip the check, no `--force-init`, no obvious way to tell the node to proceed anyway.

To understand why this is not a bug you can work around but a design choice you have to accommodate, it helps to look at how Nitro decides whether it is safe to initialize at all.

## How Nitro initializes from a snapshot

### The init flags

Nitro's snapshot init system is configured through `--init.*` flags (or the JSON config equivalent):

*   `init.latest: "pruned"` - downloads the latest pruned snapshot automatically
    
*   `init.url: "https://..."` - pins to a specific snapshot URL
    
*   `init.download-path` - where to stage downloaded tar parts, separate from the database directory; if not set, Nitro defaults to `<persistent.chain>/nitro/tmp/`
    

### Download flow

The sequence, from [`cmd/nitro/init.go`](https://github.com/OffchainLabs/nitro/blob/v3.9.8/cmd/nitro/init.go):

1.  Fetches the manifest from the snapshot URL
    
2.  Downloads multipart tar files (`.part0000`, `.part0001`, ...) to `download-path`
    
3.  Joins the parts into a single archive
    
4.  Extracts the archive directly into `<persistent.chain>/nitro/l2chaindata/`
    
5.  Cleans up the `tmp/` staging directory - but only if `download-path` was not explicitly set
    

### `checkEmptyDatabaseDir`

Before any initialization work begins, Nitro checks the database directory against a strict allowlist in [`checkEmptyDatabaseDir`](https://github.com/OffchainLabs/nitro/blob/v3.9.8/cmd/nitro/init.go#L399-L420):

```go
// cmd/nitro/init.go
allowedFiles := map[string]bool{
    "LOCK": true, "classic-msg": true, "l2chaindata": true,
}
for _, entry := range entries {
    if !allowedFiles[entry.Name()] {
        unexpectedFiles = append(unexpectedFiles, entry.Name())
    }
}
if len(unexpectedFiles) > 0 {
    return fmt.Errorf("found %d unexpected files in database directory, including: %s", ...)
}
```

When `download-path` is not set (the default), Nitro creates `<persistent.chain>/nitro/tmp/` as the staging directory. If the pod crashes mid-download, `tmp/` is left behind with whatever partial tar parts had been written. On the next restart, `checkEmptyDatabaseDir` finds `tmp/` in the database directory - it is not in the allowlist - and the node refuses to start.

This is the most common failure mode when setting up a new Nitro node.

> The check runs before any initialization work. There is no flag to skip it, no recovery mode. The only way forward is to either remove the unexpected files manually or prevent them from being there in the first place.

## The fix: init.download-path

Set `init.download-path` to a path on a separate PVC. Nitro stages all download artifacts there, leaving the main data PVC containing only `LOCK` (and eventually `l2chaindata`). If the pod crashes, `checkEmptyDatabaseDir` passes on restart and the download resumes.

### Resuming downloads

The download library Nitro uses - `github.com/cavaliergopher/grab` - detects already-downloaded files by comparing local file size to the server's `Content-Length`. Existing complete parts are skipped. This makes crash recovery essentially free - you lose at most one in-progress part.

### Cleanup is manual

When `download-path` is set, Nitro does not clean up the download artifacts after extraction completes. The parts and joined archive remain on the init PVC. Delete them manually once you have confirmed the node is syncing.

### Disk space math

| Item | Size | Location |
| --- | --- | --- |
| Compressed tar parts (5 parts) | ~2.1 TiB | Init PVC |
| Joined archive (before extraction) | ~2.1 TiB | Init PVC (peak, simultaneous with parts) |
| Extracted pebble database (`l2chaindata`) | ~600-700 GiB | Main data PVC |

Peak init PVC usage: ~4.2 TiB (parts + joined archive simultaneously). We used 6 TiB for headroom.

PVC sizing:

*   Init PVC: 6 TiB on cheaper HDD-backed storage - sufficient because downloads only need sequential throughput, not live-database IOPS
    
*   Main data PVC: 4 TiB on SSD-backed storage - needed for the IOPS of a live pebble database
    

### Helm chart implementation

`init.download-path` is auto-injected into `config.json` via the Helm template when `initStorage.enabled: true`. When disabled, the init PVC is removed by Helm automatically - it is a standalone PVC, not a `volumeClaimTemplate`, so Helm can delete it without touching stateful data.

> The extraction destination is always <persistent.chain>/nitro/l2chaindata/ regardless of download-path. Setting download-path only affects where the parts and joined archive live during the download phase.

## Resource sizing

Observed on an 8-core node, April 2026:

| Phase | CPU | Memory |
| --- | --- | --- |
| Downloading snapshot parts | ~1 core | 1-5 GiB |
| Joining + extracting snapshot | ~1 core | 1-5 GiB |
| Syncing to head (feed catchup) | 3-6 cores | ~15 GiB |
| Steady state (synced, live) | ~0.5 cores | ~14 GiB |

Memory stays elevated after sync completes - pebble keeps a large portion of the database in the block cache. The jump to 3-6 cores during feed catchup is from processing the backlog of L2 transactions.

Our Kubernetes resource requests: `cpu: "4", memory: "8Gi"`. The 8 GiB request is conservative relative to the ~14-15 GiB actual usage during and after sync. Set limits accordingly or leave memory unlimited.

> During feed catchup the node is CPU-heavy: it executes the backlog of L2 transactions as fast as it can. The 4-core request lets the pod schedule on a reasonably sized node while allowing burst up to 6 cores. If your cluster enforces CPU limits, expect catchup to take longer.

## Sync phases

### Phase 1: Blob-fetching sync (historical)

Nitro replays historical L1 batches from the snapshot date forward. Batches posted after Dencun (EIP-4844, March 2024) are stored as blobs on the beacon chain rather than as calldata. Nitro fetches them from the beacon chain as it processes each batch.

Because our local Prysm node was still backfilling at this point and did not yet cover the older blob range, these historical blobs were served by Chainstack, the external beacon endpoint we ran as the primary during sync. Chainstack saw ~150-200 req/min during this phase. The phase ends once the node catches up near the chain head and switches to the sequencer feed.

(The beacon URL configuration - Chainstack as primary, local Prysm as fallback - is covered in the next section.)

### Phase 2: Feed-driven sync (near head)

Once close enough to the chain head, Nitro switches to receiving transactions from the sequencer feed (`wss://arb1.arbitrum.io/feed`). No blob fetching is needed in this mode - the feed delivers transaction data directly.

Observable: Chainstack beacon requests drop to ~0.

The log line `Reading message result remotely` repeating on the same `msgIdx` is normal during this phase - it is polling the sequencer feed while waiting for the next message to be confirmed:

```plaintext
INFO Reading message result remotely.  msgIdx=432935973
INFO Reading message result remotely.  msgIdx=432935973
```

### Phase 3: Live (synced)

`created block` logs appear at real-time cadence. The node is at head.

### Timing

Real measurements, pruned snapshot, April 2026:

| Phase | Duration |
| --- | --- |
| Download ~2.1 TiB (5 parts) | ~15 hours |
| Verification/hashing of existing parts | ~45 min (CPU-bound, ~2K IOPS reads) |
| Joining parts into archive | ~3 hours |
| Extraction to l2chaindata | ~3 hours (~10.6 GiB/min on SSD-backed storage) |
| Catch-up from snapshot date to head | ~10 hours (1.5M blocks at ~2,500 blocks/min) |

Total from scratch to synced: ~31 hours, of which ~15 hours is the download.

### Background cleanup during catchup

During Phase 1 and 2 you will see log lines like these:

```plaintext
INFO Unindexing transactions  blocks=44,635,000 txs=75,903,733 total=327,775,881 elapsed=14m33s
INFO Deleting tail epoch #1 in progress...
```

These are normal - not errors. Nitro is pruning old transaction indexes from the snapshot data in the background while it syncs.

## Beacon and blob setup

Nitro needs a beacon endpoint to fetch EIP-4844 blobs. We run two: an external Chainstack endpoint and a local Prysm instance. During the initial sync we set Chainstack as the primary and local Prysm as the fallback, because Prysm had only just been deployed and did not yet hold historical blobs. Nitro queries the primary first and falls back to the secondary when the primary does not have the blob.

```yaml
parent-chain:
  blob-client:
    beacon-url: "https://chainstack-url/beacon/..."  # primary (external Chainstack)
    secondary-beacon-url: "http://prysm.svc:3500"    # fallback (local Prysm)
```

The field names are `beacon-url` (primary) and `secondary-beacon-url` (fallback). During sync we pointed the primary at Chainstack so historical blobs resolved immediately, with local Prysm as the fallback.

**Why two URLs, and the switch:** local Prysm was deployed on April 20, 2026 with `--enable-backfill`. It takes approximately 18 days (~May 8) to backfill full blob history. Until then, Prysm returns `404` for pre-April-20 slots, so keeping Chainstack as the primary kept the sync from stalling on those misses. After ~May 8, once Prysm had full history, we flipped the primary to local Prysm and dropped the Chainstack `secondary-beacon-url`, relying on local Prysm alone.

### Monitoring blob fetches

Prysm exposes HTTP API request counts. Three useful queries:

```promql
# Blob request rate by status code
rate(http_request_count{exported_endpoint="blob.Blobs", job="prysm-beacon"}[5m])

# Blob request latency p99
histogram_quantile(0.99,
  rate(http_request_latency_seconds_bucket{exported_endpoint="blob.Blobs", job="prysm-beacon"}[5m])
)

# Backfill progress
backfill_blobs_download_count{job="prysm-beacon"}
```

> Prysm's HTTP metric label is exported\_endpoint, not endpoint. Prometheus relabeling renames it during scrape. A 200 status means Prysm has that blob locally; 404 means it has not backfilled that slot yet. While Prysm is still backfilling, Chainstack stays the primary so those gaps do not stall the sync.

## Kubernetes operational notes

### Diagnostic mode

We added a `diagnosticMode` flag to the Helm chart. When `true`, it replaces the Nitro command with a signal-handled sleep:

```yaml
command:
  - /bin/sh
  - -c
  - "trap 'exit 0' TERM INT; sleep infinity & wait"
```

The `sleep infinity & wait` form matters. `wait` is a shell builtin and is interrupted by signals immediately. Bare `sleep infinity` ignores SIGTERM, so `kubectl delete pod` hangs until the grace period expires - and if you resort to `--force --grace-period=0`, network teardown is skipped, leaving stale interfaces on the host.

### Stale veth after force-delete

Force-deleting a pod (`--force --grace-period=0`) bypasses network teardown. The host-side veth interface remains, causing the replacement pod to fail with:

```plaintext
Failed to create pod sandbox: ... container veth name ("eth0") peer provided ("gke33b862248e7") already exists
```

GKE kubelet GC resolves this automatically in ~10 minutes (observed). If urgent: drain and uncordon the node, or SSH in and `ip link delete <veth-name>`.

> Use --force --grace-period=0 only when you are certain the pod is truly stuck. The stale veth can prevent the replacement pod from starting on the same node for up to 10 minutes.

### Moving files between PVCs

`mv` across different PVCs falls back to copy+delete (different filesystems). Use rsync for large transfers:

```bash
rsync -a --info=progress2 /source/ /dest/
```

`--info=progress2` shows overall bytes/speed/ETA instead of per-file output - much cleaner for multi-TiB transfers.

## Snapshots

Browse available snapshots at [Arbitrum snapshot explorer](https://snapshot-explorer.arbitrum.io/)

Available types for Arbitrum One (`arb1`):

*   **Pruned** (recommended for RPC nodes): ~2.1 TiB compressed, new snapshot every ~12 days
    
*   **Archive-path**: much larger, for full history access
    
*   **Archive**: LevelDB format (legacy)
    

`init.latest: "pruned"` always picks the newest available pruned snapshot. To pin a specific snapshot:

```yaml
init:
  url: "https://snapshot.arbitrum.io/arb1/2026-04-08-a3b0e3cf/"
```

Pinning is useful when you have already downloaded parts and want to resume a specific snapshot after a crash. Set `init.url` to the same snapshot URL and `grab` will detect the existing parts by Content-Length and skip re-downloading them.

The [community Helm chart](https://github.com/OffchainLabs/community-helm-charts/tree/main/charts/nitro) ships with an empty `nitro.initContainers` stub - no built-in snapshot handling. The `tmp/` problem exists there too with no upstream workaround. Custom init containers can be injected via `initContainers:` (raw YAML passthrough).

> To verify what snapshot a running node initialized from, check the logs for INFO Downloading initial database url=... or INFO Successfully joined parts into archive. These appear once during the init phase and are not repeated on subsequent restarts.

The non-obvious part of this setup is that Nitro's default staging location for snapshot downloads sits inside the same directory it checks for unexpected files before initializing - so a crash leaves behind a file that permanently blocks the next start. Setting `init.download-path` to a separate PVC moves the staging artifacts out of that directory, making downloads crash-safe and resumable without manual intervention. For a closer look at the logic, [`cmd/nitro/init.go`](https://github.com/OffchainLabs/nitro/blob/v3.9.8/cmd/nitro/init.go) is the place to start.