Running an Arbitrum One RPC Node on Kubernetes
A routine Arbitrum Nitro deployment on Kubernetes turned into an hours-long crash loop over a single leftover tmp/ directory. Here's the snapshot-init trap, the fix, and what running the node in production actually takes.
We needed an Arbitrum One RPC node in our own infrastructure. The setup looked straightforward: deploy Nitro v3.9.8 as a GKE StatefulSet using a Helm chart, point it at our internal Geth for L1, configure init.latest: "pruned" to pull the latest snapshot, and let it do its thing. Chain ID 42161, non-validator, standard pruned sync. Nothing exotic.
The snapshot download ran for about 15 hours. Five parts, roughly 500 GiB each (the last one 163 GiB), totalling ~2.1 TiB. We watched the logs tick through pruned.tar.part0000 through pruned.tar.part0004 and waited. Then the pod crashed mid-extraction.
On the next restart, the node refused to initialize. The error was this:
ERROR error initializing database err="found 1 unexpected files in database directory, including: tmp"
That was it. No further detail. l2chaindata was never created - the database directory contained only the LOCK file and the tmp/ staging directory left behind from the download. Every restart produced the same error. Every restart ended in the same crash loop. There was no flag to skip the check, no --force-init, no obvious way to tell the node to proceed anyway.
To understand why this is not a bug you can work around but a design choice you have to accommodate, it helps to look at how Nitro decides whether it is safe to initialize at all.
How Nitro initializes from a snapshot
The init flags
Nitro's snapshot init system is configured through --init.* flags (or the JSON config equivalent):
init.latest: "pruned"- downloads the latest pruned snapshot automaticallyinit.url: "https://..."- pins to a specific snapshot URLinit.download-path- where to stage downloaded tar parts, separate from the database directory; if not set, Nitro defaults to<persistent.chain>/nitro/tmp/
Download flow
The sequence, from cmd/nitro/init.go:
Fetches the manifest from the snapshot URL
Downloads multipart tar files (
.part0000,.part0001, ...) todownload-pathJoins the parts into a single archive
Extracts the archive directly into
<persistent.chain>/nitro/l2chaindata/Cleans up the
tmp/staging directory - but only ifdownload-pathwas not explicitly set
checkEmptyDatabaseDir
Before any initialization work begins, Nitro checks the database directory against a strict allowlist in checkEmptyDatabaseDir:
// cmd/nitro/init.go
allowedFiles := map[string]bool{
"LOCK": true, "classic-msg": true, "l2chaindata": true,
}
for _, entry := range entries {
if !allowedFiles[entry.Name()] {
unexpectedFiles = append(unexpectedFiles, entry.Name())
}
}
if len(unexpectedFiles) > 0 {
return fmt.Errorf("found %d unexpected files in database directory, including: %s", ...)
}
When download-path is not set (the default), Nitro creates <persistent.chain>/nitro/tmp/ as the staging directory. If the pod crashes mid-download, tmp/ is left behind with whatever partial tar parts had been written. On the next restart, checkEmptyDatabaseDir finds tmp/ in the database directory - it is not in the allowlist - and the node refuses to start.
This is the most common failure mode when setting up a new Nitro node.
The check runs before any initialization work. There is no flag to skip it, no recovery mode. The only way forward is to either remove the unexpected files manually or prevent them from being there in the first place.
The fix: init.download-path
Set init.download-path to a path on a separate PVC. Nitro stages all download artifacts there, leaving the main data PVC containing only LOCK (and eventually l2chaindata). If the pod crashes, checkEmptyDatabaseDir passes on restart and the download resumes.
Resuming downloads
The download library Nitro uses - github.com/cavaliergopher/grab - detects already-downloaded files by comparing local file size to the server's Content-Length. Existing complete parts are skipped. This makes crash recovery essentially free - you lose at most one in-progress part.
Cleanup is manual
When download-path is set, Nitro does not clean up the download artifacts after extraction completes. The parts and joined archive remain on the init PVC. Delete them manually once you have confirmed the node is syncing.
Disk space math
| Item | Size | Location |
|---|---|---|
| Compressed tar parts (5 parts) | ~2.1 TiB | Init PVC |
| Joined archive (before extraction) | ~2.1 TiB | Init PVC (peak, simultaneous with parts) |
Extracted pebble database (l2chaindata) |
~600-700 GiB | Main data PVC |
Peak init PVC usage: ~4.2 TiB (parts + joined archive simultaneously). We used 6 TiB for headroom.
PVC sizing:
Init PVC: 6 TiB on cheaper HDD-backed storage - sufficient because downloads only need sequential throughput, not live-database IOPS
Main data PVC: 4 TiB on SSD-backed storage - needed for the IOPS of a live pebble database
Helm chart implementation
init.download-path is auto-injected into config.json via the Helm template when initStorage.enabled: true. When disabled, the init PVC is removed by Helm automatically - it is a standalone PVC, not a volumeClaimTemplate, so Helm can delete it without touching stateful data.
The extraction destination is always <persistent.chain>/nitro/l2chaindata/ regardless of download-path. Setting download-path only affects where the parts and joined archive live during the download phase.
Resource sizing
Observed on an 8-core node, April 2026:
| Phase | CPU | Memory |
|---|---|---|
| Downloading snapshot parts | ~1 core | 1-5 GiB |
| Joining + extracting snapshot | ~1 core | 1-5 GiB |
| Syncing to head (feed catchup) | 3-6 cores | ~15 GiB |
| Steady state (synced, live) | ~0.5 cores | ~14 GiB |
Memory stays elevated after sync completes - pebble keeps a large portion of the database in the block cache. The jump to 3-6 cores during feed catchup is from processing the backlog of L2 transactions.
Our Kubernetes resource requests: cpu: "4", memory: "8Gi". The 8 GiB request is conservative relative to the ~14-15 GiB actual usage during and after sync. Set limits accordingly or leave memory unlimited.
During feed catchup the node is CPU-heavy: it executes the backlog of L2 transactions as fast as it can. The 4-core request lets the pod schedule on a reasonably sized node while allowing burst up to 6 cores. If your cluster enforces CPU limits, expect catchup to take longer.
Sync phases
Phase 1: Blob-fetching sync (historical)
Nitro replays historical L1 batches from the snapshot date forward. Batches posted after Dencun (EIP-4844, March 2024) are stored as blobs on the beacon chain rather than as calldata. Nitro fetches them from the beacon chain as it processes each batch.
Because our local Prysm node was still backfilling at this point and did not yet cover the older blob range, these historical blobs were served by Chainstack, the external beacon endpoint we ran as the primary during sync. Chainstack saw ~150-200 req/min during this phase. The phase ends once the node catches up near the chain head and switches to the sequencer feed.
(The beacon URL configuration - Chainstack as primary, local Prysm as fallback - is covered in the next section.)
Phase 2: Feed-driven sync (near head)
Once close enough to the chain head, Nitro switches to receiving transactions from the sequencer feed (wss://arb1.arbitrum.io/feed). No blob fetching is needed in this mode - the feed delivers transaction data directly.
Observable: Chainstack beacon requests drop to ~0.
The log line Reading message result remotely repeating on the same msgIdx is normal during this phase - it is polling the sequencer feed while waiting for the next message to be confirmed:
INFO Reading message result remotely. msgIdx=432935973
INFO Reading message result remotely. msgIdx=432935973
Phase 3: Live (synced)
created block logs appear at real-time cadence. The node is at head.
Timing
Real measurements, pruned snapshot, April 2026:
| Phase | Duration |
|---|---|
| Download ~2.1 TiB (5 parts) | ~15 hours |
| Verification/hashing of existing parts | ~45 min (CPU-bound, ~2K IOPS reads) |
| Joining parts into archive | ~3 hours |
| Extraction to l2chaindata | |
| Catch-up from snapshot date to head | ~10 hours (1.5M blocks at ~2,500 blocks/min) |
Total from scratch to synced: ~31 hours, of which ~15 hours is the download.
Background cleanup during catchup
During Phase 1 and 2 you will see log lines like these:
INFO Unindexing transactions blocks=44,635,000 txs=75,903,733 total=327,775,881 elapsed=14m33s
INFO Deleting tail epoch #1 in progress...
These are normal - not errors. Nitro is pruning old transaction indexes from the snapshot data in the background while it syncs.
Beacon and blob setup
Nitro needs a beacon endpoint to fetch EIP-4844 blobs. We run two: an external Chainstack endpoint and a local Prysm instance. During the initial sync we set Chainstack as the primary and local Prysm as the fallback, because Prysm had only just been deployed and did not yet hold historical blobs. Nitro queries the primary first and falls back to the secondary when the primary does not have the blob.
parent-chain:
blob-client:
beacon-url: "https://chainstack-url/beacon/..." # primary (external Chainstack)
secondary-beacon-url: "http://prysm.svc:3500" # fallback (local Prysm)
The field names are beacon-url (primary) and secondary-beacon-url (fallback). During sync we pointed the primary at Chainstack so historical blobs resolved immediately, with local Prysm as the fallback.
Why two URLs, and the switch: local Prysm was deployed on April 20, 2026 with --enable-backfill. It takes approximately 18 days (~May 8) to backfill full blob history. Until then, Prysm returns 404 for pre-April-20 slots, so keeping Chainstack as the primary kept the sync from stalling on those misses. After ~May 8, once Prysm had full history, we flipped the primary to local Prysm and dropped the Chainstack secondary-beacon-url, relying on local Prysm alone.
Monitoring blob fetches
Prysm exposes HTTP API request counts. Three useful queries:
# Blob request rate by status code
rate(http_request_count{exported_endpoint="blob.Blobs", job="prysm-beacon"}[5m])
# Blob request latency p99
histogram_quantile(0.99,
rate(http_request_latency_seconds_bucket{exported_endpoint="blob.Blobs", job="prysm-beacon"}[5m])
)
# Backfill progress
backfill_blobs_download_count{job="prysm-beacon"}
Prysm's HTTP metric label is exported_endpoint, not endpoint. Prometheus relabeling renames it during scrape. A 200 status means Prysm has that blob locally; 404 means it has not backfilled that slot yet. While Prysm is still backfilling, Chainstack stays the primary so those gaps do not stall the sync.
Kubernetes operational notes
Diagnostic mode
We added a diagnosticMode flag to the Helm chart. When true, it replaces the Nitro command with a signal-handled sleep:
command:
- /bin/sh
- -c
- "trap 'exit 0' TERM INT; sleep infinity & wait"
The sleep infinity & wait form matters. wait is a shell builtin and is interrupted by signals immediately. Bare sleep infinity ignores SIGTERM, so kubectl delete pod hangs until the grace period expires - and if you resort to --force --grace-period=0, network teardown is skipped, leaving stale interfaces on the host.
Stale veth after force-delete
Force-deleting a pod (--force --grace-period=0) bypasses network teardown. The host-side veth interface remains, causing the replacement pod to fail with:
Failed to create pod sandbox: ... container veth name ("eth0") peer provided ("gke33b862248e7") already exists
GKE kubelet GC resolves this automatically in ~10 minutes (observed). If urgent: drain and uncordon the node, or SSH in and ip link delete <veth-name>.
Use --force --grace-period=0 only when you are certain the pod is truly stuck. The stale veth can prevent the replacement pod from starting on the same node for up to 10 minutes.
Moving files between PVCs
mv across different PVCs falls back to copy+delete (different filesystems). Use rsync for large transfers:
rsync -a --info=progress2 /source/ /dest/
--info=progress2 shows overall bytes/speed/ETA instead of per-file output - much cleaner for multi-TiB transfers.
Snapshots
Browse available snapshots at Arbitrum snapshot explorer
Available types for Arbitrum One (arb1):
Pruned (recommended for RPC nodes): ~2.1 TiB compressed, new snapshot every ~12 days
Archive-path: much larger, for full history access
Archive: LevelDB format (legacy)
init.latest: "pruned" always picks the newest available pruned snapshot. To pin a specific snapshot:
init:
url: "https://snapshot.arbitrum.io/arb1/2026-04-08-a3b0e3cf/"
Pinning is useful when you have already downloaded parts and want to resume a specific snapshot after a crash. Set init.url to the same snapshot URL and grab will detect the existing parts by Content-Length and skip re-downloading them.
The community Helm chart ships with an empty nitro.initContainers stub - no built-in snapshot handling. The tmp/ problem exists there too with no upstream workaround. Custom init containers can be injected via initContainers: (raw YAML passthrough).
To verify what snapshot a running node initialized from, check the logs for INFO Downloading initial database url=... or INFO Successfully joined parts into archive. These appear once during the init phase and are not repeated on subsequent restarts.
The non-obvious part of this setup is that Nitro's default staging location for snapshot downloads sits inside the same directory it checks for unexpected files before initializing - so a crash leaves behind a file that permanently blocks the next start. Setting init.download-path to a separate PVC moves the staging artifacts out of that directory, making downloads crash-safe and resumable without manual intervention. For a closer look at the logic, cmd/nitro/init.go is the place to start.
