Non-Blocking Eval Dispatch

Non-Blocking Eval Dispatch#

Port of miles@ba28991834 (ray/rollout/eval_dispatch.py).

By default vime evaluates on the shared rollout engines with a blocking call in the training loop. The eval dispatcher adds the infrastructure for snapshot-based, non-blocking evaluation: each eval point exports an HF snapshot of the current actor weights and runs the eval asynchronously, bounded by an in-flight limit with a configurable overflow policy.

Flags#

Flag

Default

Meaning

--eval-num-gpus

0

Dedicated eval engine fleet size. >0 runs eval on its own engines behind their own router, synced by loading exported HF snapshots, never by joining training weight updates. 0 keeps shared-engine eval.

--eval-num-gpus-per-engine

1

GPUs per eval engine (TP size).

--eval-hf-dir

None

Staging dir for per-eval HF snapshots at {eval_hf_dir}/step_{rollout_id} (tmpfs recommended). Required for snapshot eval (--eval-num-gpus or a CheckpointEvalFn); --save-hf checkpoints are never reused as eval snapshots. Requires --eval-interval.

--eval-max-in-flight

2

Max concurrently pending async evals.

--eval-overflow-policy

backpressure

backpressure awaits the oldest pending eval; skip drops the point and logs eval/skipped_busy.

--eval-keep-snapshots

2

Exported snapshot dirs retained under --eval-hf-dir before GC.

Eval fleet#

With --eval-num-gpus N, the vLLM config gains an eval model (update_weights=False) behind its own vllm-router, and N extra GPU bundles are appended to the placement group past every other region — trainer offsets never shift, including in colocate layouts. Each eval point pins the fleet: every engine loads the snapshot via reload_weights, reads back the weight version (one stale engine skips the point as pin_violation), and passes a health probe. The eval itself runs on a driver-owned EvalRunner actor, so a slow snapshot load never occupies the RolloutManager actor.

Known gaps (v1): eval engines are not part of the detached, recoverable serving topology — they die with the driver and are not restarted by fault tolerance. --eval-num-gpus is not supported with external rollout engines or --prefill-num-servers; with --vllm-config the YAML must define exactly one model named eval covering --eval-num-gpus GPUs.

CheckpointEvalFn#

Point --eval-function-path at a subclass of vime.rollout.checkpoint_eval.CheckpointEvalFn to eval snapshots on a backend you own (an external vLLM server, a scoring service, a black-box HTTP API). Each eval point delivers a finished HF checkpoint directory to evaluate_checkpoint(checkpoint_dir, input); raise EvalSkip(reason) to skip a point with attribution. Mutually exclusive with --eval-num-gpus (the fleet already owns weight delivery); requires --eval-hf-dir. examples/eval/external_vllm_eval_fn.py is a working implementation, and retarget_args(args, router_ip, router_port, engine_gpu_counts) redirects the stock eval datasets at any vLLM router.

Behavior#

  • The dispatcher runs on a daemon thread with its own event loop, driven from the synchronous driver (train.py); drain() settles all pending points before teardown.

  • Without the eval fleet (--eval-num-gpus 0) or a CheckpointEvalFn, eval_uses_snapshots is false and dispatch degrades to the previous blocking shared-engine eval.

  • Every skipped point is reported with a reason (busy, export_failed, crashed); under --ci-test any skip fails the run.