Skip to lab
NextRunDigitalRL LabInteractive prototypeRL Lab platform ↗

Human feedback → Better model decisions

Review evidence. Resolve disagreement.

Inspect candidate responses, record rubric scores and export adjudicated preferences.

Synthetic cases only

Reviews stay in this browser. No upload, model call or shared reviewer service.

Download the complete scoring guide (0–4 anchors) ↓

Awaiting review

Payments and retries

Engineering architecture · calibration

Task: Design a retry-safe charging workflow. Explain the crash window, durable state and reconciliation. Do not claim exactly-once delivery.

System instruction and generation provenance

You are an engineering architecture assistant. Use only the supplied scenario facts. State assumptions and uncertainty. Provide a concise design, trade-offs, failure handling and validation. Do not invent measurements or claim changes were executed. Keep your answer below 220 words.

Model: Qwen/Qwen3-4B-Instruct-2507 · revision: cdbee75f17c01a7cc42f958dc650907174af0554. Temperature 0.7 · top-p 0.9 · max 512 new tokens. Candidate identity is shown during adjudication.

  • E1A queue delivers events at least once and may redeliver after 24 hours.
  • E2A worker charges a payment provider then records success in PostgreSQL.
  • E3The provider supports idempotency keys retained for 48 hours.

Actual unedited Qwen3-4B-Instruct-2507 baseline outputs on original synthetic scenarios. These calibration cases are excluded from training and final testing. Candidate order is counterbalanced between demo roles. Real independent, authenticated reviewers are not connected.

Response A

Blind candidate

344 tokens · stop: eos

**Retry-Safe Charging Workflow Design** - **Crash Window**: 24 hours (queue redelivery window). Workers must handle duplicates within this period. - **Durable State**: On successful charge, store a record in PostgreSQL with: - Event ID, - Payment provider ID, - Idempotency key, - Timestamp, - Status (success/failure). - **Idempotency Enforcement**: Use provider’s idempotency key (retained 48h) to detect duplicates. Before charging, validate if the same key already succeeded. If yes, skip. - **Reconciliation**: - After 48 hours, scan PostgreSQL for entries with no match in provider’s idempotency log. - Re-attempt only if provider confirms failure and idempotency key is still valid. - Log all retries and failures for audit. **Trade-offs**: - No exactly-once delivery; at-least-once with risk of duplicates. - 48h window aligns with provider retention, but longer than queue redelivery, allowing safe reconciliation. **Failure Handling**: - On failure, retry up to 3 times with exponential backoff (1m, 2m, 4m). - If all retries fail, record failure and alert. **Validation**: - Test with known idempotency keys to confirm no double charges. - Simulate redelivery after 24h and verify state consistency. *Assumption: Provider idempotency keys are reliable and not reused after 48h.*

0: core design wrong · 2: meaningful gaps · 4: sound mechanisms and limitations

0: contradicts requirements · 2: partial coverage · 4: justified choices across constraints

0: invented facts · 2: unstated assumptions · 4: evidence-linked claims and clear uncertainty

0: no usable plan · 2: partial checks · 4: actionable failure handling and tests

0: boundary violation · 2: ambiguity · 4: relevant permissions and isolation respected

Response B

Blind candidate

333 tokens · stop: eos

**Design:** Events from the queue (E1) are processed by a worker that charges the provider and writes success to PostgreSQL. Each event includes a unique idempotency key (E3). The worker checks PostgreSQL for existing records using the idempotency key before charging. If found, it skips the charge. If not, it proceeds. **Crash Window:** Up to 24 hours of redelivery (E1) may occur. The idempotency key is retained for 48 hours (E3), so charges within this window are idempotent. **Durable State:** Success is recorded in PostgreSQL with the idempotency key and timestamp. This serves as durable state. **Reconciliation:** After 48 hours, if an event was redelivered and no record exists in PostgreSQL, the worker reprocesses it. If a record exists, the charge is skipped. If the provider rejects a charge (e.g., due to invalid key), the worker logs and retries only if no prior success exists. **Trade-offs:** - Risk of duplicate charges if provider fails to validate idempotency. - Latency in reconciliation due to 48-hour retention. - No guarantee of exactly-once delivery due to redelivery and state lag. **Failure Handling:** On failure, the worker logs the error and retries with backoff. If the provider returns a 409 (conflict), it skips and logs. *No exactly-once delivery is claimed; design ensures idempotency within 48 hours.*

0: core design wrong · 2: meaningful gaps · 4: sound mechanisms and limitations

0: contradicts requirements · 2: partial coverage · 4: justified choices across constraints

0: invented facts · 2: unstated assumptions · 4: evidence-linked claims and clear uncertainty

0: no usable plan · 2: partial checks · 4: actionable failure handling and tests

0: boundary violation · 2: ambiguity · 4: relevant permissions and isolation respected

Your judgment

Evidence supporting your judgment

Mark at least one evidence item. Unsupported explanations should not become training targets.

Use this synthetic case only. Notes stay on this device and appear in your local export.

NextRun Digital · RL Lab · Review workbenchPublic demonstration · no customer or mailbox content