76일차 자료 추가 (답 포함): 실습 15개 · 의도 · 생활 속 예 · 실행 결과, LAB-A · 정답 워크플로우와 25단계 실행 기록, 교안 개요 30장 · 4단계 정리, 바이브 코딩 tretry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mcchaeandClaude Opus 5.5 committed
1 parent
3f36605043
commit
bbf17e4696
141 files changed
+34072
No files matched your search
@@ -0,0 +1,4 @@
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
@@ -0,0 +1,33 @@
|
||||
# tretry IMP-PLAN
|
||||
|
||||
## 엔진 1단계 · 다시 하기 · 대체 · 종료 코드 (R1 · R2 · R3 · R4 · R12 · R13)
|
||||
- [x] retry.py: `Policy(tries, backoff, max_wait, jitter, deadline)`, `wait(i) = min(backoff × 2^(i-1), max_wait) × (1 - jitter × 무작위)`
|
||||
- [x] `run_with_retry(attempt, policy, fallback, per_try_timeout, sleep, clock, rand)` → `Result(ok, output, attempts, stopped, used_fallback, seconds)`
|
||||
- [x] `Outcome(ok, retryable, error, output, status, retry_after)`: 다시 할 수 있나는 실행기가 정함
|
||||
- [x] runners.command(argv): 종료 코드 ≠ 0 · 시간 초과 → 다시, 명령 없음 → 다시 안 함
|
||||
- [x] cli: `tretry [옵션] -- <명령>`, 종료 코드 0 · 1 · 2 · 3, `--json`
|
||||
- [x] tests: 가짜 시계 [0.5, 1.0], 상한 1 · 2 · 4 · 4, jitter, 대체, 다시 못 함은 1번 · 대체 안 함
|
||||
- 확인: `uv run pytest -q`, `uv run tretry --tries 3 --backoff 0.2 -- false; echo $?`
|
||||
|
||||
## 엔진 2단계 · HTTP · 멱등 키 · 기록 (R5 · R6 · R8)
|
||||
- [x] runners.http(method, url, body, headers, idem_key, body_key): 2xx 성공 · 5xx · 408 · 425 · 429 · 시간 초과 · 연결 실패 다시 · 그 밖 4xx 다시 안 함, `Retry-After`
|
||||
- [x] `--idem-key auto|값`: 실행에 한 번 만들어 헤더 `Idempotency-Key`, `--body-key`면 본문에도
|
||||
- [x] log.py: JSON Lines `~/.tretry/log.jsonl` (`--log`, `$TRETRY_LOG`), 시도마다 한 줄, 키 · 토큰은 `***`
|
||||
- [x] tests: 가짜 서버 500 → 201 · 400 · 느림, 같은 키 3번, 기록에 `secret123` · `sk-` 없음
|
||||
- 확인: 데이터 API `flaky`에서 `tretry http POST .../tickets --idem-key x --body-key key`
|
||||
|
||||
## TUI 1단계 · 기록 보기 (R11)
|
||||
- [x] app.py `LogApp`: 왼쪽 대상 순위(실패한 시도 많은 순), 고르면 오른쪽에 그 대상의 시도, `[f]` 실패만, `[r]` 다시 읽기
|
||||
- [x] 엔진(`log.stats` · `log.runs`)을 불러 쓰기만
|
||||
- [x] tests: `run_test`로 두 표 · `down` · `f`
|
||||
- 확인: `uv run tretry log`
|
||||
|
||||
## 설치 (R14)
|
||||
- [x] `[project.scripts] tretry = "tretry.cli:main"`
|
||||
- [x] `uv tool install --editable .`, `/tmp`에서 `tretry --help`
|
||||
|
||||
## 엔진 3단계 · 마감 · 순위 · 회로 차단 (R7 · R9 · R10)
|
||||
- [x] `--deadline`: 시도 전 남은 시간, 마감을 넘는 기다림은 하지 않음, 마지막 시도의 시간 제한을 남은 시간으로
|
||||
- [x] `tretry stats [--json]`: 대상마다 runs · 성공률 · 실행당 시도 · 실패한 시도 · 대체 · p95
|
||||
- [x] `--breaker N`(기본 5, 0 끔): 최근 60초 그 대상의 시도(대체 제외) N개가 모두 실패면 바로 대체, 대체가 없으면 멈춤
|
||||
- [x] tests: 3초 시도 · 마감 7 → 2번, 마감이 시간 제한을 줄임, 실패 5번 뒤 바로 대체, stats 숫자
|
||||
@@ -0,0 +1,37 @@
|
||||
# tretry · 강사가 수업 준비 때 직접 만든 본 (2026-10-29)
|
||||
|
||||
> **정답 자료** · 학생 저장소에는 수업 뒤 [VIBE-A-TRETRY.md](../VIBE-A-TRETRY.md)와 함께 올립니다.
|
||||
> 이 본은 강사가 CLI 에이전트에 단계 요청문을 보내 만든 것이 아니라, Claude Code가 VIBE-Q의 요구 R1~R14와 [Doc/IMP-PLAN.md](Doc/IMP-PLAN.md)를 보고 직접 썼습니다. 학생이 만든 것과 파일 구성이 달라도 됩니다. 요구의 **확인 방법**이 같으면 됩니다.
|
||||
|
||||
## 설치 · 실행
|
||||
|
||||
```bash
|
||||
cd ~/aio-01d4-stu/Last3Mon/76/tretry
|
||||
uv sync && uv run pytest -q # 테스트 17개 (기다리지 않음, 가짜 서버)
|
||||
uv tool install --editable . # 어느 폴더에서나 tretry (R14)
|
||||
cd /tmp && tretry --help
|
||||
```
|
||||
|
||||
| 하고 싶은 것 | 명령 |
|
||||
| --- | --- |
|
||||
| 명령을 다시 해 보기 | `tretry --tries 4 --backoff 0.5 --max-wait 8 --jitter 0.2 --timeout 10 -- <명령 ...>` |
|
||||
| 다 실패하면 대신 | `--fallback "cat ~/cache/faq.json"` |
|
||||
| 전체 마감 | `--deadline 5` |
|
||||
| HTTP 호출 | `tretry http POST http://localhost:8800/tickets --body '{"title": "t"}' --idem-key auto --body-key key` |
|
||||
| 순위 | `tretry stats` (`--json`) |
|
||||
| 화면(TUI) | `tretry log` (`[f]` 실패만, `[r]` 다시 읽기, `[q]` 끝) |
|
||||
| 다른 프로그램이 읽게 | `--json` |
|
||||
|
||||
- 종료 코드: 0 됨(대체 포함) · 1 다 실패 · 2 쓰는 법이 틀림 · 3 고칠 수 없는 실패 (HTTP 4xx, 408 · 425 · 429 빼고).
|
||||
- 기록: `~/.tretry/log.jsonl` (`--log`, `$TRETRY_LOG`). 헤더 값은 남기지 않고 URL · 오류의 `sk-…` · `Bearer …` · `api_key=…`는 `***`.
|
||||
- 회로 차단: 같은 대상의 최근 60초 시도(대체 제외) 5개가 모두 실패면 시도 없이 바로 대체 (`--breaker 0`은 끔). 대상은 `METHOD URL` 또는 명령 줄.
|
||||
|
||||
## 파일
|
||||
|
||||
| 파일 | 하는 일 | 요구 |
|
||||
| --- | --- | --- |
|
||||
| `src/tretry/retry.py` | 다시 하기 엔진 (`Policy` · `run_with_retry`), sleep · clock · 무작위를 바꿔 끼움 | R1 · R2 · R4 · R7 · R13 |
|
||||
| `src/tretry/runners.py` | 한 번 해 보기: 명령(종료 코드 · 시간 초과) · HTTP(상태 코드 · `Retry-After` · 멱등 키) | R1 · R3 · R5 · R6 |
|
||||
| `src/tretry/log.py` | JSON Lines 기록 · 키 지우기 · 순위 · 회로 차단 | R8 · R9 · R10 |
|
||||
| `src/tretry/cli.py` | 명령 · `--json` · 종료 코드 | R12 |
|
||||
| `src/tretry/app.py` | Textual 화면 | R11 |
|
||||
@@ -0,0 +1,23 @@
|
||||
[project]
|
||||
name = "tretry"
|
||||
version = "0.1.0"
|
||||
description = "Run a command or an HTTP call with timeout, retry (backoff), a total deadline and a fallback; keep a log and show it"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["textual>=0.80"]
|
||||
|
||||
[project.scripts]
|
||||
tretry = "tretry.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/tretry"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pytest>=8", "pytest-asyncio>=0.23"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
asyncio_default_fixture_loop_scope = "function"
|
||||
@@ -0,0 +1 @@
|
||||
"""tretry: retry, backoff, deadline, fallback and a log for any command or HTTP call."""
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Textual screen (R11): targets ranked by failed tries on the left, the runs of the chosen target on the right."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from textual.app import App, ComposeResult
|
||||
from textual.containers import Horizontal
|
||||
from textual.widgets import DataTable, Footer, Header, Static
|
||||
|
||||
from . import log
|
||||
|
||||
|
||||
class LogApp(App):
|
||||
TITLE = "tretry"
|
||||
BINDINGS = [("r", "reload", "Reload"), ("f", "failed_only", "Failed only"), ("q", "quit", "Quit")]
|
||||
CSS = "#targets { width: 55%; } #tries { width: 45%; } #note { height: auto; margin: 0 1; }"
|
||||
|
||||
def __init__(self, path: Path):
|
||||
super().__init__()
|
||||
self.path = Path(path)
|
||||
self.rows: list[dict] = []
|
||||
self.failed_only = False
|
||||
self.chosen: str | None = None
|
||||
|
||||
def compose(self) -> ComposeResult:
|
||||
yield Header()
|
||||
with Horizontal():
|
||||
yield DataTable(id="targets", cursor_type="row")
|
||||
yield DataTable(id="tries", cursor_type="row")
|
||||
yield Static("", id="note", markup=False)
|
||||
yield Footer()
|
||||
|
||||
def on_mount(self):
|
||||
self.sub_title = str(self.path)
|
||||
self.action_reload()
|
||||
|
||||
def action_reload(self):
|
||||
self.rows = log.read(self.path)
|
||||
table = self.query_one("#targets", DataTable)
|
||||
table.clear(columns=True)
|
||||
table.add_columns("target", "runs", "ok%", "tries/run", "failed", "fallback", "p95 s")
|
||||
for r in log.stats(self.rows):
|
||||
table.add_row(r["target"][:60], r["runs"], f"{r['success_rate'] * 100:.0f}%", r["attempts_per_run"],
|
||||
r["failed_attempts"], r["fallbacks"], f"{r['p95_seconds']:.2f}", key=r["target"])
|
||||
total = len(log.runs(self.rows))
|
||||
self.query_one("#note", Static).update(f"{total} run(s) · {len(self.rows)} tr(ies) in {self.path} · "
|
||||
"[enter] show tries · [f] failed only · [r] reload")
|
||||
if self.chosen is None and table.row_count:
|
||||
self.chosen = str(table.coordinate_to_cell_key((0, 0)).row_key.value)
|
||||
self.show_tries()
|
||||
|
||||
def action_failed_only(self):
|
||||
self.failed_only = not self.failed_only
|
||||
self.show_tries()
|
||||
|
||||
def on_data_table_row_highlighted(self, event: DataTable.RowHighlighted):
|
||||
if event.data_table.id == "targets" and event.row_key is not None:
|
||||
self.chosen = str(event.row_key.value)
|
||||
self.show_tries()
|
||||
|
||||
def show_tries(self):
|
||||
table = self.query_one("#tries", DataTable)
|
||||
table.clear(columns=True)
|
||||
table.add_columns("run", "try", "at +s", "result", "error", "s")
|
||||
for number, group in enumerate(log.runs(self.rows), 1):
|
||||
if group[-1]["target"] != self.chosen:
|
||||
continue
|
||||
for r in group:
|
||||
if self.failed_only and r["ok"]:
|
||||
continue
|
||||
table.add_row(number, "fallback" if r.get("fallback") else r["attempt"], f"{r['started']:.2f}",
|
||||
"✔" if r["ok"] else "✘", (r.get("error") or "")[:40], f"{r['seconds']:.2f}")
|
||||
@@ -0,0 +1,166 @@
|
||||
"""Command line. Exit codes: 0 worked (maybe by the fallback) · 1 all tries and the fallback failed ·
|
||||
2 usage error · 3 not retryable (e.g. HTTP 400: the request itself is wrong)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
from . import log, runners
|
||||
from .retry import Policy, Result, run_with_retry
|
||||
|
||||
SUBCOMMANDS = ("http", "stats", "log")
|
||||
|
||||
|
||||
def options(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--tries", type=int, default=3, help="how many tries (default 3)")
|
||||
parser.add_argument("--backoff", type=float, default=0.5, help="wait before try 2, doubled each time (default 0.5s)")
|
||||
parser.add_argument("--max-wait", type=float, default=8.0, help="longest single wait (default 8s)")
|
||||
parser.add_argument("--jitter", type=float, default=0.0, help="make each wait up to this share shorter, e.g. 0.2")
|
||||
parser.add_argument("--timeout", type=float, help="seconds for one try")
|
||||
parser.add_argument("--deadline", type=float, help="seconds for all tries together")
|
||||
parser.add_argument("--fallback", help='shell command to run once when every try failed, e.g. "cat cached.json"')
|
||||
parser.add_argument("--breaker", type=int, default=5,
|
||||
help="skip straight to the fallback when the last N tries at this target in 60s all failed (0 = off)")
|
||||
parser.add_argument("--log", type=Path, help="log file (default $TRETRY_LOG or ~/.tretry/log.jsonl)")
|
||||
parser.add_argument("--json", action="store_true", help="print the result as JSON for other programs")
|
||||
parser.add_argument("-q", "--quiet", action="store_true", help="print only the output of the command that worked")
|
||||
|
||||
|
||||
def policy(args) -> Policy:
|
||||
if args.tries < 1 or args.backoff < 0 or args.max_wait < 0 or not 0 <= args.jitter < 1:
|
||||
raise SystemExit(usage_error("--tries must be ≥ 1, --backoff/--max-wait ≥ 0, --jitter in [0, 1)"))
|
||||
return Policy(args.tries, args.backoff, args.max_wait, args.jitter, args.deadline)
|
||||
|
||||
|
||||
def usage_error(message: str) -> int:
|
||||
print(f"tretry: {message}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
def report(args, kind: str, target: str, result: Result, path: Path) -> int:
|
||||
log.write(path, kind, target, result)
|
||||
code = 0 if result.ok else (3 if result.stopped == "not-retryable" else 1)
|
||||
if args.json:
|
||||
print(json.dumps({"ok": result.ok, "exit_code": code, "stopped": result.stopped, "used_fallback": result.used_fallback,
|
||||
"seconds": result.seconds, "attempts": [a.__dict__ for a in result.attempts], "output": result.output},
|
||||
ensure_ascii=False, indent=2))
|
||||
return code
|
||||
if not args.quiet:
|
||||
for a in result.attempts:
|
||||
mark = "✔" if a.ok else "✘"
|
||||
label = "fallback" if a.fallback else f"#{a.attempt}"
|
||||
wait = f" (waited {a.waited:.2f}s)" if a.waited else ""
|
||||
print(f"{label:<9}{a.started:>7.2f}s {mark} {a.error or 'ok'} · {a.seconds:.2f}s{wait}", file=sys.stderr)
|
||||
how = "by the fallback" if result.ok and result.used_fallback else ("" if result.ok else f"({result.stopped})")
|
||||
tries = sum(1 for a in result.attempts if not a.fallback)
|
||||
state = " ".join(part for part in ("ok" if result.ok else "FAILED", how) if part)
|
||||
print(f"result: {state} · {tries} tr{'y' if tries == 1 else 'ies'} · {result.seconds:.2f}s"
|
||||
+ (" · breaker open: tries skipped" if result.stopped == "breaker" else ""), file=sys.stderr)
|
||||
if result.output:
|
||||
sys.stdout.write(result.output if result.output.endswith("\n") else result.output + "\n")
|
||||
return code
|
||||
|
||||
|
||||
def run(args, kind: str, target: str, attempt) -> int:
|
||||
path = args.log or log.default_path()
|
||||
skip = args.breaker > 0 and log.breaker_open(log.read(path), target, failures=args.breaker)
|
||||
if skip and not args.fallback:
|
||||
print(f"tretry: breaker open for {log.clean(target)} (last {args.breaker} tries failed); no --fallback, so stopping",
|
||||
file=sys.stderr)
|
||||
fallback = runners.command(args.fallback, shell=True) if args.fallback else None
|
||||
result = run_with_retry(attempt, policy(args), fallback=fallback, per_try_timeout=args.timeout, skip_tries=skip)
|
||||
return report(args, kind, target, result, path)
|
||||
|
||||
|
||||
def main_command(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(prog="tretry", usage="tretry [options] -- <command ...>",
|
||||
description="Run a command with timeout, retry (backoff), deadline and fallback.",
|
||||
epilog="also: tretry http METHOD URL · tretry stats · tretry log (screen)")
|
||||
options(parser)
|
||||
if "--" not in argv:
|
||||
if "-h" in argv or "--help" in argv:
|
||||
parser.print_help()
|
||||
return 0
|
||||
return usage_error("put the command after --, e.g. tretry --tries 3 -- curl -sf http://localhost:8800/health")
|
||||
split = argv.index("--")
|
||||
args = parser.parse_args(argv[:split])
|
||||
command = argv[split + 1:]
|
||||
if not command:
|
||||
return usage_error("no command after --")
|
||||
return run(args, "command", shlex.join(command), runners.command(command))
|
||||
|
||||
|
||||
def main_http(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(prog="tretry http", description="HTTP call: 5xx, 408/425/429, timeout and connection errors are "
|
||||
"tried again; other 4xx are not.")
|
||||
parser.add_argument("method")
|
||||
parser.add_argument("url")
|
||||
parser.add_argument("--body", help="JSON body, or @file.json")
|
||||
parser.add_argument("--header", action="append", default=[], help="K: V (values are never logged)")
|
||||
parser.add_argument("--idem-key", help="idempotency key sent with every try (header Idempotency-Key); 'auto' makes one")
|
||||
parser.add_argument("--body-key", help="also put the idempotency key in this JSON body field if it is missing, e.g. key")
|
||||
options(parser)
|
||||
args = parser.parse_args(argv)
|
||||
body = args.body
|
||||
if body and body.startswith("@"):
|
||||
body = Path(body[1:]).read_text(encoding="utf-8")
|
||||
if body:
|
||||
try:
|
||||
json.loads(body)
|
||||
except json.JSONDecodeError as error:
|
||||
return usage_error(f"--body is not JSON: {error}")
|
||||
headers = {}
|
||||
for item in args.header:
|
||||
name, _, value = item.partition(":")
|
||||
headers[name.strip()] = value.strip()
|
||||
key = uuid.uuid4().hex if args.idem_key == "auto" else args.idem_key
|
||||
if args.body_key and not key:
|
||||
return usage_error("--body-key needs --idem-key")
|
||||
target = f"{args.method.upper()} {args.url}"
|
||||
return run(args, "http", target, runners.http(args.method, args.url, body, headers, key, args.body_key))
|
||||
|
||||
|
||||
def main_stats(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(prog="tretry stats", description="Per target: runs, success rate, tries per run, p95 time.")
|
||||
parser.add_argument("--log", type=Path)
|
||||
parser.add_argument("--json", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
rows = log.stats(log.read(args.log or log.default_path()))
|
||||
if args.json:
|
||||
print(json.dumps(rows, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if not rows:
|
||||
print("no log yet")
|
||||
return 0
|
||||
print(f"{'target':<48} {'runs':>4} {'ok%':>5} {'tries/run':>9} {'failed':>6} {'fallback':>8} {'p95 s':>7}")
|
||||
for r in rows:
|
||||
print(f"{r['target'][:48]:<48} {r['runs']:>4} {r['success_rate'] * 100:>4.0f}% {r['attempts_per_run']:>9} "
|
||||
f"{r['failed_attempts']:>6} {r['fallbacks']:>8} {r['p95_seconds']:>7.2f}")
|
||||
return 0
|
||||
|
||||
|
||||
def main_log(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(prog="tretry log", description="Screen: targets by failures, and the tries of one target.")
|
||||
parser.add_argument("--log", type=Path)
|
||||
args = parser.parse_args(argv)
|
||||
from .app import LogApp
|
||||
LogApp(args.log or log.default_path()).run()
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
argv = sys.argv[1:] if argv is None else argv
|
||||
try:
|
||||
if argv and argv[0] in SUBCOMMANDS:
|
||||
return {"http": main_http, "stats": main_stats, "log": main_log}[argv[0]](argv[1:])
|
||||
return main_command(argv)
|
||||
except SystemExit as stop:
|
||||
return stop.code if isinstance(stop.code, int) else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,95 @@
|
||||
"""The attempt log as JSON Lines (R7), stats from it (R9) and the circuit breaker that reads it (R10)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
from .retry import Result
|
||||
|
||||
SECRET = re.compile(r"(sk-[A-Za-z0-9_-]{8,}|Bearer\s+\S+|(api[_-]?key|token|password)=[^&\s]+)", re.I)
|
||||
|
||||
|
||||
def default_path() -> Path:
|
||||
return Path(os.environ.get("TRETRY_LOG") or Path.home() / ".tretry" / "log.jsonl")
|
||||
|
||||
|
||||
def clean(text: str) -> str:
|
||||
"""Never keep a key or a token in the log, even when it is part of a URL or an error message."""
|
||||
return SECRET.sub("***", text)
|
||||
|
||||
|
||||
def write(path: Path, kind: str, target: str, result: Result, now: float | None = None) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
at = now if now is not None else time.time()
|
||||
with path.open("a", encoding="utf-8") as out:
|
||||
first = at - result.seconds # wall clock time of the first try
|
||||
for item in result.attempts:
|
||||
row = {"at": round(first + item.started, 3), "kind": kind, "target": clean(target), **asdict(item)}
|
||||
row["error"] = clean(row["error"])
|
||||
row["final"] = item is result.attempts[-1]
|
||||
row["result"] = "ok" if result.ok else "failed"
|
||||
row["stopped"] = result.stopped
|
||||
out.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def read(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
rows = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue # a half-written line from a killed run
|
||||
return rows
|
||||
|
||||
|
||||
def runs(rows: list[dict]) -> list[list[dict]]:
|
||||
"""Group attempt rows into runs (one tretry call = the rows up to and including its final row)."""
|
||||
groups, current = [], []
|
||||
for row in rows:
|
||||
current.append(row)
|
||||
if row.get("final"):
|
||||
groups.append(current)
|
||||
current = []
|
||||
return groups
|
||||
|
||||
|
||||
def p95(values: list[float]) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
ordered = sorted(values)
|
||||
return ordered[min(len(ordered) - 1, round(0.95 * (len(ordered) - 1)))]
|
||||
|
||||
|
||||
def stats(rows: list[dict]) -> list[dict]:
|
||||
"""Per target: runs, success rate, attempts per run, failed attempts, fallbacks, p95 run time. Most failures first."""
|
||||
table: dict[str, dict] = {}
|
||||
for group in runs(rows):
|
||||
target = group[-1]["target"]
|
||||
row = table.setdefault(target, {"target": target, "kind": group[-1]["kind"], "runs": 0, "ok": 0, "attempts": 0,
|
||||
"failed_attempts": 0, "fallbacks": 0, "times": []})
|
||||
row["runs"] += 1
|
||||
row["ok"] += group[-1]["result"] == "ok"
|
||||
row["attempts"] += len(group)
|
||||
row["failed_attempts"] += sum(1 for r in group if not r["ok"])
|
||||
row["fallbacks"] += any(r.get("fallback") for r in group)
|
||||
row["times"].append(group[-1]["started"] + group[-1]["seconds"])
|
||||
out = []
|
||||
for row in table.values():
|
||||
times = row.pop("times")
|
||||
out.append({**row, "success_rate": round(row["ok"] / row["runs"], 3), "attempts_per_run": round(row["attempts"] / row["runs"], 2),
|
||||
"p95_seconds": round(p95(times), 3)})
|
||||
return sorted(out, key=lambda r: (-r["failed_attempts"], r["target"]))
|
||||
|
||||
|
||||
def breaker_open(rows: list[dict], target: str, failures: int = 5, window: float = 60.0, now: float | None = None) -> bool:
|
||||
"""Open when the last `failures` attempts at this target (not fallbacks) all failed within `window` seconds."""
|
||||
now = now if now is not None else time.time()
|
||||
recent = [r for r in rows if r.get("target") == clean(target) and not r.get("fallback") and now - r["at"] <= window]
|
||||
last = recent[-failures:]
|
||||
return len(last) >= failures and not any(r["ok"] for r in last)
|
||||
@@ -0,0 +1,114 @@
|
||||
"""The retry engine (R1 · R2 · R6 · R13). Knows nothing about commands or HTTP: it calls attempt() and reads its Outcome.
|
||||
|
||||
sleep, clock and rand are arguments so tests never wait and always get the same numbers."""
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Outcome:
|
||||
"""What one attempt gave. retryable=False means trying again cannot help (e.g. HTTP 400)."""
|
||||
ok: bool
|
||||
retryable: bool = True
|
||||
error: str = ""
|
||||
output: str = ""
|
||||
status: int | None = None
|
||||
retry_after: float | None = None # seconds the server asked us to wait (HTTP 429 / 503 Retry-After)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Policy:
|
||||
tries: int = 3
|
||||
backoff: float = 0.5 # wait before try 2; doubles each time
|
||||
max_wait: float = 8.0 # one wait is never longer than this
|
||||
jitter: float = 0.0 # 0.2 = each wait is randomly 0..20% shorter, so many clients do not retry at the same moment
|
||||
deadline: float | None = None # seconds for all tries together; no new try that could not finish in time
|
||||
|
||||
def wait(self, attempt: int, rand: Callable[[], float] = random.random) -> float:
|
||||
"""Wait after attempt number `attempt` (1-based) and before the next one."""
|
||||
base = min(self.backoff * 2 ** (attempt - 1), self.max_wait)
|
||||
return round(base * (1 - self.jitter * rand()), 3)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Attempt:
|
||||
attempt: int
|
||||
started: float # seconds since the first try started
|
||||
seconds: float
|
||||
ok: bool
|
||||
error: str = ""
|
||||
status: int | None = None
|
||||
waited: float = 0.0 # wait before this attempt
|
||||
fallback: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
ok: bool
|
||||
output: str = ""
|
||||
attempts: list[Attempt] = field(default_factory=list)
|
||||
stopped: str = "" # why it stopped: success · tries · not-retryable · deadline · breaker
|
||||
used_fallback: bool = False
|
||||
seconds: float = 0.0
|
||||
|
||||
|
||||
def run_with_retry(attempt: Callable[[float | None], Outcome], policy: Policy, *,
|
||||
fallback: Callable[[float | None], Outcome] | None = None,
|
||||
per_try_timeout: float | None = None,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
rand: Callable[[], float] = random.random,
|
||||
skip_tries: bool = False) -> Result:
|
||||
"""Call attempt(timeout) until it works, up to policy.tries. Then fallback once, if given.
|
||||
|
||||
skip_tries=True goes straight to the fallback (the circuit breaker is open)."""
|
||||
start = clock()
|
||||
result = Result(ok=False)
|
||||
waited = 0.0
|
||||
number = 0
|
||||
while not skip_tries and number < policy.tries:
|
||||
number += 1
|
||||
timeout = per_try_timeout
|
||||
if policy.deadline is not None:
|
||||
left = policy.deadline - (clock() - start)
|
||||
if left <= 0:
|
||||
result.stopped = "deadline"
|
||||
break
|
||||
timeout = min(timeout, left) if timeout else left
|
||||
began = clock()
|
||||
outcome = attempt(timeout)
|
||||
result.attempts.append(Attempt(number, round(began - start, 3), round(clock() - began, 3), outcome.ok,
|
||||
outcome.error, outcome.status, waited))
|
||||
if outcome.ok:
|
||||
result.ok, result.output, result.stopped = True, outcome.output, "success"
|
||||
break
|
||||
if not outcome.retryable:
|
||||
result.output, result.stopped = outcome.output, "not-retryable"
|
||||
break
|
||||
if number == policy.tries:
|
||||
result.stopped = "tries"
|
||||
break
|
||||
waited = policy.wait(number, rand)
|
||||
if outcome.retry_after is not None:
|
||||
waited = min(max(waited, outcome.retry_after), policy.max_wait)
|
||||
if policy.deadline is not None and clock() - start + waited >= policy.deadline:
|
||||
result.stopped = "deadline"
|
||||
break
|
||||
sleep(waited)
|
||||
if skip_tries:
|
||||
result.stopped = "breaker"
|
||||
# a request that cannot be right (4xx) gets no fallback: the fallback would hide the caller's mistake
|
||||
if not result.ok and fallback is not None and result.stopped != "not-retryable":
|
||||
began = clock()
|
||||
outcome = fallback(per_try_timeout)
|
||||
result.attempts.append(Attempt(len(result.attempts) + 1, round(began - start, 3), round(clock() - began, 3),
|
||||
outcome.ok, outcome.error, outcome.status, 0.0, fallback=True))
|
||||
result.used_fallback = True
|
||||
if outcome.ok:
|
||||
result.ok, result.output = True, outcome.output
|
||||
result.seconds = round(clock() - start, 3)
|
||||
return result
|
||||
@@ -0,0 +1,71 @@
|
||||
"""One attempt of a command or of an HTTP call, as an Outcome (R1 · R3 · R4 · R5)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import socket
|
||||
import subprocess
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
from .retry import Outcome
|
||||
|
||||
# 408 timeout · 425 too early · 429 too many requests: waiting can help. Other 4xx: the request itself is wrong.
|
||||
RETRY_4XX = {408, 425, 429}
|
||||
|
||||
|
||||
def command(argv: list[str] | str, shell: bool = False):
|
||||
"""attempt(timeout) for a command: exit code 0 is success; a timeout is retryable."""
|
||||
def attempt(timeout: float | None) -> Outcome:
|
||||
try:
|
||||
done = subprocess.run(argv, shell=shell, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
return Outcome(False, True, f"timeout {timeout:.1f}s")
|
||||
except FileNotFoundError as error:
|
||||
return Outcome(False, False, f"not found: {error.filename}")
|
||||
if done.returncode == 0:
|
||||
return Outcome(True, output=done.stdout)
|
||||
tail = (done.stderr or done.stdout).strip().splitlines()[-1:] or [""]
|
||||
return Outcome(False, True, f"exit {done.returncode} {tail[0][:120]}".strip(), output=done.stdout)
|
||||
return attempt
|
||||
|
||||
|
||||
def retry_after(value: str | None) -> float | None:
|
||||
try:
|
||||
return max(0.0, float(value)) if value else None
|
||||
except ValueError:
|
||||
return None # an HTTP date: ignore, use our own backoff
|
||||
|
||||
|
||||
def http(method: str, url: str, body: str | None = None, headers: dict[str, str] | None = None,
|
||||
idem_key: str | None = None, body_key: str | None = None):
|
||||
"""attempt(timeout) for an HTTP call: 2xx success · 5xx, 408/425/429, timeout, connection error retryable ·
|
||||
other 4xx not retryable. The same idempotency key goes with every try (header, and the body field if asked)."""
|
||||
data = None
|
||||
send = dict(headers or {})
|
||||
if body is not None:
|
||||
if body_key and idem_key:
|
||||
parsed = json.loads(body)
|
||||
parsed.setdefault(body_key, idem_key)
|
||||
body = json.dumps(parsed, ensure_ascii=False)
|
||||
data = body.encode("utf-8")
|
||||
send.setdefault("Content-Type", "application/json")
|
||||
if idem_key:
|
||||
send["Idempotency-Key"] = idem_key
|
||||
|
||||
def attempt(timeout: float | None) -> Outcome:
|
||||
request = urllib.request.Request(url, data=data, method=method.upper(), headers=send)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout or 30) as response:
|
||||
return Outcome(True, output=response.read().decode("utf-8", "replace"), status=response.status)
|
||||
except urllib.error.HTTPError as error:
|
||||
text = error.read().decode("utf-8", "replace")
|
||||
retryable = error.code >= 500 or error.code in RETRY_4XX
|
||||
return Outcome(False, retryable, f"HTTP {error.code} {text[:120]}".strip(), output=text, status=error.code,
|
||||
retry_after=retry_after(error.headers.get("Retry-After")))
|
||||
except (TimeoutError, socket.timeout):
|
||||
return Outcome(False, True, f"timeout {timeout or 30:.1f}s")
|
||||
except urllib.error.URLError as error:
|
||||
if isinstance(error.reason, (TimeoutError, socket.timeout)):
|
||||
return Outcome(False, True, f"timeout {timeout or 30:.1f}s")
|
||||
return Outcome(False, True, f"connection: {error.reason}")
|
||||
return attempt
|
||||
@@ -0,0 +1,60 @@
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class FakeClock:
|
||||
"""sleep() only moves the clock, so a test never waits and the waits can be checked."""
|
||||
def __init__(self):
|
||||
self.now = 0.0
|
||||
self.slept: list[float] = []
|
||||
|
||||
def sleep(self, seconds: float):
|
||||
self.slept.append(seconds)
|
||||
self.now += seconds
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clock():
|
||||
return FakeClock()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def server():
|
||||
"""A local server whose answers come from a script: path -> list of (status, body, headers), one per call."""
|
||||
script: dict[str, list] = {}
|
||||
seen: list[dict] = []
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def answer(self):
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
seen.append({"path": self.path, "method": self.command, "headers": dict(self.headers),
|
||||
"body": self.rfile.read(length).decode() if length else ""})
|
||||
queue = script.get(self.path) or [(200, {"ok": True}, {})]
|
||||
status, body, headers = queue.pop(0) if len(queue) > 1 else queue[0]
|
||||
if status == "sleep":
|
||||
import time
|
||||
time.sleep(body)
|
||||
status, body, headers = 200, {"late": True}, {}
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
for key, value in headers.items():
|
||||
self.send_header(key, value)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
do_GET = do_POST = answer
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
threading.Thread(target=httpd.serve_forever, daemon=True).start()
|
||||
yield f"http://127.0.0.1:{httpd.server_port}", script, seen
|
||||
httpd.shutdown()
|
||||
@@ -0,0 +1,29 @@
|
||||
"""R11: the screen ranks targets by failed tries and lists the tries of the chosen one."""
|
||||
from textual.widgets import DataTable
|
||||
|
||||
from tretry import log
|
||||
from tretry.app import LogApp
|
||||
from tretry.retry import Attempt, Result
|
||||
|
||||
|
||||
def make_log(tmp_path):
|
||||
path = tmp_path / "log.jsonl"
|
||||
log.write(path, "http", "POST http://x/tickets", Result(False, attempts=[Attempt(1, 0, 3, False, "timeout"), Attempt(2, 3.5, 3, False, "timeout")],
|
||||
stopped="tries", seconds=6.5), now=100)
|
||||
log.write(path, "command", "curl ok", Result(True, attempts=[Attempt(1, 0, 0.1, True)], stopped="success", seconds=0.1), now=101)
|
||||
return path
|
||||
|
||||
|
||||
async def test_tables(tmp_path):
|
||||
app = LogApp(make_log(tmp_path))
|
||||
async with app.run_test() as pilot:
|
||||
await pilot.pause()
|
||||
targets = app.query_one("#targets", DataTable)
|
||||
assert targets.row_count == 2 and targets.get_row_at(0)[0] == "POST http://x/tickets" # most failures first
|
||||
assert app.query_one("#tries", DataTable).row_count == 2
|
||||
await pilot.press("down")
|
||||
await pilot.pause()
|
||||
assert app.query_one("#tries", DataTable).row_count == 1
|
||||
await pilot.press("f")
|
||||
await pilot.pause()
|
||||
assert app.query_one("#tries", DataTable).row_count == 0
|
||||
@@ -0,0 +1,91 @@
|
||||
"""R1 · R3 · R4 · R5 · R7 · R8 · R9 · R10 through the command line (real commands, local fake server, temp log)."""
|
||||
import json
|
||||
import sys
|
||||
|
||||
from tretry import cli, log
|
||||
|
||||
PY = sys.executable
|
||||
|
||||
|
||||
def run(capsys, *argv):
|
||||
code = cli.main(list(argv))
|
||||
out = capsys.readouterr()
|
||||
return code, out.out, out.err
|
||||
|
||||
|
||||
def test_command_fails_then_fallback(tmp_path, capsys):
|
||||
logfile = tmp_path / "log.jsonl"
|
||||
code, out, err = run(capsys, "--tries", "2", "--backoff", "0", "--log", str(logfile), "--fallback", "echo cached",
|
||||
"--", PY, "-c", "import sys; sys.exit(3)")
|
||||
assert code == 0 and out.strip() == "cached" and "by the fallback" in err
|
||||
rows = log.read(logfile)
|
||||
assert [r["fallback"] for r in rows] == [False, False, True] and rows[0]["error"].startswith("exit 3")
|
||||
|
||||
|
||||
def test_command_timeout(tmp_path, capsys):
|
||||
code, _, err = run(capsys, "--tries", "1", "--timeout", "0.3", "--log", str(tmp_path / "l"), "--",
|
||||
PY, "-c", "import time; time.sleep(5)")
|
||||
assert code == 1 and "timeout 0.3s" in err
|
||||
|
||||
|
||||
def test_http_5xx_retried_4xx_not(server, tmp_path, capsys):
|
||||
base, script, seen = server
|
||||
script["/flaky"] = [(500, {"error": "x"}, {}), (201, {"ticket_id": "T-1"}, {})]
|
||||
script["/bad"] = [(400, {"error": "title is required"}, {})]
|
||||
logfile = str(tmp_path / "log.jsonl")
|
||||
code, out, _ = run(capsys, "http", "POST", base + "/flaky", "--body", "{}", "--backoff", "0", "--log", logfile)
|
||||
assert code == 0 and "T-1" in out and len(seen) == 2
|
||||
code, _, err = run(capsys, "http", "POST", base + "/bad", "--body", "{}", "--backoff", "0", "--log", logfile,
|
||||
"--fallback", "echo cached")
|
||||
assert code == 3 and len(seen) == 3 and "not-retryable" in err # one try, no fallback
|
||||
|
||||
|
||||
def test_http_same_idempotency_key_every_try(server, tmp_path, capsys):
|
||||
base, script, seen = server
|
||||
script["/tickets"] = [(500, {}, {}), (500, {}, {}), (201, {"ticket_id": "T-1"}, {})]
|
||||
code, _, _ = run(capsys, "http", "POST", base + "/tickets", "--body", '{"title": "t"}', "--idem-key", "auto",
|
||||
"--body-key", "key", "--backoff", "0", "--log", str(tmp_path / "l"))
|
||||
keys = {s["headers"]["Idempotency-Key"] for s in seen}
|
||||
bodies = {json.loads(s["body"])["key"] for s in seen}
|
||||
assert code == 0 and len(seen) == 3 and len(keys) == 1 and keys == bodies # R5
|
||||
|
||||
|
||||
def test_http_timeout_is_retried(server, tmp_path, capsys):
|
||||
base, script, seen = server
|
||||
script["/slow"] = [("sleep", 1.0, {}), (200, {"ok": True}, {})]
|
||||
code, _, err = run(capsys, "http", "GET", base + "/slow", "--timeout", "0.3", "--backoff", "0", "--log", str(tmp_path / "l"))
|
||||
assert code == 0 and "timeout" in err and len(seen) == 2
|
||||
|
||||
|
||||
def test_log_never_keeps_a_key(server, tmp_path, capsys):
|
||||
base, script, _ = server
|
||||
script["/x?api_key=secret123"] = [(503, {}, {})]
|
||||
logfile = tmp_path / "log.jsonl"
|
||||
run(capsys, "http", "GET", base + "/x?api_key=secret123", "--tries", "1", "--header", "Authorization: Bearer sk-abcdefghijklmnop",
|
||||
"--log", str(logfile))
|
||||
text = logfile.read_text()
|
||||
assert "secret123" not in text and "sk-abcdefghijklmnop" not in text and "***" in text # R7
|
||||
|
||||
|
||||
def test_json_output_and_exit_code(tmp_path, capsys):
|
||||
code, out, _ = run(capsys, "--tries", "2", "--backoff", "0", "--json", "--log", str(tmp_path / "l"), "--", PY, "-c", "raise SystemExit(1)")
|
||||
data = json.loads(out)
|
||||
assert code == 1 and data["exit_code"] == 1 and data["stopped"] == "tries" and len(data["attempts"]) == 2 # R8
|
||||
|
||||
|
||||
def test_stats_and_breaker(tmp_path, capsys):
|
||||
logfile = str(tmp_path / "log.jsonl")
|
||||
fail = ["--backoff", "0", "--log", logfile, "--", PY, "-c", "raise SystemExit(1)"]
|
||||
run(capsys, "--tries", "3", *fail)
|
||||
run(capsys, "--tries", "2", *fail) # 5 failed tries in a row → breaker opens
|
||||
code, out, err = run(capsys, "--tries", "3", "--fallback", "echo cached", *fail)
|
||||
assert code == 0 and out.strip() == "cached" and "breaker open" in err # R10: no try, straight to fallback
|
||||
code, out, _ = run(capsys, "stats", "--log", logfile, "--json")
|
||||
row = json.loads(out)[0]
|
||||
assert row["runs"] == 3 and row["failed_attempts"] == 5 and row["fallbacks"] == 1 and row["success_rate"] == 0.333 # R9
|
||||
|
||||
|
||||
def test_usage_errors(capsys):
|
||||
assert run(capsys, "echo", "hi")[0] == 2 # no --
|
||||
assert run(capsys, "http", "POST", "http://x", "--body", "{bad")[0] == 2
|
||||
assert run(capsys, "--tries", "0", "--", "true")[0] == 2
|
||||
@@ -0,0 +1,67 @@
|
||||
"""R1 · R2 · R3 · R6: the engine with fake attempts and a fake clock (no waiting)."""
|
||||
from tretry.retry import Outcome, Policy, run_with_retry
|
||||
|
||||
|
||||
def failing(times, retryable=True):
|
||||
calls = []
|
||||
|
||||
def attempt(timeout):
|
||||
calls.append(timeout)
|
||||
if len(calls) <= times:
|
||||
return Outcome(False, retryable, f"boom {len(calls)}")
|
||||
return Outcome(True, output="done")
|
||||
return attempt, calls
|
||||
|
||||
|
||||
def test_success_after_two_failures_waits_with_backoff(clock):
|
||||
attempt, calls = failing(2)
|
||||
result = run_with_retry(attempt, Policy(tries=4, backoff=0.5), sleep=clock.sleep, clock=clock)
|
||||
assert result.ok and result.output == "done" and len(calls) == 3
|
||||
assert clock.slept == [0.5, 1.0] # R1: 0.5 → 1.0 (doubles)
|
||||
|
||||
|
||||
def test_wait_has_a_cap_and_jitter_only_shortens():
|
||||
policy = Policy(tries=10, backoff=1, max_wait=4, jitter=0.5)
|
||||
assert [policy.wait(n, rand=lambda: 0.0) for n in range(1, 6)] == [1, 2, 4, 4, 4] # R1: cap
|
||||
assert policy.wait(3, rand=lambda: 1.0) == 2.0 # R2: up to 50% shorter
|
||||
|
||||
|
||||
def test_all_fail_then_fallback(clock):
|
||||
attempt, _ = failing(9)
|
||||
result = run_with_retry(attempt, Policy(tries=3), fallback=lambda t: Outcome(True, output="cached"),
|
||||
sleep=clock.sleep, clock=clock)
|
||||
assert result.ok and result.used_fallback and result.output == "cached" # R3
|
||||
assert [a.fallback for a in result.attempts] == [False, False, False, True]
|
||||
|
||||
|
||||
def test_not_retryable_stops_at_once_and_skips_fallback(clock):
|
||||
attempt, calls = failing(9, retryable=False)
|
||||
result = run_with_retry(attempt, Policy(tries=5), fallback=lambda t: Outcome(True, output="cached"),
|
||||
sleep=clock.sleep, clock=clock)
|
||||
assert not result.ok and len(calls) == 1 and result.stopped == "not-retryable" and not result.used_fallback
|
||||
|
||||
|
||||
def test_deadline_stops_before_a_wait_that_cannot_fit(clock):
|
||||
def slow(timeout):
|
||||
clock.now += 3 # every try takes 3 s
|
||||
return Outcome(False, True, "timeout")
|
||||
result = run_with_retry(slow, Policy(tries=5, backoff=1, deadline=7), sleep=clock.sleep, clock=clock)
|
||||
assert result.stopped == "deadline" and len(result.attempts) == 2 # R6: 3 + 1 + 3 = 7 → no third try
|
||||
assert result.seconds <= 7
|
||||
|
||||
|
||||
def test_deadline_shortens_the_last_timeout(clock):
|
||||
seen = []
|
||||
|
||||
def attempt(timeout):
|
||||
seen.append(timeout)
|
||||
clock.now += timeout
|
||||
return Outcome(False, True, "timeout")
|
||||
run_with_retry(attempt, Policy(tries=3, backoff=0.5, deadline=5), per_try_timeout=3, sleep=clock.sleep, clock=clock)
|
||||
assert seen == [3, 1.5] # 3 + 0.5 wait → 1.5 s left for try 2
|
||||
|
||||
|
||||
def test_retry_after_is_respected_up_to_the_cap(clock):
|
||||
answers = [Outcome(False, True, "HTTP 429", status=429, retry_after=3), Outcome(True, output="ok")]
|
||||
result = run_with_retry(lambda t: answers.pop(0), Policy(tries=3, backoff=0.5, max_wait=8), sleep=clock.sleep, clock=clock)
|
||||
assert result.ok and clock.slept == [3] # R4: the server's 3 s, not our 0.5 s
|
||||
Generated
+195
@@ -0,0 +1,195 @@
|
||||
version = 1
|
||||
revision = 3
|
||||
requires-python = ">=3.11"
|
||||
|
||||
[[package]]
|
||||
name = "colorama"
|
||||
version = "0.4.6"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "iniconfig"
|
||||
version = "2.3.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/01/e1/2069291243c926a2ff1cd706c7f3eeb9b62144bf60f77c9fb9ff2fb26bd3/iniconfig-2.3.1.tar.gz", hash = "sha256:67f4b9c50da0dedf52af349e7749a80a9057a5031199791b906c3bb3ae878960", size = 21209, upload-time = "2026-10-06T22:48:38.076Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/56/43/4ca9e49d27a1fcf6bece6f6aec0ea46bb9112489b93d4b688fb415457bdb/iniconfig-2.3.1-py3-none-any.whl", hash = "sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", size = 7552, upload-time = "2026-10-06T22:48:36.959Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "linkify-it-py"
|
||||
version = "2.2.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/45/98/7a1a5f31fd5c7ba93e963b168e244b8e3dd705b3d2a718e3c3307583bf57/linkify_it_py-2.2.0.tar.gz", hash = "sha256:907acd2d17ac1fbb9ddb62c8957ccbd6158cac602231a15c3b0cd1e215f03cee", size = 32939, upload-time = "2026-08-29T07:07:08.305Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/13/d4/1152d1c7ab42d8b908be64fd200ddc870dc9d4925e951198702084aa1a7d/linkify_it_py-2.2.0-py3-none-any.whl", hash = "sha256:3adc40eb5af300b2605fcfdb968c24e1d780a90f1f2221af7c15e5111e94d443", size = 21971, upload-time = "2026-08-29T07:07:07.164Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "markdown-it-py"
|
||||
version = "4.2.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "mdurl" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
linkify = [
|
||||
{ name = "linkify-it-py" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "mdit-py-plugins"
|
||||
version = "0.6.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "markdown-it-py" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "mdurl"
|
||||
version = "0.1.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "packaging"
|
||||
version = "26.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "platformdirs"
|
||||
version = "4.14.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/40/28/1c058a9143d2304b3078f8dadcbb9dc27f7d3581b5d05157c84869c4cc0d/platformdirs-4.14.0.tar.gz", hash = "sha256:83f009c1e8c5fc36869b5171755b291887bd2cade7965233cdfabd23d754720b", size = 70387, upload-time = "2026-10-28T18:27:30.973Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/69/2b/887969ef4bb53e0cb271dc7c3f4a73d50c5d58bb88aeaae311e3340cd57e/platformdirs-4.14.0-py3-none-any.whl", hash = "sha256:1dadfc59a8a5d7565b3d9270ae8fa71c065595352c4277b41c11ee6c222c4b91", size = 35259, upload-time = "2026-10-28T18:27:29.659Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pluggy"
|
||||
version = "1.7.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/bf/db/7fc19e6f2dc92a966727031389fc2e08b558f0f25eb7403c1119ad4713cd/pluggy-1.7.0.tar.gz", hash = "sha256:d1eaa46ebb595891b860ab086b4d09c8588af65ebd4361b8e8f4bb8920b90ba8", size = 123304, upload-time = "2026-10-15T09:50:58.343Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/40/9e/2b38731e0fc536806f16490e1a12d7f0dc2a1235aa8cc07bcc75416a7daa/pluggy-1.7.0-py3-none-any.whl", hash = "sha256:7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec", size = 27082, upload-time = "2026-10-15T09:50:56.808Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pygments"
|
||||
version = "2.21.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/49/2e/ced460408999b33da6b31b0021b0f37d329e202d4169aeb164493778f25b/pygments-2.21.0.tar.gz", hash = "sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c", size = 5005329, upload-time = "2026-08-17T08:02:48.824Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pytest"
|
||||
version = "9.2.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "colorama", marker = "sys_platform == 'win32'" },
|
||||
{ name = "iniconfig" },
|
||||
{ name = "packaging" },
|
||||
{ name = "pluggy" },
|
||||
{ name = "pygments" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/79/42/bb4641652e58dedeac2ec51abc1f8cae4fee4ec63899a0999016229a0f2a/pytest-9.2.0.tar.gz", hash = "sha256:97c4f3b1f3c7414aa36465e09b8220f1d03627993bd51b1282e193753c80557b", size = 1797555, upload-time = "2026-10-27T21:30:20.728Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/51/39/1f5d8c8877a0fa327db959c19341d4be61660de1ab6b7391f9eb805bcd67/pytest-9.2.0-py3-none-any.whl", hash = "sha256:07acdcf88ec174f907dc01af70bf52d49b014bbab03faeeade31c96b18c2e588", size = 411776, upload-time = "2026-10-27T21:30:19.131Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pytest-asyncio"
|
||||
version = "1.4.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pytest" },
|
||||
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f8/9a/887535e477847f5159377ab35303923644840b13c0ca03c0a467f84d4680/pytest_asyncio-1.4.1.tar.gz", hash = "sha256:e8266eb094e5ee9ffdb2068c9836f33e209ea77a970eb846933d416b5602669b", size = 145237, upload-time = "2026-10-25T21:05:37.135Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/46/d7/cd9e2708afdbf059658adaf08bdb0792ca52489871051d6ce7d498b51277/pytest_asyncio-1.4.1-py3-none-any.whl", hash = "sha256:507afa78ec77c6661164ef1a2368bc7820383fd1a14b2a7f813de8b5a92949cc", size = 17061, upload-time = "2026-10-25T21:05:35.473Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rich"
|
||||
version = "15.0.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "markdown-it-py" },
|
||||
{ name = "pygments" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "textual"
|
||||
version = "8.2.8"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "markdown-it-py", extra = ["linkify"] },
|
||||
{ name = "mdit-py-plugins" },
|
||||
{ name = "platformdirs" },
|
||||
{ name = "pygments" },
|
||||
{ name = "rich" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/00/21/39a76b01bd5eea82a04baaca7580e105d8c59450df03998345bb2cfb307b/textual-8.2.8.tar.gz", hash = "sha256:3f106a9fbc73e39dd266c9712432087de78a6d644084c7c241d6a25c3169115b", size = 1860502, upload-time = "2026-06-30T06:51:24.495Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/fb/be/35261223d9416a0751cdff1c7b4a6f881387218a12d439fe22fefebc8c04/textual-8.2.8-py3-none-any.whl", hash = "sha256:267375fd402dc8d981457212efa71f0e3365fd17bba144ba9bb3ed7563cb374a", size = 731418, upload-time = "2026-06-30T06:51:26.364Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tretry"
|
||||
version = "0.1.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "textual" },
|
||||
]
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
{ name = "pytest" },
|
||||
{ name = "pytest-asyncio" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [{ name = "textual", specifier = ">=0.80" }]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [
|
||||
{ name = "pytest", specifier = ">=8" },
|
||||
{ name = "pytest-asyncio", specifier = ">=0.23" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typing-extensions"
|
||||
version = "4.16.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" },
|
||||
]
|
||||
Reference in new issue
Block a user