Agentic Setup Checklist The full stack most teams skip — router docs, self-healing systems, cross-agent review, worksheets, and night-shift validation
← Part of the AI Agents knowledge base
·
AI
·
Productivity
Source & Credit
Outline and recommendations from
Jamon Holmgren
(@jamonholmgren ),
founder/CEO of Infinite Red.
Original thread:
x.com/i/status/2076001786700394610
Jamon’s framing: too many people burn tokens without the basics below — and then say they “don’t have time.”
One person with a full life built this (and more). Treat the list as a checklist you grow into, not a day-one mandate.
Why This Matters
Dry prompting + babysitting every step is a different experience from an agent that can route itself, run the app,
write tests, review with other models, leave a resume-able worksheet, and improve your workflow from its own feedback.
This sits next to
loops, not prompts ,
stop babysitting your agents ,
and
harness-level coding tips
— but as an implementable inventory of systems, not only philosophy.
How to Use This Page Scan 0–18 as a maturity map for your agentic coding stack. Check off what you already have; star the next 2–3 gaps that would cut the most rework. Prefer systems that fix and document over systems that only flag problems. Commit worksheets, feedback docs, and tags with the work so sessions compose over time. The Checklist (0–18) 0. AGENTS.md as a router
A top-level AGENTS.md that does not try to hold everything — it routes the agent to the right
skills, docs, and tools. Think index + dispatch, not encyclopedia.
1. Standard workflow doc / skill
A customized workflow skill you pull into most sessions (e.g. @/AGENT_WORKFLOW.md).
If you have nothing yet, start from solid public skill packs (Matt Pocock’s skills are a common baseline).
Related: Claude skills patterns , agent config .
2. Self-healing docs for every system
Agents are instructed to keep system docs updated. Tag the docs you know you need, or let the agent find them via AGENTS.md.
Put a denser summary in the first 7 lines of every doc so they are greppable; document that convention in AGENTS.md.
3. Agents always run the app
The agent should start the app itself, exercise its work, and fix issues as it goes — especially in autonomous / async runs.
“Looks good in the editor” is not done. Related: building agent loops .
4. End-to-end tests + living test docs
Instructions to write and update E2E tests; docs on how to write tests, what to avoid, and a markdown inventory of
all tests and what they cover. Write and run targeted tests during implementation; improve and commit with the work.
5. Custom linters at pre-commit (that actually fix)
Pre-commit hooks catch problems you’ve already paid for once. Prefer --fix autofix.
If autofix is not feasible, shell out to a cheaper model (e.g. Composer 2.5 / Sonnet) to clean the code —
not only flag issues, but leave cleaned code.
6. Cross-agent review at major gates
At research, plan, implementation, and wrap-up: have a different agent/model review
(Codex, Claude, Cursor, etc. — not the same model reviewing its own work). Maintain review docs: what to look for, how to approach.
Use personas (maintainability, quality, security, performance, “AI smells,” domain experts) and let each persona
own and update a set of system docs. Related: multi-agent review gates .
7. Agent traces / worksheets
Session worksheets that track what the agent is doing. If it fails mid-run, another agent can pick up the worksheet and finish.
Commit the worksheet with the work. Have the agent apply git tags that match worksheet names for later lookup.
You will reference these later.
8. Automatic end-of-session feedback
At session end, agent writes feedback into a committed doc. Periodically ingest that feedback in an interactive session
and improve workflows, skills, and AGENTS.md. The loop closes on the harness, not only the product code.
9. Tools / bin folder + skills to grow it
A tools/ or bin/ of python/bash scripts the agent can run and is instructed to extend.
Example: an agent_review script so the agent kicks off reviews via CLI without memorizing each vendor’s flags.
Document how to write effective scripts and keep expanding them.
10. Periodic sweeps across recent commits
Higher-level agent passes over recent commits looking for problems and gotchas that single-PR review misses.
11. Coding conventions doc
Project-specific conventions review agents use heavily. Push as much as possible into linters (item 5);
keep the markdown for judgment calls linters cannot own.
12. Agent loop / night-shift skill
A skill that defines how the agent approaches autonomous work from an orchestration standpoint —
night shift, background runs, long loops without constant human steering.
Related: loops not prompts , verification + background routines .
13. Task queue the agent can access
Simple is fine: TODOS.md. Or Linear/etc. with a CLI so the agent can fetch and update via API.
Work without a queue turns into chat amnesia.
14. False-confidence test audit skill
Periodic skill that hunts tests which do not actually test what you think they test — then fixes them.
Green CI that lies is worse than a failing suite.
15. Visual regression tests
Screenshots + tool comparison + agent visual review. Commit with the work (Git LFS is useful) or at least attach to the PR.
16. Automatic performance benchmarks
Tests that notice when performance degrades so agents (and humans) cannot “accidentally” ship a regression.
17. Performance profiling tools for agents
Profilers agents can run for targeted benchmarks: try techniques, compare outputs, compare profiles —
not only watch a single CI threshold.
18. End-of-shift full validation
When the night shift ends: all tests, performance, agent reviews, sweeps — everything.
You return to a tree that is as pristine as the system can make it.
Suggested Rollout Order If you are starting from almost nothing, do not implement 0–18 in one weekend. A practical sequence:
Router + workflow: 0, 1, 11 — so every session starts with the same map and conventions.Prove work in the real app: 3, 4 — run the app, write targeted tests with the change.Make failures transferable: 7, 13 — worksheets + queue so another agent (or you) can resume.Stop repeating the same bugs: 5, 2 — autofixing linters + self-healing docs.Second opinions: 6, 9 — cross-agent review + CLI helpers that hide vendor ceremony.Close the learning loop: 8, 10 — session feedback + commit sweeps.Autonomous mode: 12, 18 — night-shift skill + end-of-shift validation.Quality of tests and UX: 14, 15, 16, 17 — false-confidence audits, visual + performance systems. Quick Scorecard Band Rough coverage What it feels like Prompt-only 0–2 of 19 You re-explain the repo every session; tokens burn, progress is fragile. Guided agent 3–7 Router, workflow, app runs, basic tests — useful, still high babysitting. Harnessed 8–13 Worksheets, autofix, multi-model review, queue — agents compose across sessions. Night-shift ready 14–19 Autonomous loops, false-confidence audits, visual/perf gates, full end-of-shift validation.
Related Guides Loops, Not Prompts Boris Cherny on writing autonomous agent loops instead of one-shot prompts.
Read the deep dive → Building AI Agent Loops Cron + judgment, reusable skills, agent-friendly CLIs.
Read the tutorial → Stop Babysitting Your Agents Verification loops, self-improving skills, multi-agent parallelization, background routines.
Read the guide → Harness: AI Coding Tips Day-to-day practices that keep agentic coding effective.
Read the tips → Agent Config Configuration patterns for agent systems and project-level instructions.
Read the guide → Grok Build Workflow Human-in-the-loop multi-agent pipeline with review gates — pairs with item 6.
Read the tutorial → ChatGPT Work Product-side agent mode for finished deliverables; different surface, same “don’t just chat” mindset.
Read the action guide → ChatGPT + Codex Super App Computer use, loops, multi-thread, Sites — power-user Codex workflows that pair with this harness checklist.
Read part 2 → Ryan Carson — Team of Agents Live founder setup: Devon PR factory, Sentry automation, OnePassword prod secrets, Codex local.
Watch & read → AI Agent Coding Stack Code graphs, lossless context, and output design when scaling agentic development.
Read the stack →
Comments
Approved comments appear below. Log in once with GFAVIP — it applies across the whole site. GFAVIP login
View comments archive