How Google Builds and Tests Agent Skills

An Agent Skill can look deceptively small: a SKILL.md, a few references, perhaps a script. The maintenance problem is not small. Product APIs change, links rot, model behavior shifts, agent harnesses alter tool semantics, and instructions that helped one model may waste another model’s context.

Google published a useful behind-the-scenes account in August 2026 explaining how its Agent Skills program moved from a launch swarm to a governed library. The most important lesson is not a prompt-writing trick. It is that a skill must prove measurable value against a no-skill baseline, then keep proving it as the surrounding system changes.

TL;DR

• Google standardizes repository structure, metadata, naming, line limits, links, and instruction guardrails before behavioral evaluation begins.
• Every launch candidate supplies multiple prompts, expectations, and scoring rubrics.
• Evaluations compare an agent with and without the skill on accuracy and efficiency, not merely whether the skilled run looks good.
• Runs are repeated across agent frameworks, and the full library receives scheduled weekly checks.
• Long-term ownership is mandatory: maintainers own repository health; product teams own their skills when APIs or quality change.
• Internal-to-public export strips private assets, ownership data, and evaluation suites, which is a governance boundary as much as a publishing step.

From a swarm to a supply chain

Google says the project began as a cross-functional “swarm” before Google Cloud Next 2026. Developer Advocates and Technical Writers packaged Cloud knowledge into structured, agent-readable instructions. After the public repository attracted strong interest, more product teams wanted to contribute skills, including teams beyond Google Cloud.

Google’s August 2026 post describes the authoring, governance, CI, and evaluation process used to scale the library. Source: Google Cloud Blog.

That transition changes the problem. A small founding team can coordinate through conversation. A distributed set of product contributors needs a contribution contract, automated gates, ownership, release hygiene, and regression detection.

In other words, the skill library becomes a software supply chain. The files are instructions rather than binaries, but they can still cause a coding agent to execute commands, access data, create resources, or make costly mistakes.

Standardize the package before judging its intelligence

Google uses a consistent folder layout and frontmatter contract. Its automated check-in pipeline validates metadata, line counts, directory layout, naming conventions, and every URL. It also runs AI-assisted checklists to confirm required structures and guardrails.

These checks are intentionally boring. They turn easy-to-detect quality failures into deterministic CI errors:

Gate	What it prevents
Frontmatter schema	Skills that cannot be discovered or trigger correctly
Name and directory rules	Collisions and inconsistent installation paths
Line or size limits	One giant instruction file polluting context
Link checker	Dead or invented documentation references
Required sections	Missing validation, safety, or completion guidance
Script checks	Broken helpers and undeclared runtime assumptions

Behavioral evaluation is too expensive to spend on a malformed package. Lint first, then run agents.

The public google/skills repository shows the product-oriented catalog: Agent Platform APIs, Cloud basics, onboarding recipes, and Well-Architected Framework guidance, all under active development.

The Apache-2.0 repository is the exported public surface; Google evaluates skills internally before publishing. Source: google/skills.

The baseline is the heart of the evaluation

Many skill tests ask an agent to perform a task with the skill, then grade the result. That can tell you whether the run succeeded. It cannot tell you whether the skill helped.

Google compares two conditions:

same task + comparable environment + agent without skill
same task + comparable environment + agent with skill

The difference is the skill’s incremental value. This matters because modern coding agents already know common Cloud concepts. A 2,000-token skill that repeats model knowledge might produce the same answer more slowly. A good-looking skilled response can still be a negative product change.

The comparison needs controlled inputs: same task, model, tool availability, workspace state, and as close to the same sampling settings as the harness permits. Record version identifiers for the model, harness, skill, CLI, and referenced documentation. Otherwise a later regression has no reproducible environment.

Accuracy and efficiency form a 2×2 decision

Google measures two headline dimensions:

1. Accuracy: response quality and task completion rate.
2. Efficiency: consumed tokens and completion time.

That creates four outcomes:

Accuracy	Efficiency	Decision
Better	Better	Clear win; verify safety and maintainability
Better	Worse	Decide whether quality gain justifies cost and latency
Worse	Better	Usually reject; speed does not excuse wrong work
Worse	Worse	Reject and redesign

The upper-right quadrant is not the only acceptable result. A deployment skill may use extra tokens to prevent a dangerous configuration. The trade-off should be explicit and tied to task risk, not hidden behind an aggregate score.

“Accuracy” also needs decomposition. A useful rubric might separately score trigger selection, plan correctness, tool choice, argument correctness, artifact validity, safety compliance, and final explanation. One judge score can conceal that the output reads well while the wrong project was modified.

Repeat runs because agents are stochastic

Google reports running evaluations multiple times and across different agent frameworks to obtain statistically meaningful results. That protects against two common mistakes: celebrating one lucky run, and overfitting instructions to one host’s skill-loading behavior.

For each test case, store per-run data rather than only the mean:

• pass/fail and rubric dimensions;
• tool-call trajectory and denied actions;
• input/output and cached tokens;
• wall-clock time and retries;
• model and harness versions;
• activation decision and references loaded;
• failure category.

Report confidence intervals or at least pass counts across repeats. A skill that passes 6/10 times is not “60% correct” in a portable scientific sense, but that distribution is far more honest than showcasing the best transcript.

Cross-harness evaluation should include activation. A skill is useless if Gemini CLI loads it correctly but another supported agent never selects it because the description is vague. Conversely, an over-broad description can trigger the skill on unrelated tasks and increase cost or risk.

Unit tests and agent evals have different jobs

Google’s agents-cli workflow makes a sharp distinction: use pytest for code correctness and agents-cli eval for agent behavior. It explicitly warns against asserting nondeterministic response wording in unit tests.

The workflow requires evaluation after implementation and separates deterministic code contracts from response quality, tool use, and safety behavior. Source: google/agents-cli workflow skill.

The same division works for skills:

• Deterministic tests: metadata parses, scripts return expected types, sample files validate, URLs resolve, forbidden paths stay inaccessible.
• Behavioral evals: the agent activates correctly, follows the workflow, chooses appropriate tools, asks for approval, completes the task, and explains evidence.
• Adversarial evals: instructions embedded in retrieved content attempt to override the skill, request secrets, widen scope, or skip validation.

Do not use an LLM judge when a deterministic assertion exists. If the task creates a Cloud Run service with ingress disabled, inspect the resulting configuration. A judge’s opinion about the final paragraph is weaker evidence.

On-submit and weekly evaluations catch different failures

On-submit evaluation answers: “Does this proposed skill improve the current system?” Weekly scheduled evaluation answers: “Does the released skill still improve today’s system?”

The second question exists because dependencies move independently. A model update can follow an instruction differently. A CLI can rename a flag. An API can deprecate a field. An agent host can change skill discovery or consent behavior. No edit to the skill repository is required for the skill to regress.

Weekly checks should therefore pin a stable comparison lane and also run a current-production lane. The stable lane isolates skill changes; the current lane detects ecosystem drift. When the two disagree, maintainers know whether to repair the skill or investigate an upstream change.

At library scale, run a small canary suite frequently and a broader matrix on schedule. Reserve expensive cross-model, cross-harness, repeated runs for release candidates, high-risk skills, and periodic audits.

Remote MCP as the preferred tool boundary

Google’s published architecture prefers remote MCP tools when possible, using CLI or raw API calls as fallback. The stated reason is that MCP servers can provide agent-ready tools with authentication and IAM governance.

That preference is sensible when the remote server genuinely enforces identity and exposes narrow schemas. It is not an automatic security win. Evaluate the MCP server’s authorization, tenant scoping, tool descriptions, output handling, and prompt-injection exposure. A broad execute_command tool behind MCP is still broad command execution.

Skill evals should run with the same identity shape used in production. Testing with an owner credential hides missing permissions and makes unsafe calls appear reliable.

Public export is a security and product boundary

Google builds and evaluates internally, then uses automated export rules to publish a clean public repository. Internal assets, ownership information, and evaluation suites are removed.

Stripping private material prevents leakage, but exported skills still need validation as independent packages. An internal reference may be removed while an instruction continues to depend on it. Run lint, link checks, install tests, and a public-environment smoke eval against the exported artifact—not only the source tree.

Keeping evaluation suites private also limits community reproducibility. Organizations adopting this pattern can publish a sanitized core benchmark while retaining abuse cases or internal infrastructure details privately. The public suite documents expected behavior; the private suite protects sensitive failure modes.

Skills are products with named owners

Google assigns repository maintainers responsibility for CI and architecture, while skill owners remain responsible for API changes and quality regressions. This is the operational rule most teams skip.

Every skill should have:

• an owner and backup owner;
• supported models and harnesses;
• dependency and documentation inventory;
• risk tier and required approval behavior;
• baseline evaluation suite and thresholds;
• review or expiry date;
• rollback path to a known-good version.

Without an owner, a broken skill does not become shared responsibility. It becomes nobody’s priority.

This product mindset also appears in UiPath for Coding Agents, where individual skill maturity matters more than umbrella compatibility. The ecosystem is converging on the same lesson: instruction packages need release engineering.

A practical template for your own team

Start with one high-frequency task whose output can be validated. Capture 10–20 representative prompts, including ambiguous and unsafe requests. Run the agent without a skill, categorize failures, then write the minimum instructions and references that address those failures.

Add deterministic lint and script tests. Run repeated with/without evaluations. Review accuracy, token delta, latency, activation precision, and safety. If it passes, release to a small user group with versioned telemetry. Schedule regression runs and assign an owner.

When the skill stops helping, fix or retire it. A library should not grow by accumulation alone; deleting obsolete context is part of quality control.

FAQ

Why compare against an agent without the skill?

Because a successful skilled run does not prove incremental value. The baseline reveals whether the skill improves accuracy or efficiency over the agent’s existing capability.

How many times should each eval run?

There is no universal number. Use enough repeats to expose variance for the task’s risk and cost, and report the distribution. High-risk or unstable cases deserve more runs.

Should LLM judges grade every criterion?

No. Use deterministic inspection for files, configurations, tool calls, and policy outcomes. Reserve judges for genuinely qualitative dimensions with explicit rubrics.

Why test multiple agent frameworks?

Discovery, activation, tool semantics, context loading, and approval behavior differ by host. A portable skill must work beyond the environment in which it was authored.

Should evaluation suites be public?

Publish enough to communicate expected behavior and enable community checks. Keep cases private when they expose internal systems, abuse techniques, or sensitive data.

Bottom line

Google’s method treats Agent Skills as continuously evaluated products, not clever Markdown snippets. Structure and links are checked deterministically; behavioral value is measured against a baseline; stochastic runs are repeated across harnesses; owners remain accountable after launch.

That process is less exciting than writing the first prompt. It is also the reason a skill library can grow without turning every coding-agent session into an uncontrolled experiment.