How does KCAI evaluate agencies?
DATA AS OF 2026-08-03 · METHODOLOGY v0.2
Evidence grades
| Grade | Meaning | Examples |
|---|---|---|
| E1 | Official / legal records | National Tax Service, DART filings |
| E2 | Platform-verified | Shopify, Google, Amazon partner directories |
| E3 | Client-confirmed projects | De-identified contracts, client emails |
| E4 | Independent public sources | Press, award bodies, client press releases |
| E5 | Agency self-reported | Agency websites, submitted forms |
| E6 | Editorial judgment | Classification inferred from multiple sources |
Trust grades
- A — verified entity + 5 client-verified projects (24 mo) + 3 independent source types
- B — verified entity + 2 client-verified projects + 2 independent source types
- C — entity or official partner verified; client evidence pending
- D — primarily self-reported
- U — unverified or conflicting data
D and U agencies may appear in the directory but are excluded from “best” rankings. Commercial relationships are structurally excluded from ranking inputs at the database-permission level.
Current version: v0.2
v0.2: until client-verified project data (V2+) accumulates, list pages (rankings, platform hubs, directory) default-sort by verified annual revenue (annual_revenue_krw), source cited, descending. Agencies without verified revenue are ranked after, and within that group the v0.1 evidence-strength rule still applies (official platform evidence E2 → source count → name). Scores are still not computed.
Change history
- v0.2 (2026-08-03) — Changed default list ordering from evidence strength to verified revenue and company-size information (employees, founding year). Reason: user feedback that company scale is more directly useful than evidence grade when deciding which agency to hire.
- v0.1 (bootstrap) — ordering used evidence grade and source count only; no scores were computed.
Ranking formula
v0.2 sorting rules (current)
| Priority | Input | Evidence grades | Description |
|---|---|---|---|
| 1st | Verified annual revenue (descending) | E1 (disclosure), E4 (company info) | Annual revenue confirmed by official or independent public sources. Source, year, and verification date shown. |
| 2nd (no verified revenue) | Evidence strength | E2 official directory presence | Agencies without verified revenue are ranked by official platform evidence (E2) presence. |
| 3rd (tied on 2nd) | Source count | E1–E4 combined | Agencies equally ranked on E2 or lacking E2 are ranked by number of verified sources (E1–E4). |
| 4th (final tiebreaker) | Agency name | — | If all conditions are equal, sort by name (alphabetically). |
Planned formula: v1 (Verified Outcomes based)
Once sufficient client-verified project data (V2+) accumulates, the following weighted formula will apply:
- Verified Outcomes: 30% — independently verified campaign and project results
- Bayesian Client Evidence: 25% — Bayesian trust inferred from V2+ reviews
- Category Specialization: 15% — platform and industry expertise
- Official Platform Credentials: 10% — Shopify Plus tier, Google/Amazon official certifications
- Delivery Continuity: 10% — recent project activity, project frequency
- Evidence Freshness: 10% — weight toward recent observations (24-month half-life applied)
This approach yields more accurate trust scores over time. Detailed implementation will be published after v0.2 is finalized.
Editorial separation and no paid placement
Current policy
KCAI absolutely prohibits paid listings, paid advertising, sponsorships, or any commercial relationship from influencing rank inputs. All sort criteria (verified revenue, evidence grade, source count) use only public facts, and commercial relationships are structurally excluded at the database-permission level.
Principles if ads or paid products are introduced
If KCAI introduces partner advertising or paid products (e.g., premium profiles) in the future, the following principles will be observed:
- Ranking input exclusion: commercial relationships will never be used as ranking inputs.
- Clear labeling: advertising and sponsored content will be visually and textually marked as such.
- Independent verification: agency core information (trust grade, sources, evidence) will be verified by identical standards regardless of paid status.
- Transparency log: any advertising or commercial agreements will be recorded in the changelog.
This policy is grounded in ADR-007 “Editorial separation” and prioritizes data transparency and user trust above all.
Data sources
Every source KCAI collects facts from, and the evidence grade attached to it.
| Source | Grade | Used for |
|---|---|---|
| National Tax Service business-registration status/verification API | E1 | Entity/registration status (planned — not yet integrated in the current dataset) |
| OpenDART | E1 | Financial disclosure confirmation (planned — not yet integrated in the current dataset) |
| Shopify Partner Directory (South Korea) | E2 | Partner tier, capability, and market confirmation |
| Google Partners Directory | E2 | Partner tier confirmation |
| Amazon Ads Partner Directory | E2 | Partner tier confirmation |
| Saramin · JobKorea · Wanted · NICE | E4 | Hiring listings and company-information checks (operating-continuity signal) |
| Client’s official website, press releases, or public filings | E4 | Portfolio and outcome-fact confirmation |
| Client-site technology fingerprinting | E4 | Direct confirmation of platform use in a portfolio item (e.g., detecting Shopify) |
| Agency’s own website, case studies, hiring listings | E5 | Agency self-reported facts (before cross-confirmation) |
How output and display work
- Sort order— v0.2: verified annual revenue (source cited), descending; agencies without verified revenue are ranked after. See the “Current version” section above for detail.
- Sample size N— the N in the research strip is the sample size shown on that specific list. On ranking/platform-hub pages it is the number of items in that list; on other pages it is the total number of agencies KCAI tracks.
- Update cadence— data is re-collected and re-verified weekly; site-wide data-quality metrics are re-assessed in a monthly measurement report.
- “Not verified” labeling principle— a value without a source URL is never published as a fact; it is shown as “not verified”. A value that exists but lacks a checkable source is not treated as a fact.
- Why scores are withheld— a composite score is not computed until client-verified project data (V2+) accumulates. Only displayable facts (revenue, evidence grade) are used as sort inputs in the meantime.
Limitations
- Sample bias — of the 33 agencies currently tracked, 24 (73%) were discovered through the Shopify partner directory, so the sample skews toward the Shopify channel. Coverage of agencies centered on other platforms, such as Cafe24, is comparatively lower.
- Share of self-reported (E5) evidence— most newly listed agencies start on self-reported (E5) evidence alone and are classified as trust grade D (18 agencies, 55%). Legal/official-record (E1) sources are not yet integrated in the current dataset (5facts) — automated National Tax Service/DART integration remains on the backlog.
- External citations— as of now, no outside press or research has cited KCAI data. If and when that changes, it will be reflected here and in the changelog.