Methodology

This dashboard scores payment rails against the Ethereum Foundation's CROPS properties: Censorship Resistance, Open Source and Free, Privacy, Security. It is built for institutions choosing a rail, so every rule below is chosen so that a score can be defended to a risk committee by pointing at an artifact anyone can re-check.

2.1 Unit of assessment

One row is one provider, scored on its primary (flagship) chain or settlement path. The chain used is named in the provider file and shown in the row header. Notes about other deployments may be recorded; they do not affect the score.

Why: onchain evidence is chain-specific. Scoring every deployment separately is more honest but multiplies maintenance; scoring the flagship keeps one verifiable target per provider while stating plainly which deployment was checked. If you are evaluating a different deployment, treat the score as unverified for it.

2.2 Scale

Each criterion is scored 0–3 or unknown:

• 3: property fully held and verified from code or chain.
• 2: property substantially held, with a bounded, disclosed compromise and a working user escape.
• 1: property weakly held, or held only on documented (non-code) evidence.
• 0: property absent, only claimed, or disproven by evidence.
• unknown: no admissible evidence submitted yet.

Each criterion page gives the anchors for every level.

Why: four levels are enough to separate "holds", "holds with a disclosed escape hatch", "weak or paper-only" and "absent", which are the distinctions a risk reader acts on. Finer scales invite false precision.

2.3 Evidence tiers and caps

Evidence tiers

Tier	What qualifies	Caps score at
verified	Onchain state anyone can read (contract address plus role, function or storage slot, and the block number at check time); source in a public repository at a pinned commit SHA and file path; or deployed bytecode matched to source on Sourcify or an explorer.	3
documented	Terms of service, privacy policy, audit report, technical docs, or an observed network trace with written reproduction steps. Must be URL-linked and dated.	1
claimed	Vendor statements, blog posts, marketing, support replies, third-party articles.	0

A criterion's cap is the highest tier among its evidence items. Evidence that lowers a score is admissible when it is verified, documented, or the provider's own statement (a privacy policy disclosing data sharing, a trace showing an analytics call, a vendor's admission). A third party's claim cannot lower a score on its own: it is recorded in the finding and may mark the cell disputed until someone supplies admissible evidence.

Why: institutions need scores they can defend to a risk committee. Only artifacts a third party can re-check without trusting the provider or the contributor meet that bar. Documents are useful but describe intent, not behaviour: they may establish that something exists and surface problems, never that a property holds. Claims are inadmissible because they cannot be re-checked.

2.4 Pillar rollup

Pillar score = the minimum of its criterion scores, with unknown counting as 0. The grid shows the limiting criterion next to each pillar score. There is no overall composite score and no leaderboard. The ratings page shows a CROPS tally: how many criteria hold up (Partial or Strong), fall short (Weak or Fails) or have no evidence. It is a count with every criterion weighted equally, not a score, and it does not change any pillar rating.

Why: a single unbounded freeze key or an unverifiable custody path is disqualifying however well the rest scores. Means and medians hide exactly the finding a risk reader needs. Counting unknown as 0 keeps the burden of proof on the evidence, not the reader, and gives providers and contributors a direct incentive to fill gaps. No composite exists because the weights between pillars are a judgment for the reader, not the dashboard.

2.5 Categories

Each provider has a type: checkout, card, remittance, or incumbent. An incumbent is a rail with no public chain settlement and no public code (for example M-Pesa). Incumbents are scored with the identical rubric and caps and shown in a separate, labelled baseline group, never interleaved with onchain rails.

Why: readers use incumbents daily and will dismiss an unexplained 0. Stated as a verifiability gap (rails whose behaviour can be inspected versus rails whose behaviour must be trusted), the comparison is accurate and defensible. It is not a claim that the incumbent is unsafe.

2.6 Freshness

Every evidence item records when it was checked (checked_at), plus the block number for onchain items and the commit for repository items. A weekly job re-reads every source it can: recorded onchain roles and parameters, every pinned repository file, and every quoted document, across ratings, availability and the operational layer. When the source is unchanged it records the date as confirmed_at. A cell is fresh if its newest evidence was checked or confirmed in the last 180 days; otherwise it shows a stale badge. Confirmation means the source still says the same thing; it does not re-judge the rating. A changed licence or lockfile, a changed onchain value, or a quote that has gone from its page is drift. Drift found by the weekly job opens an issue and marks the cell under review. Until a human re-checks it, the cell counts as unknown in the pillar rollup, because a changed fact can no longer support the old rating. The job never edits a score.

Changes to other code files are not drift, because the pinned commit still holds; they are listed in a weekly summary issue for a human to look at. Pages that cannot be read (blocked, client-rendered, or down) are listed there too and are neither confirmed nor marked as drift.

Why: roles get rotated, licences get changed and terms get rewritten. A score is only as good as the date it was checked, so the date is shown and every machine-readable source is re-read automatically.

2.7 Right of reply

Each criterion on each provider has an optional provider response, submitted by pull request like any other change. It is shown beside the score and never changes it. A response that links new admissible evidence should come with a score change in the same pull request, reviewed as normal.

Why: providers know things contributors miss, and readers should see their side. Keeping responses separate from scores means the only way to move a score is the same way anyone else does: with an artifact.

2.8 Review

All changes land by pull request. CODEOWNERS names the maintainers for data/. A pull request that touches a score must pass CI and one maintainer review; one that sets a score of 2 or 3 needs two reviewers, at least one not affiliated with the contributor. Contributors and reviewers declare conflicts of interest in the pull request, and nobody affiliated with a provider approves changes to that provider's ratings. Every rating records who assessed it and whether, by whom and when it was reviewed. A provider's dispute gets a maintainer response within 30 days and can be appealed to a second maintainer.

Why: CI enforces the mechanical rules (tiers, caps, commits that exist, URLs that resolve). It cannot tell whether an artifact actually supports the score, so a human re-runs the check before merging.

2.9 Evidence integrity rules

CI cannot judge whether an artifact supports a rating, so the rules make that judgment cheap for a human and hard to fake:

• Any rating of 2 or 3 carries a re-check command that reproduces the fact, and code evidence behind it cites exact line ranges, which CI confirms exist at the pinned commit.
• Onchain evidence must point at an address that has code, unless it is marked as an account.
• A 3 on a contract-behaviour criterion (admin powers, direct settlement, custody, upgrades) needs onchain evidence about the deployed contracts, not only source at a commit. A 3 on a client-behaviour criterion (filtering, telemetry, third parties, approvals, secrets) needs the distributed package version and hash, or a network trace.
• Each provider lists every component in its payment path, public or closed. A closed required component cannot be forked, so the licensing criterion is 0, and criteria that reward the absence of a problem cannot reach 3 while part of the request path is hidden.
• Document evidence quotes the passage relied on. CI checks the live page still contains it; pages that block automated access or render in the browser are cited through a committed copy.
• The unit of assessment (which chain and deployment) is stated with its evidence, so it cannot be chosen to flatter.

Why: a red team of the first version found that CI checked only that artifacts existed, that closing code could raise a score, and that the deployment, the package, and the quoted text could all drift from what was cited. Each rule closes one of those gaps.

2.10 What this does not assess

CROPS rates inspectability, admin powers, licensing, data flows and custody. It does not rate regulation and supervision, safeguarding of customer funds, deposit or e-money protection, legal recourse, operational track record, liquidity, or cost. A risk assessment should weigh all of these. Where a provider's file records such protections, they are shown as context under "Outside this assessment" and never scored.

Why: without this statement a low rating on custody could be read as a claim that a regulated, safeguarded rail is unsafe. The ratings measure what can be verified from code and chain, nothing more.

2.10a Country availability

Each provider may state where its product is offered, taken only from its own terms, documents or code, as a global, allow-list or block-list of countries. The Countries pages show, per country, which rails say they are offered there, which say they are not, and which publish nothing. Availability is recorded as evidence with the same rules as a rating, and it never enters any score.

Why: readers choosing a rail need to know whether it is offered to them, and that fact is often buried in terms of service. Keeping it out of the scores avoids rewarding reach, which is a commercial property, not a CROPS one. Comparison sites are excluded as sources because their lists cannot be re-checked against anything the provider committed to.

2.10c Settlement token (cards)

For cards, the provider page records what the balance is held in and debited from: each token, its class, the issuer, the contract, and when it is converted to fiat. Classes are stablecoin (fiat-backed, crypto-backed or synthetic), native token (a chain's gas token), platform token (issued by the provider or its ecosystem) and other (wrapped, liquid-staking or yield tokens). The tables show the token under the card's name. Evidence follows the same rules as availability: the provider's own docs, terms or code, or an onchain read; a third-party listing alone is not enough.

The settlement token is context, not a criterion. It does not change any rating. Where a fiat-backed stablecoin is used, the issuer's freeze power is already part of Censorship Resistance, and the page links there.

Why: the classes carry very different risks: issuer freeze and redemption for fiat-backed stablecoins, collateral and governance for crypto-backed ones, depeg under stress for synthetic ones, price movement for native tokens, and dependence on the provider for platform tokens. Folding them into the rubric would weigh those risks for the reader; showing them lets each reader weigh them.

2.10d Security audit log

Each provider's Security section lists every published audit, formal verification, audit contest, security review or penetration test of the code in its payment path: auditor, date, scope, audited commit where stated, the report link, and whether the audited code is what is deployed today. The log also says where it was compiled from, so a reader can judge how complete it is. A provider's statement that it was audited, without a published report, is noted there but not listed.

The log is context. The Audits and bug bounty criterion scores audits, and it rewards an audit only when the audited commit matches deployed code and findings were fixed traceably. A long list of audits of older code does not raise any rating.

2.10b Operational and procurement layer

A second layer records what an institution's procurement, finance and security functions ask next: fees, the custody model and licences, cash-out options per country, dated maturity facts, transacted volumes, and the provider's position against a set of vendor due-diligence controls derived from a UN humanitarian agency's software practices and Authority-to-Operate requirements. It is shown in its own section on every provider page and on the Operational page, and it never enters a CROPS pillar or any combined number.

Every item is a fact in one of five states: evidenced (a document or chain state anyone can re-check), attested (the provider states that a document such as a SOC 2 report exists, but it is available only under NDA), absent (evidence that it does not exist), not applicable (there is no operator to hold the control), and unknown. Volumes are shown only when computed from public chain logs by a query the repository can re-run; everything else is "not observable" or "not verifiable". Engineering-practice controls are read automatically from public repositories by scripts/repo-facts.mjs.

Why: a rail can hold every CROPS property and still be unusable for an institution that must know the fee, the regulated entity and how a payee reaches local currency. Keeping those facts out of the scores avoids rewarding reach or size, which are commercial properties. The five states exist because "available under NDA" is neither unknown nor evidenced, and because most vendor controls assume an operator: for an operator-free rail they are not failed, they fall to whoever runs the client, and the layer says so rather than showing a blank.

2.11 Criterion dependencies

Some criteria share causes. A required closed relayer fails direct settlement (C2), hosted dependence (C3), open-core dependence (F3), full stack public (O2), self-hosting (O3) and the walkaway test (S4) at once. The walkaway test is derived from custody, direct settlement, hosted dependence and self-hosting, and is marked as derived. Count a shared cause once when comparing providers.

2.12 Known limits

• The weekly job can re-read 1286 of 1492 rating evidence items (86%): onchain reads with a recorded check, pinned repository files, and documents with a quote. Traces, onchain items without a recorded check, and non-EVM state are not re-read; their checked date is shown on every rating.
• Pillars have different numbers of criteria (Censorship Resistance 5, Open 4, Free 3, Privacy 5, Security 6). A pillar with more criteria has more chances to hit its minimum, so compare a pillar across providers, not pillars against each other.
• Anchors still require judgment in borderline cases. Findings say "Judgment call" where one was made.
• Non-EVM chains (currently Solana) are supported for evidence, but the weekly job does not re-read their state and CI does not confirm that cited accounts exist. Solana programs are recorded as verified only through a reproducible-build registry.
• Where a provider publishes no contract addresses, contracts may be attributed from onchain activity (token flows, shared owners). The provider file says so in its basis; such attributions are judgment calls and are re-done if the provider publishes addresses.
• Many card products run on a shared card-issuing platform (for example Rain). Shared platform contracts that cannot be tied to one provider are not attributed to it, which is why several cards appear in the incumbent baseline.
• 1150 of 1150 ratings are awaiting human review; each rating shows its status.