github cardano-foundation/cf-token-metadata-registry 2.0.1
v2.0.1

pre-releaseone hour ago

cf-token-metadata-registry 2.0.1

2.0.1 is the 2.x release to upgrade to from 1.7.0. It includes everything in 2.0.0 (a pre-release) and the fixes found while it was tested, so these notes describe the whole change from 1.7.0 to 2.0.1. Coming from 2.0.0? See Upgrading from 2.0.0.

2.x changes what the registry serves and how it reports problems: a token that breaks its CIP is now served with its validation errors instead of being dropped, and two new kinds of token are served from the V2 API.

Important

  • API clients: two changes can break a client written for 1.7.0, see Breaking Changes. Everything else in the API is additive.
  • Operators who run the registry themselves: upgrading from 1.x needs a full resync from an empty database, see For operators.

Key Improvements

  • Validation errors are reported, not enforced. Every CIP-26 entry and CIP-68 datum is served with a validation_errors list. Clients choose what they get with ?validation=ALL|MANDATORY_ONLY|STRICT (V2 only) (#111, ADR-019).
  • CIP-68 rich fungible tokens (label 444) are served, with image as metadata.logo (#110, ADR-018).
  • CIP-113 programmable tokens are indexed from the official deployments. V2 adds type (NATIVE / PROGRAMMABLE) and an extensions.cip113 block (#68).
  • The OpenAPI spec at /apidocs now describes what the API sends. In 1.7.0 and 2.0.0 it did not allow the nulls the API sends, so V2 responses failed validation against it; now every response validates, and a new regression suite checks it on mainnet (#157, #158, #161).
  • The indexer is hardened against untrusted on-chain datums. A bad output is skipped with a WARN and can no longer stop the sync (#105, #107, #109).
  • Real-world CIP-68 datums are handled: texts split into lists of byte strings (593 fungible-token descriptions on mainnet), an empty name after the label prefix, RFTs whose 444 token was minted in another transaction (#117, #121, #131, #132).
  • yaci 0.4.7, which fixes blocks with 128 to 255 transactions being read as empty (mainnet block 14,053,493 onwards) (#108).

Breaking Changes

These reach every API client as soon as the API it uses is upgraded, including the API hosted by the Cardano Foundation.

  1. name and description can be null. 1.7.0 answered 404 for a token without a name and a description (and left it out of a batch); 2.x serves every indexed token with its validation_errors, so a served token is no longer guaranteed a name or a description. Its metadata.valid is false: valid is now true only for a token with no validation errors. V1 serves such CIP-26 entries too. For the closest to the 1.7.0 behaviour, ask V2 with ?validation=MANDATORY_ONLY (#111, ADR-019).
  2. The CIP-26 logo in the V2 metadata block is base64, no longer hex. A bug fix: CIP-26 defines the logo as base64, and standards.cip26.logo and V1 always returned it so; 1.7.0 hex-encoded it in metadata.logo only. A client that hex-decodes metadata.logo must decode base64 instead, or it loses every CIP-26 logo. CIP-68 logos are URIs and are not affected (#88).

API changes

All additive: a client written for 1.7.0 keeps working, apart from the two changes above.

New in V2:

What Where Values Default
validation query parameter GET /api/v2/subjects/{subject}, POST /api/v2/subjects/query ALL: every indexed token. MANDATORY_ONLY: tokens without a MANDATORY error. STRICT: tokens without any error from the checks the registry runs (not full CIP compliance, see ADR-019). A token left out is a 404, or missing from a batch ALL
validation_errors response field on the subject, standards.cip26 and standards.cip68 A list of {cip, sub_standard, field_name, field_category, code, error}; under ?property= limited to the properties asked for
type response field subject NATIVE or PROGRAMMABLE (CIP-113)
extensions.cip113 response field subject, for a token of a registered CIP-113 policy The minting, transfer, third-party and unfracking logic scripts and the global state policy of its registry node; any of them can be null
  • New tokens are served: CIP-68 rich fungible tokens (444), with image as metadata.logo (#110), and CIP-113 programmable tokens, also without CIP-26 or CIP-68 metadata (#68, #118).
  • ?property= no longer has to include name and description; 1.7.0 answered 400 without them.
  • OpenAPI spec (/apidocs): it now describes the nulls the API sends (nullable type lists, anyOf for nullable references), and the V2 batch has its own response schema, SubjectBatchResponse. Responses don't change; a client generated from the spec gets the corrected types when regenerated (#157, #158, #160, #165).
  • Fixed: a V2 batch with no subjects returns 200 with an empty list (was a 500), and {"subjects": null} is a 400 on V1 and V2 (was a 500) (#167). metadata.empty, a helper method serialized by mistake, is no longer in V2 responses (#166).

For operators

  • From 1.x: a full resync from an empty database is required. Tokens earlier versions dropped exist nowhere yet, old rows have no validation errors, and CIP-26 subjects are now stored in lower case with no migration (#111, #117, #118). Migrations V3 to V7 are additive, so an old database starts, but it does not hold what 2.0.1 serves. Plan for about 12 hours for a fresh mainnet sync from the shipped start slot to reach the chain tip.
  • Monitoring: health components are renamed: onchainConnection → nodeHealth, onchainReadiness → nodeSync, in /actuator/health and the startup, liveness and readiness groups (#93).
  • Configuration: CIP113_REGISTRY_NFT_POLICY_IDS (new, cip113.registry.nft.policy-ids) lists the registry-node NFT policy IDs of the CIP-113 deployments to index; CIP-113 is on when it is not empty. The shipped .env files set the official deployments (mainnet 484e733d122af44e6101988bcc47ed261a7af5c43c797e89d60e3075, preprod 3083d387537f9318b6ff5aeab7de5f5629d26495319b8fafd409222a); set it to empty to turn CIP-113 off. The bundled docker-compose.yml passes it to the read-only API (api-ro) as well (#68, #168).
  • Docker images: pin a version. A release tag moves latest (JVM) and latest-native on Docker Hub, so an instance on latest gets this release, and its resync, on its next pull.
    • API, JVM: cardanofoundation/cf-token-metadata-registry-api:2.0.1
    • API, GraalVM native, experimental and not supported yet: cardanofoundation/cf-token-metadata-registry-api:2.0.1-native (the same repository; -native is a suffix of the tag). The 2.0.1 native image does not start; see Known Issues. Use the JVM image.
    • CIP-26 sync job: cardanofoundation/cf-token-metadata-registry-job:2.0.1

Upgrading from 2.0.0

No resync is needed. Three things to know:

  • A token affected by #145 (a CIP-68 metadata change in the same block as its mint) keeps its old metadata until its next metadata change, or a resync.
  • The new invalid_constructor and wrong_type errors (#146) are written when a datum is indexed, so rows indexed by 2.0.0 get them only after a resync (3 rows on mainnet, among them BuyBTC).
  • API: metadata.empty is gone, and the OpenAPI spec is corrected, as described in API changes.

Known Issues

The GraalVM native image is experimental and not supported yet. Use the JVM image (cardanofoundation/cf-token-metadata-registry-api:2.0.1), which 2.0.1 was tested with. The native image has these known problems:

  • The 2.0.1 native image does not start. jOOQ's static initializer fails (ExceptionInInitializerError from org.jooq.impl.SQLDataType), so the API never comes up. It is a regression since 2.0.0; the suspected cause is the newer native-maven-plugin (#152) (#175).
  • It needs an x86-64-v3 CPU (AVX2, Haswell or newer), although it is built for any x86-64 CPU (#127): it is linked against objects that require x86-64-v3. On an older CPU, or a VM with a basic CPU model, it stops with CPU ISA level is lower than required. This affects 2.0.0 as well (#176).
  • It cannot run in read-only mode (#90). A separate read-only native image is in #113.

Features in detail
  • Validation errors are reported, not enforced (#111, ADR-019, ADR-020):
    • every CIP-26 entry and CIP-68 datum is indexed as written, with a validation_errors list (cip, sub_standard, field_name, field_category MANDATORY/OPTIONAL, code, error) on the subject and on standards.*;
    • metadata.valid is true only when there are no errors;
    • ?validation=ALL|MANDATORY_ONLY|STRICT (default ALL) chooses which tokens are returned; under ?property= the errors are limited to the properties asked for. STRICT means no error from the checks the registry runs, not full CIP compliance: a CIP-68 text given as a list of byte strings is accepted, a URI is checked for its scheme only, and a CIP-26 logo for base64 and size, not for being a PNG (#148);
    • what cannot be stored at all (an undecodable datum, an unusable CIP-26 file, an unparseable CIP-113 node, a logo over the hard size cap) is recorded in a new invalid_token table.
  • CIP-68 rich fungible tokens (label 444, prefix 001bc280) are served from the V2 API. Their image is served as metadata.logo, and description is optional for them, as CIP-68 says (#110, ADR-018).
  • CIP-113 programmable tokens (#68):
    • registry nodes are indexed from the official CIP-113 deployments when CIP113_REGISTRY_NFT_POLICY_IDS is set;
    • V2 adds type (NATIVE or PROGRAMMABLE) and an extensions.cip113 block with the minting, transfer, third-party and unfracking logic scripts;
    • a token of a registered policy without CIP-26 or CIP-68 metadata is served as PROGRAMMABLE with one MANDATORY companion_metadata error naming the missing fields (#118, #122).
Other bug fixes
  • CIP-26 subjects written in upper case (39 mainnet registry files): they are found whatever the case of the query, and served with the subject as the registry file writes it, so all their signatures verify (106 signatures; V1, and standards.cip26 in V2). Subjects are stored in lower case (#117, #144).
  • Indexing can no longer stop the sync. An output whose CIP-68 indexing fails unexpectedly is skipped with a WARN; the rest of the block is indexed (#109).
  • A CIP-68 metadata change in the same block as the mint is no longer missed: the "reference NFT already stored" check sees an earlier transaction of the same block (#145, ADR-023).
  • Hardening against untrusted on-chain input, ported from yaci-store assets-ext (#105, #107):
    • decimals range checks for CIP-26 and CIP-68;
    • CIP-68 version 4 nested-map datums, Plutus constructors as property values, text that is not UTF-8, several reference NFTs in one output;
    • a logo size cap and a limit on how deeply a datum can be nested.
  • CIP-68 rules are checked when a token is served, for the label asked for (333 or 444), instead of the label guessed when the datum was indexed. A 333 updated without its user token is no longer served as valid (#118).
  • New CIP-68 errors: a datum that is not constructor 0, which CIP-68 requires, gets a MANDATORY invalid_constructor error (on mainnet BuyBTC, a 333 token, was served as valid); an element of an image/logo list that is not a byte string is left out with a wrong_type error, where it was left out silently (none on mainnet) (#146).
  • Texts given as a list of byte strings (name, description, url, ...) are joined and served as the text, with no error, instead of being dropped (#117, #132). CIP-68 defines them as one bounded_bytes, which can be any length in a datum (the 64-byte limit per text is a rule of transaction metadata, CIP-25), but most minting tools still split a long text into a list: 593 fungible-token descriptions on mainnet. Each description problem is reported once, with the label's category (#117).
  • A CIP-68 token whose name after the label prefix is empty (<policy>0014df10 or <policy>001bc280, which the ledger allows) is served from its reference NFT <policy>000643b0. It was a 404 (PBG Token on mainnet) (#131).
  • What is stored:
    • a datum without a user token in its transaction is stored, so RFTs whose 444 token was minted in another transaction are served (#121);
    • a datum minted with a 222 NFT and no 333 or 444 token is not stored, so NFTs are not served as fungible tokens (#121);
    • a datum that is not token metadata (dApp state under the 000643b0 prefix, an empty map) is no longer indexed as a 333 (#117).
  • CIP-68 version: every problem with a version that is present but unusable (not an integer, too large, not 1 to 4) is one invalid_version code (#123).
  • CIP-26:
    • decimals is checked against 0 to 19, the range CIP-26 allows (none on mainnet is outside it; the highest is 18) (#147);
    • a url without a host (https:/example.com) fails validation (#117).
  • invalid_token: stale CIP-26 rows are deleted in a transaction. They failed for every file, with 7,982 WARNs per sync (#116).
  • Docker:
    • the image build fails on a trust store that still has the roots of the compromised eMudhra intermediate CAs (emSign ECC Root CA - G3, AAA Certificate Services, removed by Temurin 25.0.5+8), as a stale cached base image would have; the native image embeds Temurin's trust store, so both images trust the same roots (#151);
    • the API healthcheck probes the container port, so a custom host port (API_LOCAL_BIND_PORT) no longer leaves the container unhealthy forever. Contributed by @Kartiiyer12 (#96);
    • the database restarts after a host reboot. The API crash-looped without it (#116).
Dependencies
  • yaci-store 2.0.4 → 2.0.7, and the yaci library pinned on its own to 0.4.7. yaci 0.4.6 and earlier read a block with 128 to 255 transactions as empty; mainnet block 14,053,493 is the first one affected (#108).
  • springdoc-openapi 2.8.4 → 2.10.0 and swagger-core 2.2.28 → 2.2.55, the newest for Spring Boot 3.x (#158).
  • Security: Bouncy Castle 1.80 → 1.86, JGit 7.1.0 → 7.8.1 (#137, #152).
  • cardano-client-lib 0.7.2 → 0.7.3 (an upstream Byron address validation fix) (#129).
  • commons-cli 1.11.0, and the build tooling: Maven 3.9.16, maven-compiler-plugin 3.16.0, Surefire/Failsafe 3.6.1, JaCoCo 0.8.16, native-maven-plugin 0.11.5 (#152).
Chores, tests and documentation

Chores and maintenance

  • On-chain health uses yaci-store's own NodeHealthIndicator and NodeSyncIndicator, so each check is reported once instead of twice (#93).
  • Docker images are built on pull requests (#115).
  • The SonarCloud project is named cf-token-metadata-registry instead of server-base (#103).
  • The release-please workflow uses an access token, so a release triggers the image builds (3bd5217, 293b6f8, 0a8b15f).
  • The native code is compiled for any x86-64 CPU (-march=compatibility, #127; the executable still needs x86-64-v3, see #176), native images get their own tags, and the workflows use Earthbuild instead of Earthly (#128).
  • Dependabot opens security updates only (#142).

Tests

  • 2.0.1 was tested on mainnet with a sync from scratch (from the shipped start slot to the tip), then again as merged, JVM image: 4,946 regression tests passed, 0 failed (CIP-26 with signature verification, CIP-68 333 and 444, the validation contract, CIP-113, the V2 batch, and responses validated against /apidocs). On preprod, the CIP-113 suite passed for 301 tokens of 70 registered policies, including programmable tokens with CIP-26 or CIP-68 metadata (308 passed).
  • New regression suites: responses validated against the OpenAPI spec (-m openapi), CIP-26 signatures (-m signatures), batches without subjects (#144, #158, #167).
  • The mainnet test steps select every regression test, and fail on one neither selects; 24 tests (the V2 batch and CIP-113 suites) never ran before (#172).
  • Integration test for a CIP-113 policy without companion metadata (#119). The mainnet CIP-68 fixture is corrected entry by entry against the chain, not regenerated (#120). The regression suites write one Allure report per run (#126).

Documentation

  • ADR-017 (superseded by ADR-019), ADR-018 (rich fungible tokens), ADR-019 (validation errors are reported, not enforced), ADR-020 (invalid_token, CIP-113), ADR-022 (rejected: a 333/444 subject is served whether or not its user token was minted). ADR-004, ADR-010, ADR-016, ADR-019 and ADR-023 are amended. ADR-018 records that the name after the label prefix can be empty (#131, #132), and ADR-019 why a text given as a list of byte strings is accepted (#132, #133), the case of CIP-26 subjects, invalid_constructor, the CIP-26 decimals range and what STRICT means (#144, #146, #147, #148).

Full changelog: 1.7.0...2.0.1

Don't miss a new cf-token-metadata-registry release

NewReleases is sending notifications on new releases.