kucherenko/jscpd v5.3.3

Release v5.3.3

on GitHub

New Features

• Semantic clones, experimental: --semantic. The token passes match runs of tokens, so they miss two functions that do the same job with different code. Such a function may be renamed and restructured, or written in another language, like a validation rule that a Rust backend enforces and a Svelte frontend repeats (Type-4 clones). --semantic (config key semantic) embeds every function with a code embedding model and reports the pairs whose vectors point the same way as clones of kind semantic.
  • jscpd embeds the functions of JavaScript, TypeScript, JSX, TSX, Vue, Svelte, Astro, Python, Rust, Go, Java, Kotlin, C#, C, C++, PHP, Ruby, Scala and Swift files that clear --min-tokens and --min-lines. It embeds each function's code without comments, starting at the name the function is declared under.
  • Two functions pair only if they are in different files, neither calls the other, and the clones already found do not cover both. Each must be the other's closest match among the functions of its language; a function that is only close to the best match needs a higher similarity. The cosine must reach --semantic-threshold for a pair across languages, or --semantic-same-threshold for a pair within one language (0.4125 and 0.6375 with the default model), and it must stand at least 3 standard deviations above each function's background. --semantic-scope same keeps the pairs within one language, and --semantic-scope cross keeps the pairs across languages.
  • The default model, CodeRankEmbed (MIT), runs inside jscpd on the CPU once jscpd --semantic-download has fetched it (548 MB, pinned by revision and SHA-256), and a scan makes no network call. In a comparison of nine open models, it found more known clones than jina-embeddings-v2-base-code at the same precision, and reviewers judged more of its pairs to be duplicates. jscpd also runs jina-embeddings-v2-base-code, which is faster: jscpd --semantic-download jina-embeddings-v2-base-code fetches it (324 MB), and --semantic-model jina-embeddings-v2-base-code scans with it.
  • --semantic-url sends the functions to an OpenAI-compatible embeddings API instead, such as Ollama, llama.cpp, text-embeddings-inference or a hosted API. jscpd reads the API key only from JSCPD_SEMANTIC_API_KEY, and sends it only to a URL given on the command line or to a server on this machine. A config file can neither hold a key nor send code to another host on its own.
  • jscpd caches the vectors, so a repeat run embeds only the functions whose code changed. Each set of scanned paths has its own cache file. Once the vectors of changed and deleted functions make up more than a quarter of that file, the next run that embeds something rewrites it without them. --semantic-rebuild-cache embeds everything again and replaces the file at once.
  • Each model scores similarity on its own scale, so each model gets its own thresholds. jscpd --semantic-models lists the nine models jscpd has calibrated, with their thresholds, licenses and where they run. --semantic-model takes any of them by the name in that list (CodeRankEmbed), by Hugging Face id or by Ollama name, and the thresholds follow the model. The rule for groups of copies has two more settings, tuned with jina-embeddings-v2-base-code: a near-best margin of 0.05 and a floor of 0.8. jscpd scales both for every other model by the model's gap between its two thresholds, which gives 0.075 and 0.7125 for CodeRankEmbed. Qwen3-Embedding-0.6B and jina-code-embeddings-0.5b expect an instruction before the text, and jscpd puts the one they were calibrated with before every function; the config key prefix replaces it. A model that jscpd has not calibrated gets 0.6 and 0.75, and the run warns about it.
  • The console prints Clone found (rust, semantic ~0.78), -r ai prints [~0.78 semantic], JSON carries "kind": "semantic" and similarity, and SARIF uses the rule jscpd/semantic-code. --kind semantic keeps only these clones. The code lives in a new crate, cpd-semantic, and a run without --semantic works as before.
  • fixtures/semantic-demo is a runnable example: a Rust API and a SvelteKit frontend with 8 rules written on both sides and 2 features written twice in one language. (#1101, #1103, #1105, #1108, #1110)

Bug Fixes

• A scan path inside another scanned its files twice. With jscpd . src, or a config file that lists src next to src/generated, jscpd walked every file under the inner path once for each path and reported each of those files as a clone of itself. --semantic embedded their functions twice and paired each one with itself. Version 4 read those files once, and so does jscpd now, as it already did with --follow-symlinks. See fixtures/nested-paths-demo. (#1106)

• Svelte components lost two kinds of use to --dead-code. Svelte reads a store as $name, in the script and in the markup, and that is often the only use that import { page } from '$app/stores' gets, so --dead-code reported the import as unused with 100% confidence. It also reported a name as an unused symbol when the markup read it only inside the ${…} of a template literal, as in href={(p) => `/?${base}&page=${p}`}, because the markup scan skipped template literals the way it skips plain strings. $name now counts as a read of name (runes such as $state and $props, and $$props, do not), and jscpd reads the placeholders of a template literal as code in every component format. On two SvelteKit apps, sshx and the RealWorld example, the six and three findings of basta 0.3.0 were all of these two kinds, and neither app has a finding now. See fixtures/dead-code-demo. (#1102)

• Positions in files with Windows line endings drifted. The generic tokenizer, which reads Python, Go, Java, C# and most other formats, moved one byte forward per line where a CRLF line ends in two bytes, so every position was short by the number of lines above it. The position values in the JSON report were off, and an --ignore-pattern match removed tokens a byte behind per line, which changed the token counts of the clones around it. Files with CRLF endings now report true byte offsets, and where a pattern applies, their clones can count different tokens than before. Files with LF endings are unaffected. See fixtures/crlf-demo. (#1101)

Dependencies

• paste is gone from the build. The crate is unmaintained (RUSTSEC-2024-0436), and candle's matrix kernels, pulp and tokenizers still depend on it. The workspace now patches it with a small local crate that hands its one macro to pastey, its maintained successor, so cargo audit and cargo deny pass without an ignore for the advisory. The patch goes away once those crates release their switch to pastey. sha2 moved to 0.11, and every other dependency to its latest compatible release. (#1107, #1109)
• oxc moved to 0.151 and ruff to 0.0.15, each as one family of crates (#1098), and clap to 4.6.7 (#1097).

Deprecations

• --min-duplicated-lines never did anything, and now says so. Since the first 5.x release, the docs described it as a minimum percentage of duplication to report, but no code ever read it: a scan with --min-duplicated-lines 100 found the same clones as one without it. jscpd now hides the flag from --help and prints a warning when it is passed, and a later release will remove it. The flag is still accepted, so a script that passes it keeps working. To fail a run on too much duplication, use --threshold; to set the smallest clone worth reporting, use --min-lines or --min-tokens. (#1100)

Published Packages

• basta@0.3.0 on crates.io
• cpd-core@0.1.19 on crates.io
• cpd-finder@0.1.19 on crates.io
• cpd-reporter@0.1.20 on crates.io
• cpd-tokenizer@0.1.18 on crates.io
• cpd@5.3.3 on npm
• jscpd@5.3.3 on npm
• jscpd-darwin-arm64@5.3.3 on npm
• jscpd-darwin-x64@5.3.3 on npm
• jscpd-linux-x64-gnu@5.3.3 on npm
• jscpd-linux-arm64-gnu@5.3.3 on npm
• jscpd-linux-x64-musl@5.3.3 on npm
• jscpd-linux-arm64-musl@5.3.3 on npm
• jscpd-windows-x64-msvc@5.3.3 on npm
• jscpd-windows-arm64-msvc@5.3.3 on npm
• jscpd==5.3.3 on PyPI

Not Yet Published

• cpd-semantic@0.1.0 (published: none)
• jscpd@5.3.3 (published: 5.3.2)

Verify

Archives are signed with Sigstore (keyless, <asset>.sigstore.json)
and carry SLSA build provenance. Replace jscpd-linux-x64-gnu.tar.gz with your asset:

cosign verify-blob \
  --bundle jscpd-linux-x64-gnu.tar.gz.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/kucherenko/jscpd/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  jscpd-linux-x64-gnu.tar.gz
gh attestation verify jscpd-linux-x64-gnu.tar.gz --repo kucherenko/jscpd
sha256sum --check --ignore-missing checksums.txt