JerBouma/FinanceDatabase 2.5.1

FinanceDatabase 2.5.1

on GitHub

2.5.0 introduced the MCP server and a cleaned-up database. 2.5.1 is about keeping that database current without manual work, and about making the MCP server's answers trustworthy. Sampling hundreds of real searches on the hosted server showed where results went wrong: depositary receipts or foreign regional lines shown as a company's main listing, funds and bonds filed as equities, stale data, and accent or spelling variants that found nothing. Each fix below is a general, data-derived rule rather than a list of exceptions, so it keeps working as the data changes every week.

Funds and money markets are retired

Funds and money markets are no longer maintained. The decision followed a scan of how current they were:

• Badly out of date. Compared with Nasdaq's list of US funds (tickers A to I):

  • 42% of the US mutual funds in the database are no longer listed: they were merged, liquidated or matured, often unit trusts with a fixed end date;
  • 47% of the funds listed today are missing.

  Money markets looked the same: 40% are no longer listed, and only 352 of the 970 money market funds listed in that range are present.

• No source to keep them current. Mutual and money market funds aren't traded on an exchange, so there is no public, official directory that lists them worldwide, unlike stocks and ETFs. The roughly 27,000 non-US funds (Morningstar IDs; funds on Frankfurt, Stuttgart, Madrid, Vienna and Mexico) have no public source at all. Keeping them current would mean scraping fund portals or relying on paid data, which this project doesn't do.

• No way to mark closures. Funds have no ISIN or delisted column, so a closed or merged fund can't even be flagged; it just stays in the data as if it still existed.

• Little use. In the MCP server's usage statistics, funds were among the least called asset classes.

An outdated answer is worse than no answer, especially for an AI assistant that presents it confidently, so the classes were retired rather than left to decay further:

• Python package: fd.Funds() and fd.MoneyMarkets() still load their data unchanged, but warn when loaded (FutureWarning) that it's no longer maintained. Existing code keeps working.
• MCP server: the funds and moneymarkets tools are removed, and both classes are left out of search_instruments, show_options and search_categories, so an assistant never returns them.
• Exchange-listed funds move to ETFs. Funds that trade on a main stock exchange under an ordinary ticker do have a public source and stay maintained: the Chinese LOFs, the listed funds on Euronext Amsterdam (Kempen, Actiam, ASN), Copenhagen, Milan, Jakarta and a few others. A fund qualifies when its symbol has the shape of that exchange's real tickers. Spanish SICAVs therefore stay retired: their symbols are CNMV registry numbers, and most left the exchange after Spain's 2022 tax change.

The database's own equity classification (breaking)

Equities are now classified in the database's own three levels, listed in scripts/categories.json.

• Sectors: the 11 names are unchanged.
• Industry groups and industries: they have new names, for example:
  • Media, Entertainment & Internet;
  • Semiconductors & Chip Equipment;
  • Payments & Specialty Finance;
  • Consumer Lending;
  • Rail & Road Transport.
• New structure:
  • REITs are split by property type.
  • Real estate companies that aren't REITs have their own industry.
  • Savings banks stay Banks, and other mortgage lenders become Mortgage Finance.

This is a hard switch: filters on an old industry group or industry name return nothing. fd.show_options("equities") lists the new names.

MCP server

• Results by relevance.
  • Exact symbol and name matches come first, followed by the same company's listing on its largest exchange (ASML.AS, then ASML on NASDAQ).
  • Asset classes take turns, so a search such as "bitcoin" shows the coin, ETFs, companies and indices together.
  • A query without spaces matches whole words: loreal finds L'Oréal, but toyota no longer finds Toyo Tanso.
• Primary listings worked out from the data. only_primary_listing keeps a company's real home listing. It leaves out:
  • euro lines of US companies on German regional exchanges;
  • Canadian depositary receipts (CDRs) and similar local receipts;
  • mandatory convertible preferreds;
  • duplicate share classes (one of BRK-A and BRK-B).
• Forgiving filters.
  • Filter values ignore accents (curacao → Curaçao).
  • A synonym list maps common alternatives to the database's values: Côte d'Ivoire → Ivory Coast, USA, UK, Holland, Korea, Czechia; sectors such as Tech or Healthcare; and tiers such as megacap.
• search_instruments adds a delisted column when include_delisted is on.
• Unique users in /stats. A user is a keyed hash (HMAC) of the address and user agent, and neither is ever stored raw. PRIVACY.md describes this.
• Reliability.
  • Computing listing ranks for funds and indices no longer needs several gigabytes of memory; it used to restart a small server.
  • Ties are broken the same way every time, so results don't change between restarts.
  • Under a test load of 20 simultaneous users, the hosted server answers every call within seconds.
• Server name: the server is named "Finance Database".
• Primary listings across borders.
  • A venue counts by the listings of its own country, so Airbus is AIR.PA rather than a German regional line.
  • Brazilian depositary receipts (BDRs) are never a primary listing.
  • Abroad, the line with the share's ISIN wins (Glencore stays GLEN.L).
• Clearer answers.
  • Asking for funds or money markets explains that they are retired and still available in the Python package.
  • A currency's name finds its pairs ('japanese yen').

Package

• fd.Funds() and fd.MoneyMarkets() warn that they're no longer maintained (see above).
• The primary-listing logic (listings_model.py) gains the rules described under the MCP server. ETFs and indices, which have no country, keep their primary listings.

Database

• ETFs.

  • European ETFs from Xetra, Euronext and SIX were added, among them many 2025–2026 launches and new lines of popular funds such as VWCE.AS and SPYL.AS. These exchanges publish only abbreviated names, so full names come from the same fund's other listings or its Börse Frankfurt / Euronext page.
  • Tickers that were replaced are marked delisted (e.g. Xtrackers' old DBX* tickers on Xetra).
  • US ETFs that couldn't be added because an old company held their ticker were moved over (e.g. the Sprott physical trusts).
  • Blank categories are filled from the fund's other listings, or from name words that clearly decide a category ("treasury" → Fixed Income / Treasury Bonds). Issuer names decide nothing, and a category always comes with its group. The rule is right about 94% of the time on ETFs with known categories.

• Cryptocurrencies. Coins in CoinGecko's top 500 were added (such as Sui, Shiba Inu, NEAR, Pepe), and coins that no longer exist were removed. Numbered duplicates of a coin (SOL1, DOT1, UNI3) were removed.

• Currencies. The list now follows ISO 4217: new currencies (SLE, ZWG, VED, XCG) were added, and withdrawn ones (HRK and BGN after the euro, SLL, ANG, CUC) removed.

• Indices.

  • The indices Euronext publishes were added, plus the missing headlines MSCI World and the SSE Composite.
  • ETF iNAVs take the categories of the index they track.

• Equities.

  • Bonds, notes, warrants, certificates and other papers that aren't company shares were removed, and are kept out from now on.
  • Lines of securities whose home listing is delisted were marked delisted (e.g. the old Royal Dutch Shell A/B lines).
  • Chinese funds were moved out of equities.
  • ISINs were filled from CUSIPs and from symbols that are ISINs.
  • Names and countries were corrected (Aegon Ltd., Curaçao instead of the former Netherlands Antilles).

• Classification and countries reviewed.

  • Every large and mega cap company's classification was checked against its business description, with 347 corrections (e.g. Philips → Medical Devices & Supplies, Visa and Mastercard → Payments & Specialty Finance).
  • Smaller companies without an industry, or whose listings disagreed, were classified the same way (3,233 newly classified, 706 conflicts resolved).
  • Wrong and blank countries were corrected where the company's description and its exchange or ISIN agree, e.g. Malaysian companies marked Japan.

• Listing status.

  • US equities were checked against the Nasdaq Trader directory: 803 rows were delisted and 15 relisted.
  • Lines abroad of shares delisted at home are delisted too, e.g. Electronic Arts and Twitter in Germany.
  • Ferrovial SE's Euronext Amsterdam listing was added.

• ETFs.

  • 16,786 mutual funds were removed from the German regional ETF files.
  • 4,430 regional ETF lines took the ISIN, full name and categories of their Xetra line.

• Non-shares. More instruments that aren't company shares are recognised and removed: debentures, Nordic knock-out certificates, subscription warrants, rights issues and listed indices.

Weekly update

• What it maintains. The weekly update now maintains every live asset class, not just US equities. New sources sit next to the existing exchange directories (Nasdaq Trader, HKEX, NSE, JPX, ASX, TSX):
  • Xetra, Euronext and SIX for ETFs;
  • CoinGecko for cryptocurrencies;
  • ISO 4217 for currencies;
  • Euronext for indices.
• US listing status. The weekly update now compares the US equities with the Nasdaq Trader directory and keeps their delisted flag current, with a guard against implausibly many delistings.
• Where the code lives. database_update.yml holds no Python anymore. scripts/ follows the package's MVC layout, and scripts/README.md documents every rule.
• Robustness, so an automated run can't damage the database:
  • downloads are retried with backoff;
  • each source runs on its own, so a failure skips only that source;
  • files are only rewritten when they change, through a temporary file;
  • implausible changes are skipped (for example, a large share of market cap tiers changing at once);
  • scripts/check_database.py compares each update with the last commit and undoes it before the commit if it would introduce duplicate symbols, symbols in two asset classes, changed columns or a large loss of rows.

Documentation

• The README's Questions & Answers describe what the weekly update maintains and from which public sources, and why funds and money markets were retired.
• The README's MCP section follows the Finance Toolkit's format and links to the MCP page for the tools, examples and self-hosting.