OpenC3 COSMOS Core 7.5.1 - Data Flow Metrics and Performance!

COSMOS now measures how data moves through the system and tells you when processing can't keep up. New data flow metrics (#3903) track throughput and lag across interfaces and microservices. Users get a notification when a microservice falls behind (#3952).

We made some big improvements in how we parse the COSMOS command and telemetry definitions that can reduce the memory by 50% for large command and telemetry definitions containing descriptions.

Setting the new OPENC3_LOCAL_ONLY_TARGETS setting keeps the listed targets local and automatically syncs them with COSMOS. This opens up new workflows using git branching.

Migration Notes

With the addition of the compose.override.yaml and .env.local (first in 7.3.0) you should move all of your compose.yaml and .env updates into these new files. Rename your existing compose.yaml to compose.override.yaml and .env to .env.local. Get the newly released compose.yaml and .env files from the release. Then diff with these files to determine your project specific modifications and remove any defaults or other settings that you're not explicitly overriding. This will make upgrading much easier going forward because you don't have to sync with our defaults.

🛑 Breaking Changes from 6.x

S3 Backend Migration: Migrated from MINIO to versitygw.
- See the Migration Guide for details.

See the COSMOS 7.0.0 release notes for all the amazing new features in COSMOS 7!

🚀 Performance and scalability

This release reduces memory use and keeps COSMOS responsive under heavy load:

Threaded socket reads: interfaces read sockets on their own thread so data doesn't back up behind processing (#3902)
Queued Redis writes by default: interfaces now batch Redis writes every 0.1s, which greatly reduces Redis load at high packet rates (#4015)
Lower memory use: smaller CmdTlm memory footprint (#4027), and MALLOC_ARENA_MAX=2 in the Ruby base images (#3945)
Large targets: Fix Command History with targets containing thousands of commands (#4128)
Streaming: realtime streaming skips ahead when it falls behind instead of lagging further (#3949), and message logs are read incrementally (#3946)
Bounded Redis streams: an XADD MINID safety trim caps how old target streams can get (#3951), and read_topics XREAD counts are bounded by default (#3950)

🛠 Enhancements

Case-insensitive state names in check, wait and wait_check (#4159): 'Connected' now matches the state CONNECTED, and an unquoted state name such as == Connected works
Faster limits group enable/disable (#4112): a 70k-item group now takes about 1.5s instead of about 45s
Ruby Secrets.get_client and Bucket.get_client (#4184): the snake_case names now work in Ruby as well as Python
Astro Clock: a new Astro clock driven by a telemetry item (#4145)
Packet Viewer: fuzzy search that tolerates typos (#3941)
Buckets: non-admin users can write to user-defined buckets (#4092)
Bucket Explorer: volume downloads are streamed, so files over 2GB work (#4074)
Script Runner: open_bucket_dialog now returns the selected bucket and path (#4044)
CLI: new util mirror command copies multi-arch images between registries (#3936)
Demo: new MEMDUMP memory dump example with CRC checks and recovery (#4131)

🔒 Security

Cross-scope deletes are blocked in delete_modified (#3933)
Table config overlays are now gated (#3935)
Plugin upgrades protect modified files and report diff failures (#3969)
When an item is obfuscated, any values given for it are dropped (#3998)
Regexes with reliability concerns cleaned up (#3915)
Bundled undici bumped to 6.29.1 for CVE-2026-109869 (#4146)
Traefik 3.7.15 (#4058, #4093), Versitygw 1.9.0 (#4060), anycable-go rebuilt with Go 1.26.9 (#4057), certifi 2026.10.19 (#4134), plus general dependency updates (#3934, #4071, #4130) including updating our cacert.pem files

📚 Documentation

New guides: AWS EC2, AWS EKS and Network Configuration (#4150)
CCSDS protocols plugin (#4023), Autonomic trigger operators and evaluation under load (#4108), Script Runner version history (#4068, #4069)
DB_SHARD and DONT_LOG interface options documented, router modifiers fixed (#3996)
Secret file directory mounts and initsettings overwrite rules (#3926), cross-packet reads in conversions (#4024), migration script init image (#4078), AstroUX icon link (#3955)

🪲 Bug Fixes

Tools

Command Sender: Ignore Range Checks is treated as hazardous, hazardous highlighting is restored (#4059), and the hazardous warning reliably appears (#4135)
Table Manager: edits are kept when using Save As (#3960)

Interfaces and microservices

Interface reconnect and shutdown hardened (#4001)
A stale Python periodic command thread is stopped on reconnect (#3994)
Commands are rejected when a write protocol requests a disconnect (#4004)
Only one disconnect marker is queued per connection (#4011)
Malformed connect parameters are acknowledged instead of crashing the microservice (#4006)
Malformed and released commands are acknowledged, and flag defaults are aligned between Ruby and Python (#4002)
Pre-identified telemetry is validated and stored data is kept out of the CVT (#4000). Pre-identified telemetry for unmapped targets is dropped (#4080)
Routed commands are identified against the routed target (#4003)
Connection error backtraces are logged once per message (#3999)
Fixed a bug in dynamic packet content generation (#4010)

Plugins, config and build

MAP_TARGET with an invalid enabled state now raises a config error (#3992)
Python dependency install is skipped on a modified_diff dry run (#3963)
A trailing slash is dropped from pypi_url before /simple is appended (#4081)
uv is relocked against PYPI_URL so air-gapped builds resolve (#3891)
Operator process metrics fixed (#3959)
Fix issues with Python packet config (#4154, #4163)

Streaming, topics and logging

Cable subscriptions whose disconnect never arrives are now cleaned up (#3948)
Stream safety trim config and lookup failures are handled (#4075)
Log writer Redis streams are trimmed on the target's db_shard (#3953)
Python writes the command and decom stored flag as lowercase (#4005)
Python raw logging starts for LOG_STREAM with no parameters (#3995)
Multi-shard (Enterprise DB_SHARD) routing (#4106, #4109, #4166)

Scripting and API

check, wait and wait_check now parse comparisons the same way (#3832)
goto from the Script Runner debug prompt no longer deadlocks (#4049)
Subscribe replay size is capped under gRPC's 4MB limit (#4168)
Cancelling open_bucket_dialog now works in Python (#4175, #4176)
get_limits with a LATEST packet uses the scope argument (#4169)
Interface and router state defaults to UNKNOWN when status is missing (#4070)
Python treats a green low limit of zero as a defined limit (#4025)
Python decodes Ruby json_class Float NaN and Infinity values (#3997)
Fix LATEST telemetry handling to be consistent throughout (#4052)
Python Secrets.getClient() was removed in 7.0 (#4184): this broke 6.x plugins on 7.0 through 7.4. It's back as a deprecated alias.

What's Changed

Fix notifications overlay escape race by @jmthomas in #3899
fix(script): align check, wait and wait_check comparison parsing by @jmthomas in #3832
docs: clarify secret file dir mounts and initsettings overwrite rules by @jmthomas in #3926
fix(targets): block cross-scope deletes in delete_modified by @jmthomas in #3933
Pnpm dependencies by @EmilyRagan in #3934
clean up regex with reliability concerns by @calmonroe in #3915
feat: add util mirror command for multi-arch registry copies by @jmthomas in #3936
test(python): wait for thread state in tcpip server interface tests by @EmilyRagan in #3940
Fix/table config overlay gate by @jmthomas in #3935
Add AI code review by @ryanmelt in #3943
Add Manual Malicious Code Check and Decouple AI Review by @ryanmelt in #3956
ci(ai-review): grant checks: read so reviewers see SonarQube findings by @ryanmelt in #3958
Fix icon reference link to AstroUX icons by @ryanmelt in #3955
fix(plugins): skip python dependency install on modified_diff dry run by @EmilyRagan in #3963
AI review fixes for #3957 by @openc3qa in #3961
fix(plugins): protect modified files on upgrade and surface diff failures by @EmilyRagan in #3969
feat(bridge): stop host interfaces when openc3-app goes away by @ryanmelt in #4021
Fix a bug in dynamic packet content generation by @jalexanderqed in #4010
fix: stop late cable performs and empty coverage adds by @jmthomas in #3987
perf(tlm-viewer): skip screen updates for unchanged values by @jmthomas in #3957
Image build cache improvements by @EmilyRagan in #3971
Update workflows the docker auth to handle rate limits by @ryanmelt in #4031
AI review fixes for #3998 by @openc3qa in #4032
fix(interfaces): address read queue review feedback by @EmilyRagan in #3965
AI review fixes for #4015 by @openc3qa in #4030
Add limitation on deriving a competing product by @ryanmelt in #3991
AI review fixes for #3902 by @openc3qa in #4034
OPENC3_LOCAL_ONLY_TARGETS by @ryanmelt in #3942
AI review fixes for #4027 by @openc3qa in #4035
AI review fixes for #3948 by @openc3qa in #4036
AI review fixes for #3950 by @openc3qa in #4037
refactor(uv): replace deprecated uv index options and seed a plugin pyproject template by @mcosgriff in #3896
fix(table manager): edits of a table are saved when using Save As by @calmonroe in #3960
Fix traefik Dockerfile detection of image type by @jalexanderqed in #3990
AI review fixes for #3949 by @openc3qa in #4050
AI review fixes for #3951 by @openc3qa in #4051
Add standard Plugin Workflows by @ryanmelt in #3860
Cleanup test files by @EmilyRagan in #4018
build(openc3-ruby): set MALLOC_ARENA_MAX=2 in ruby base images by @ryanmelt in #3945
Reduce cmd tlm memory by @jmthomas in #4027
AI review fixes for #3959 by @openc3qa in #4056
feat(packet viewer): implement fuzzy search with typo tolerance by @calmonroe in #3941
fix(models): raise config error for invalid MAP_TARGET enabled state by @ryanmelt in #3992
Bump traefik to 3.7.14 by @jmthomas in #4058
Add license checks and heritage scripts by @ryanmelt in #3944
fix(microservices): harden interface reconnect and shutdown by @ryanmelt in #4001
fix(interfaces): stop stale python periodic cmd thread on reconnect by @ryanmelt in #3994
docs(sr): Remove references to stale prop (from COSMOS 4) by @clayandgen in #4069
fix(topics): identify routed commands against the routed target by @ryanmelt in #4003
Bump versitygw to 1.9.0 by @jmthomas in #4060
build(deps): bump the github-actions group across 1 directory with 9 updates by @dependabot[bot] in #4073
build(deps): bump the npm group across 2 directories with 13 updates by @dependabot[bot] in #4071
Run the malicious code scan as part of AI Review by @ryanmelt in #4077
docs(sr): Include Version History docs by @clayandgen in #4068
fix(cmd-tlm-api): measure streaming lag against the stream head by @jmthomas in #4076
fix(python): treat zero green low limit as a defined limit by @jmthomas in #4025
docs(conversions): use System.telemetry.packet for cross-packet reads by @jmthomas in #4024
fix(admin): wait for version history info before modified plugin dialog by @jmthomas in #3988
fix(models): start python raw logging for LOG_STREAM with no params by @ryanmelt in #3995
fix(cmd-tlm-api): read message logs incrementally by @ryanmelt in #3946
build(openc3-ruby): update anycable-go binaries to go1.26.9 by @jmthomas in #4057
AI review fixes for #3952 by @openc3qa in #4064
fix(packets): drop given values of obfuscated items on obfuscate by @ryanmelt in #3998
fix(python): decode ruby json_class Float NaN and Infinity values by @ryanmelt in #3997
Fix/operator process metrics by @jmthomas in #3959
Align plugin_list implementations by @EmilyRagan in #4067
Add note about which init image is needed for the migration script by @ryan-pratt in #4078
fix(build): relock uv against PYPI_URL so air-gapped builds resolve by @mcosgriff in #3891
fix(plugins): drop a trailing slash from pypi_url before adding /simple by @jmthomas in #4081
refactor(plugins): centralize default plugins list by @EmilyRagan in #4079
fix(docker): improve dockerignore list by @EmilyRagan in #3986
fix(cmd-tlm-api): wait for streaming threads to exit in kill by @ryanmelt in #3947
fix(interfaces): reject commands when a write protocol requests disconnect by @ryanmelt in #4004
fix(microservices): log connection error backtraces once per message by @ryanmelt in #3999
fix(topics): harden stream safety trim config and lookup failures by @jmthomas in #4075
fix(command sender): treat Ignore Range Checks as hazardous and restore hazardous highlighting by @calmonroe in #4059
refactor(config): add default to handle_true_false for missing values by @jmthomas in #4082
fix(script runner): stop goto from the debug prompt deadlocking by @jmthomas in #4049
docs(interfaces): document DB_SHARD and DONT_LOG and fix router modifiers by @ryanmelt in #3996
Bump traefik to 3.7.15 by @jmthomas in #4093
build(anycable): copy anycable-go by TARGETARCH before API source by @jmthomas in #4094
fix(topics): write python command and decom stored flag as lowercase by @ryanmelt in #4005
fix(store): bound read_topics XREAD count by default by @ryanmelt in #3950
fix(api): default interface/router state to UNKNOWN when status is missing by @calmonroe in #4070
build(anycable): install anycable-go only in the API images by @EmilyRagan in #4083
fix(bucketexplorer): stream volume downloads to support files over 2GB by @jmthomas in #4074
feat(script-runner): expose bucket and path from open_bucket_dialog by @jmthomas in #4044
fix(vue-common): detect jpeg, gif, and bmp plugin images by @ryanmelt in #4091
docs(protocols): document ccsds protocols plugin by @jmthomas in #4023
fix(microservices): drop pre-identified tlm for unmapped targets by @jmthomas in #4080
ci(trivy): authenticate vex hub fetch and report scan errors separately by @jmthomas in #4104
feat(interfaces): queue redis writes by default at 0.1s interval by @jmthomas in #4015
build(deps): bump the github-actions group with 4 updates by @dependabot[bot] in #4127
fix(demo): cleanup plugin.txt by @EmilyRagan in #4119
ci(workflows): updates for required checks by @EmilyRagan in #4105
Update certifi to v2026.10.19 by @mcosgriff in #4134
fix(command sender): ensure hazardous warning appears by @calmonroe in #4135
test(playwright): fix fixme tests by @EmilyRagan in #4087
feat(cmd-tlm-api): allow non-admin writes to user-defined buckets by @jmthomas in #4092
Dependencies by @jmthomas in #4130
fix(migrate-s3): prompt when multiple old volumes match, rename cleanup by @jmthomas in #4113
fix(node): bump bundled undici to 6.29.1 for CVE-2026-109869 by @jmthomas in #4146
Update CCSDS protocols per new changes by @jmthomas in #4147
Scale COSMOS_ALL streaming to targets with thousands of packets by @jmthomas in #4128
List packet names without parsing every packet definition by @jmthomas in #4129
Thread socket reads to make sure sockets are serviced and don't fall behind by @ryanmelt in #3902
fix(python): restore time import in udp interface tests by @ryanmelt in #4152
feat(microservices): notify users when processing falls behind by @ryanmelt in #3952
fix(microservices): ack malformed connect params instead of crashing by @ryanmelt in #4006
docs(autonomic): describe trigger operators and evaluation under load by @jmthomas in #4108
test(playwright): run large volume download test as admin by @jmthomas in #4149
feat(topics): cap age of target streams with an XADD MINID safety trim by @ryanmelt in #3951
fix(microservices): validate pre-identified telemetry and keep stored data out of CVT by @ryanmelt in #4000
test(python): time template protocol initial delay from connect by @ryanmelt in #4155
fix(cmd-tlm-api): reap cable subscriptions whose disconnect never arrives by @ryanmelt in #3948
test(python): fix template protocol response race that could hang CI by @ryanmelt in #4153
fix(microservices): ack malformed and released commands, align flag defaults by @ryanmelt in #4002
feat(cmd-tlm-api): skip ahead when realtime streaming falls behind by @ryanmelt in #3949
feat(astro): add a telemetry item driven astro clock by @jmthomas in #4145
feat(demo): add MEMDUMP memory dump with CRC checks and recovery by @calmonroe in #4131
docs(agents): consolidate agent guidance into AGENTS.md by @jmthomas in #4151
fix(logs): trim log writer redis streams on the target's db_shard by @ryanmelt in #3953
fix(api): use scope argument for get_limits with LATEST packet by @jmthomas in #4169
fix(cmd-tlm-server): harden data flow health and match server lag levels by @jmthomas in #4162
fix(interfaces): only queue one disconnect marker per connection by @ryanmelt in #4011
fix(script-runner): cap subscribe replay size under gRPC 4MB limit by @jmthomas in #4168
docs: add AWS EC2, AWS EKS and Network Configuration guides by @jmthomas in #4150
fix(script-runner): review follow-ups for open_bucket_dialog cancel fix by @jmthomas in #4176
Data Flow Metrics by @ryanmelt in #3903
fix(script-runner): cancelling open_bucket_dialog in python by @EmilyRagan in #4175
fix(scripts): use core operator image for host setup by @jmthomas in #4177
test(db-shard): verify db_shard routing and fix routing bugs by @jmthomas in #4109
fix(python): write loadable class filenames in to_config by @jmthomas in #4163
fix(python): repair packet to_config, reset and as_json processor handling by @ryanmelt in #4154
fix(telemetry): handle LATEST across streaming, limits, writes and history by @jmthomas in #4052
fix(topics): use each target's db_shard when routing and trimming by @jmthomas in #4166
ci(docs): build artifacts for GH Pages deployment by @EmilyRagan in #4187
fix(tsdb): align cross-db_shard range lookups by packet time by @jmthomas in #4106
fix(limits): apply limits group enable/disable in one pass by @jmthomas in #4112
fix(secrets): restore python Secrets.getClient as deprecated alias by @jmthomas in #4184
feat(script): match state names case insensitively in check and wait by @calmonroe in #4159
chore(docs): stop committing built and generated docs by @jmthomas in #4188
build(deps): bump the github-actions group across 1 directory with 9 updates by @dependabot[bot] in #4181

Full Changelog: v7.4.1...v7.5.1

OpenC3/cosmos v7.5.1 on GitHub