Summary
The 1.3.0 release of OpenKAT introduces new features and small fixes. The most interesting new elements are the way boefjes and normalizers are structured and the way settings are handled. Boefjes and normalizers are decoupled, to allow more matches between boefjes, normalizers and objects. Settings for boefjes have become available through the front end of OpenKAT. Also, OpenKAT has become easier to package and install.
IMPORTANT
Celery beat had been added to the Octopoes worker and the command used to start the Octopoes worker has been changed because of this. There needs to be one Octopoes worker that runs Celery beat to schedule tasks and there can also be only one worker with celery beat. For containers we added entrypoints that can handle any future changes, see https://github.com/minvws/nl-kat-coordination/blob/main/docs/containers.md#container-commands and https://github.com/minvws/nl-kat-coordination/blob/main/docker-compose.release-example.yml
Settings for boefjes
Building on the changes that were made in the previous release, from now on boefjes can have mandatory settings. For example, before enabling Shodan, you must provide the API key in the katalogus settings. This removes the need to place all boefje settings in a config file.
Plugin directory restructure
Boefjes and normalizers are now fully decoupled. This means that a plugin directory can now have as many subdirectories with boefjes and normalizers. For a boefjes or normalizer to be found, it has to have an __init__.py, a boefje.json or normalizer.json and a file with a run method. This sets a big step towards "package-ability" of plugins. On the downside, your previously created boefjes and normalizers need a minor update. The manual on the wiki has been adapted to reflect these changes and will help you make the transition.
Pagination of Octopoes
Before, when loading the objects list, all OOIs were returned. This created performance issues, and therefore we added API side pagination to octopoes.
Full Changelog
Coordination
What's Changed
- apply changes to octopoes worker by @Lisser
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- update yarn packages by @Rieven
Boefjes
What's Changed
- update instructions for running boefjes by @errieman
- Feature/json boefje definitions by @Donnype
- support octopoes SCM versioning in docker dev setup by @errieman
- Expose environment keys (mandatory fields) by @Donnype
- Feature/recursively find plugins by @Donnype
- Feature/module to path by @Donnype
- Discovery fix by @Donnype
- add correct port and address to cert boefje by @noamblitz
- Remove redundant API key notices from description by @noamblitz
- Create dependabot.yml by @sigio
- build octopoes in main image by @errieman
- fix(binaryedge): rename ooi_type to object_type by @Lisser
- bump octopoes to 2.6.2 by @noamblitz
- optimize katalogus images by @noamblitz
- Env vars globally by @sigio
- Bump sqlalchemy from 1.4.32 to 1.4.42 by @dependabot
- Bump fastapi from 0.75.0 to 0.85.1 by @dependabot
- Bump uvicorn from 0.17.5 to 0.19.0 by @dependabot
- Bump pytest from 6.2.5 to 7.2.0 by @dependabot
- revert id to name change by @noamblitz
- get api keys from katalogus api plugin settings by @noamblitz
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump black from 21.9b0 to 22.10.0 by @dependabot
- Bump psycopg2-binary from 2.9.3 to 2.9.5 by @dependabot
- Bump alembic from 1.7.7 to 1.8.1 by @dependabot
- Bump requests from 2.27.1 to 2.28.1 by @dependabot
- Fix requirements for requests and dispython by @ammar92
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution
Octopoes
What's Changed
- fix(retirejs): avoid indexerror by @Lisser
- Minor improvements and optimalisation for scanprofile recalculate by @dekkers
- Release 2.6.0 by @Lisser
- refactor(list_ooi): add count to result by @Lisser
- Create pull_request_template.md by @Lisser
- Automated update to workflows: use checkout@v3 by @sigio
- fix packaging requirement by @noamblitz
Mula
What's changed
- Merge pull request #130 from minvws/feat/monitor-org-timeout
- Access queues directly from schedulers dict
Bytes
What's Changed
- Bump mako from 1.2.0 to 1.2.2 by @dependabot
- Hotfix/rename name to by @Donnype
- Create dependabot.yml by @sigio
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution