Change Log (v0.6.0)

Overview

As a major release, v0.6.0 is consisted of a number of large changes to enhance the performance, scalability, robustness and usability of Crawlab. This beta version is theoretically more robust than older versions mainly in task execution, files synchronization and node management, yet we still recommend users to thoroughly run tests with various samples.

Enhancements

Backend

• File Synchronization. Migrated file sync from MongoDB GridFS to SeaweedFS for better stability and robustness.
• Node Communication. Migrated node communication from Redis-based RPC to gRPC. Worker nodes indirectly interact with MongoDB by making gRPC calls to the master node.
• Task Queue. Migrated task queue from Redis list to MongoDB collection to allow more flexibility (e.g. priority queue).
• Logging. Migrated logging storage system to SeaweedFS to resolve performance issue in MongoDB.
• SDK Integration. Migrated results data ingestion from native SDK to task handler side.
• Task Related. Abstracted task related logics into Task Scheduler, Task Handler and Task Runners to increase decoupling and improve scalability and maintainability.
• Compotenization. Introduced DI (dependency injection) framework and componentized modules, services and sub-systems.
• Plugin Framework. Crawlab Plugin Framework (CPF) has been released. See more info [here](https://docs.crawlab.cn/en/guide/plugin/).
• Git Integration. Git integration is implemented as a built-in feature.
• Scrapy Integration. Scrapy integration is implemented as a plugin [spider-assistant](https://docs.crawlab.cn/en/guide/plugin/plugin-spider-assistant).
• Dependency Integration. Dependency integration is implemented as a plugin [dependency](https://docs.crawlab.cn/en/guide/plugin/plugin-dependency).
• Notifications. Notifications feature is implemented as a plugin [notification](https://docs.crawlab.cn/en/guide/plugin/plugin-notification).

Frontend

• Vue 3. Migrated to latest version of frontend framework Vue 3 to support more advanced features such as composition API and TypeScript.
• UI Framework. Built with Vue 3-based UI framework Element-Plus from Vue-Element-Admin, more flexibility and functionality.
• Advanced File Editor. Support more advanced file editor features including drag-and-drop copying/moving files, renaming, deleting, file editing, code highlight, nav tabs, etc.
• Customizable Table. Support more advanced built-in operations such as columns adjustment, batch operation, searching, filtering, sorting, etc.
• Nav Tabs. Support multiple nav tabs for viewing different pages.
• Batch Creation. Support batch creating objects including spiders, projects, schedules, etc.
• Detail Navigation. Sidebar navigation in detail pages.
• Enhanced Dashboard. More stats charts in home page dashboard.

Miscellaneous

• Documentation Site. Upgraded [documentation site](https://docs.crawlab.cn/en).
• Official Plugins. Allow users to install [official plugins](https://docs.crawlab.cn/en/guide/plugin/) on Crawlab web UI.