Completely revised the duplicate management approach by eliminating the need to recreate the vector database each time.
Instead, duplicate pairs along with their similarities are now saved locally in a SQLite database.
This allows for efficient loading of the local database and enables the identification of similarities based on customizable minimum and maximum thresholds. Additionally, I've implemented improvements and bug fixes in the deletion functionality.