Image inputs to OpenAI, Anthropic, Google, and Ollama providers
You can now upload images to a new Media Node and query with image inputs:
This is currently only limited to OpenAI, Anthropic, Google, and Ollama. If you want more
provider support, consider making a PR. The relevant calls to LLMs are in utils.ts.
Image outputs to image inputs
Also enables generating images with one model and then passing them through prompt chaining to another, e.g.,
...Lots of other internal changes
This update required making lots of changes to how ChainForge stores media.
Images take up a lot of space in the browser. To handle this, ChainForge does not store the images in the browser (anymore). Instead we keep unique IDs to each image, derive from their SHA-256 hash, which we use to query the backend and pull the image only when it's necessary (e.g., viewing images in Table Views).
When running locally, ChainForge now copies media your upload into a media directory stored in the same place it puts Saved Flows.
As a consequence, ChainForge can now export and import cfzip bundles which are compressed .cforge (JSON) files alongside the media file(s) used in the flow.
(When running on the browser on chainforge.ai/play, ChainForge will continue to cache images in the front-end.)
These changes will make it easier to add data-heavy analysis pipelines, i.e. loading many documents and chunks into a RAG pipeline, in the near future.
If you encounter any problems with this release, please let us know.
Python 3.10+ are now the only versions supported by ChainForge.
We have deprecated support for 3.8 and 3.9 since the markitdown package required 3.10. This is a forward-thinking change since many other RAG-related packages now require ~3.10.