jina-ai/serve v3.20.0

💫 Release v3.20.0

on GitHub

Release Note (3.20.0)

Release time: 2023-08-04 08:58:38
This release contains 5 new features 3 bug fixes and 8 documentation improvements.

🆕 Features

Executor can work on single documents (#5991)

Executors no longer need to work solely on a DocList, but can expose endpoints for working on single documents.

For this, the method decorated by requests must take a doc argument and an annotation for the input and output types.

from jina import Executor, requests
from docarray import BaseDoc

class MyInputDocument(BaseDoc):
    num: int

class MyOutputDocument(BaseDoc):
    label: str

class MyExecutor(Executor):
    @requests(on='/hello')
    async def task(self, doc: MyInputDocument, **kwargs) -> MyOutputDocument:
        return MyOutputDocument(label='even' if doc.num % 2 == 0 else 'odd')

This keeps Executor code clean, especially for serving models that can't benefit from working on batches of documents at the same time.

Parameters can be described as Pydantic models (#6001)

An Executor's parameters argument can now be a Pydantic model rather than a plain Python dictionary. To use a Pydantic model, the parameters argument needs to have the model as a type annotation.

Defining parameters as a Pydantic model instead of a simple Dict has two main benefits:

• Validation and default values: You can get validation of the parameters that the Executor expected before the Executor may access any invalid key. You can also easily define defaults.
• Descriptive OpenAPI definition when using the HTTP protocol.

Expose richer OpenAPI when serving Executor with HTTP inside a Flow (#5992)

Executors served with Deployments and Flows can now provide a descriptive OpenAPI when using HTTP. The description, examples and other relevant fields are used in the Gateway to provide a complete API.

Support streaming in single-Executor Flows (#5988)

Streaming endpoints now also support the Flow orchestration layer and it is no longer mandatory to use just a Deployment.
A Flow orchestration can accept streaming endpoints under both the gRPC and HTTP protocols.

with Flow(protocol=protocol, port=port, cors=True).add(
    uses=StreamingExecutor,
):
    client = Client(port=port, protocol=protocol, asyncio=True)
    i = 10
    async for doc in client.stream_doc(
            on='/hello',
            inputs=MyDocument(text='hello world', number=i),
            return_type=MyDocument,
    ):
        print(doc)

Streaming endpoints with gRPC protocol (#5921)

After adding SSE support to allow streaming documents one by one for the HTTP protocol, we added the same functionality for the gRPC protocol. A Jina server can now stream single Documents to a client, one at a time, using gRPC. This feature relies on streaming gRPC endpoints under the hood.

One typical use-case of this feature is streaming tokens from a Large Language Model. For instance, check our how-to on streaming LLM tokens.

from jina import Executor, requests, Deployment
from docarray import BaseDoc

# first define schemas
class MyDocument(BaseDoc):
    text: str

# then define the Executor
class MyExecutor(Executor):

    @requests(on='/hello')
    async def task(self, doc: MyDocument, **kwargs) -> MyDocument:
        for i in range(100):
            yield MyDocument(text=f'hello world {i}')
            
with Deployment(
    uses=MyExecutor,
    port=12345,
    protocol='grpc', # or 'http'
) as dep:
    dep.block()

From the client side, you can use the new stream_doc() method to receive documents one by one:

from jina import Client, Document

client = Client(port=12345, protocol='grpc', asyncio=True)
async for doc in client.stream_doc(
    on='/hello', inputs=MyDocument(text='hello world'), return_type=MyDocument
):
    print(doc.text)

Read more in the docs.

🐞 Bug Fixes

Fix caching models from all endpoints, inputs and outputs (#6005)

An issue was fixed that caused problems when using an Executor inside a Flow where the same document type was used as input and output in different endpoints.

Use 127.0.0.1 as local ctrl address (#6004)

The orchestration layer will use 127.0.0.1 to send health checks to Executors and Gateways when working locally. It previously used 0.0.0.0 as the default host and this caused issues in some configurations.

Ignore warnings from Google (#5968)

Warnings that used to appear in relation to the pkg_resources deprecated API are now suppressed.

📗 Documentation Improvements

• Fix errors in getting started and preliminaries (#6008)
• Add note about Flow with one Executor supported (#5990)
• Add docs for secrets and jobs (#5948)
• Remove include gateway arg (#5987)
• Add Kubernetes hint to port ignore (#5985)
• Add docs for streaming (#5980)
• Add docs for jcloud horizontal pod autoscale (#5957)
• Update docs to show single document serving (#6009)

🤟 Contributors

We would like to thank all contributors to this release:

• Joan Fontanals (@JoanFM)
• Nikolas Pitsillos (@npitsillos)
• Alex Cureton-Griffiths (@alexcg1)
• Deepankar Mahapatro (@deepankarm)
• Winston Wong (@winstonww)
• AlaeddineAbdessalem (@alaeddine-13)
• Zhaofeng Miao (@mapleeit)