jina-ai/serve v3.20.1

💫 Patch v3.20.1

on GitHub

Release Note (3.20.1)

Release time: 2023-08-10 15:49:12

This release contains 2 bug fixes and 2 documentation improvements.

🐞 Bug Fixes

Make Gateway load balancer stream results (#6024)

Streaming endpoints in Executors can be deployed behind a Gateway (when using include_gateway=True in Deployment).

In this case, the Gateway acts as a load balancer. However, prior to this release, when the HTTP protocol is used, the Gateway would wait until all chunks of the responses had been streamed from the Executor.

Only when all the chunks were received would it send them back to the client. This resulted in delays and suppressed the desired behavior of a streaming endpoint (namely, displaying tokens streamed from an LLM with a typewriter effect).
This release fixes this issue by making the Gateway stream chunks of responses from the forwarded request as soon as they are received from the Executor.

No matter whether you are setting include_gateway to True or False in Deployment, streaming endpoints should give the same desired behavior.

Fix deeply nested Schemas support in Executors and Flows(#6021)

When a deeply nested schema (DocArray schema with 2+ levels of nesting) was specified as an input or output of an Executor endpoint, and the Executor was deployed in a Flow, the Gateway would fail to fetch information about the endpoints and their input/output schemas:

from typing import List, Optional
from docarray import BaseDoc, DocList
from jina import Executor, Flow, requests

class Nested2Doc(BaseDoc):
    value: str


class Nested1Doc(BaseDoc):
    nested: Nested2Doc


class RootDoc(BaseDoc):
    nested: Optional[Nested1Doc]
    
class NestedSchemaExecutor(Executor):
    @requests(on='/endpoint')
    async def endpoint(self, docs: DocList[RootDoc], **kwargs) -> DocList[RootDoc]:
        rets = DocList[RootDoc]()
        rets.append(
            RootDoc(
                text='hello world', nested=Nested1Doc(nested=Nested2Doc(value='test'))
            )
        )
        return rets
    
flow = Flow().add(uses=NestedSchemaExecutor)
with flow:
    res = flow.post(
        on='/endpoint', inputs=RootDoc(text='hello'), return_type=DocList[RootDoc]
    )

...
2023-08-07 02:49:32,529 topology_graph.py[608] WARNING Getting endpoints failed: 'definitions'. Waiting for another trial

This was due to an internal utility function failing to convert such deeply nested JSON schemas to DocArray models.
This release fixes the issue by propagating global schema definitions when generating models for nested schemas.

📗 Documentation Improvements

• Remove extra backtick in create-app.md (#6023)
• Fix streaming endpoint reference in README (#6017)

🤟 Contributors

We would like to thank all contributors to this release:

• Saba Sturua (@jupyterjazz)
• AlaeddineAbdessalem (@alaeddine-13)
• Joan Fontanals (@JoanFM)
• Naymul Islam (@ai-naymul)