Overview
-
Dataflow continuation now works. If you run a dataflow over a finite
input, all state will be persisted via recovery so if you re-run the
same dataflow pointing at the same input, but with more data
appended at the end, it will correctly continue processing from the
previous end-of-stream. -
Fixes issue with multi-worker recovery. Previously resume data was
being routed to the wrong worker so state would be missing. -
The above two changes require that the recovery format has been
changed for all recovery stores. You cannot resume from recovery
data written with an older version. -
Adds an introspection web server to dataflow workers.
-
Adds
collect_windowoperator.
What's Changed
- Adding manylinux_2_27 wheel building to CI by @miccioest in #169
- Adds webserver by @whoahbot in #175
- Added EXPOSE command in Dockerfiles by @Psykopear in #176
- Adding 3.8, 3.9, and 3.10 python versions to colab CI job by @miccioest in #179
- Use cbfmt with pre-commit by @whoahbot in #180
- Fix issue with resume state not being routed to correct worker by @davidselassie in #182
- Adds collect window operator by @davidselassie in #183
- Preps for v0.14.0 release by @davidselassie in #184
Full Changelog: v0.13.1...v0.14.0