Dapr 1.16.4
This update includes bug fixes:
- Workflow logging loop when no pending task completed
- Deleted Jobs in all prefix matching deleted Namespaces
Workflow logging loop when no pending task completed
Problem
When Daprd is made unhealthy during a workflow execution, pending activity tasks which are made completed will result in looping logs from daprd.
Impact
Daprd will continually print log messages indicating that activity task result has been completed for no pending tasks.
Root Cause
Daprd holds a streaming connection to Schedulers which handles job execution for the Jobs API, Actor Reminders, and workflow execution.
Each stream established has a single set of types which the client supports.
When the app reports as unhealthy, the stream to Schedulers need to be re-established as daprd no longer supports the Jobs API and Actor Reminders while the app is unhealthy.
This restarts the workflow runtime, which clears all pending activity tasks.
Resulting task completed from the previous execution are then received with no pending tasks, causing an internal error.
This error is intentionally retried indefinitely, resulting in a logging loop.
Solution
The error occurring from no pending tasks is now typed as a non-retryable error, preventing the logging loop.
Deleted Jobs in all prefix matching deleted Namespaces
Problem
Deleting a namespace in Kubernetes will delete all the associated jobs in that namespace.
If there are any other namespaces with a name which has a prefix matching the deleted namespace, the jobs in those namespaces will also be deleted (i.e. deleting namespace "test" will also delete jobs in namespace "test-1" or "test-abc").
Impact
Deleting a namespace will delete jobs in other namespaces with prefix matching the deleted namespace.
Root Cause
Prefix logic did not terminate the prefix match with an exact match so that deleting a namespace would delete jobs in other namespaces with prefix matching the deleted namespace.
Solution
The prefix logic has been updated to ensure that only jobs in the exact deleted namespace are deleted.