Notes
flytekitplugins-kfpytorch
The distributed pytorch and distributed elastic-pytorch tasks in flytekitplugins-kfpytorch by default increase the shared memory limit by mounting an emptyDir volume with medium Memory to to /dev/shm as this is almost always required when working with torch multiprocessing (e.g. multi-processed data loader workers or local worker group in distributed training). To disable this, pass increase_shared_mem=False to task_config=PyTorch/Elastic. Elastic tasks now also set a default join timeout of 15 minutes to prevent timeouts when some worker pods require a node scale-up. This setting can be modified via task_config=Elastic(rdzv_configs{...}).
What's Changed
- Add myself to code owners of flytekit-kf-pytorch by @fg91 in #2556
- Add run_policy to ElasticTask by @bgedik in #2555
- Added entrypoint to imagespec and default builder by @pryce-turner in #2553
- Set default width if fail to get terminal size by @pingsutw in #2558
- Disable rich traceback for papermill plugin by @pingsutw in #2559
- fix datetime in eager workflow by @novahow in #2541
- Fix CSS with Flyte decks by @thomasjpfan in #2565
- Fix
FlyteDirectoryon Azure by @Tom-Newton in #2564 - Default image builder: Creates HOME if it does not exists by @thomasjpfan in #2562
- Removes isodate requirement by @thomasjpfan in #2568
- Read FLYTE_SDK_DEV_LOGGING_LEVEL from env by @pingsutw in #2571
- print native input/output when cache hit by @pingsutw in #2567
- Fix: Set OMP_NUM_THREADS by default in Elastic by @fellhorn in #2569
- Use logging level in env for default dev logger by @pingsutw in #2572
- Bump certifi from 2023.7.22 to 2024.7.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in #2566
- Allow for flytekit version to be specified in default image builder by @thomasjpfan in #2563
- Removes jinja2 dependency by @thomasjpfan in #2570
- Improve error when missing type annotations in task/workflow by @pingsutw in #2549
- Rename databricks task type by @pingsutw in #2574
- remove upper bound of plugin dependencies for flytekit-sqlalchemy by @Mecoli1219 in #2517
- Remove flytekitplugins-deck-standard in Dockerfile by @Future-Outlier in #2582
- Bump certifi from 2024.2.2 to 2024.7.4 by @dependabot in #2581
- handle existing sagemaker deployments gracefully by @samhita-alla in #2400
- Use /opt/micromamba for default image builder by @thomasjpfan in #2578
- Improve error message for missing return by @pingsutw in #2551
- Override Dataclass Serialization/Deserialization Behavior for
FlyteTypesbymashumaroby @Future-Outlier in #2554 - Fix FlyteDirectory docs rendering by @ppiegaze in #2584
- Make
openai_organizationOptional in OpenAI plugins by @Future-Outlier in #2585 - Adds comet-ml plugin by @thomasjpfan in #2550
- Replace super(AsyncAgentBase) with super(DatabricksAgent) by @pingsutw in #2590
- add kubernetes package to official docker image again by @flixr in #2589
- Feat: Improve UX of pytorch-elastic plugin by configuring reasonable defaults by @fg91 in #2543
- Improved date parsing in CLI with meta variables by @kumare3 in #2595
- truncate sagemaker agent outputs and automate idempotence token handling by @samhita-alla in #2588
- [BUG] support setting extended resources for array node map tasks by @pvditt in #2592
- Fix DataClass Json Schema Error for
get literal typemethod by @Future-Outlier in #2587 - Sagemaker dict determinism by @samhita-alla in #2597
- refactor(core): Enhance return type extraction logic by @pingsutw in #2598
- Feat: Make exception raised by external command authenticator more actionable by @fg91 in #2594
- Fix: Properly re-raise non-grpc exceptions during refreshing of proxy-auth credentials in auth interceptor by @fg91 in #2591
- validate idempotence token length in subsequent tasks by @samhita-alla in #2604
- Add nvidia-l4 gpu accelerator by @eapolinario in #2608
- eliminate redundant literal conversion for
Iterator[JSON]type by @samhita-alla in #2602 - [FlyteSchema] Fix numpy problems by @Future-Outlier in #2619
- add nim plugin by @samhita-alla in #2475
Full Changelog: v1.13.0...v1.13.1