To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here
Install python SDK (python 3.7 above) by running:
python3 -m pip install kfp kfp-server-api --upgradeSee the Change Log
What's Changed
- feat(components): Bump image tag used by
preview.llmpipelines by @copybara-service in #10295 - feat(components): Add
num_microbatchesto_implementation.llmtraining components by @copybara-service in #10248 - chore(sdk): remove cleanup param in local init #localexecution by @connor-mccarthy in #10293
- feat(sdk): add local execution skeleton #localexecution by @connor-mccarthy in #10292
- fix(components): Use
llama-2-7bfor the base reward model when tuningllama-2-13with thepreview.llm.rlhf_pipelineby @copybara-service in #10249 - feat(sdk): add local task dispatcher validation and ExecutorInput construction #localexecution by @connor-mccarthy in #10298
- test(sdk): install
kfp-pipeline-specfrom source inkfpsdk tests by @connor-mccarthy in #10300 - fix(sdk): use kfp.dsl.types to replace kfp.components.types Fixes #10282 by @hsinhoyeh in #10283
- feat(components): Enable text generation pipeline to generate row based metrics by @copybara-service in #10296
- test(sdk): add placeholder_utils_test.py by @connor-mccarthy in #10301
- feat(component): Migrate AutoSxS pipeline to preview and move related files to _implementation/llm directory to help Model Eval team use side by side metrics as part of their pipeline by @copybara-service in #10294
- feat(components): Release new LLM Eval image version 0.5 by @copybara-service in #10313
- chore(components): release GCPC 2.7.0 by @copybara-service in #10303
- fix(components): Add autosxs_pipeline to the all variable for the preview/model_evaluation directory by @copybara-service in #10317
- chore(components): release GCPC 2.8.0 by @copybara-service in #10316
- feat(components): Add better docstrings for AutoSxS by @copybara-service in #10320
- test(components): INTERNAL by @copybara-service in #10314
- feat(sdk): add subprocess task handler #localexecution by @connor-mccarthy in #10302
- fix(components): Use
large_model_referenceasmodel_reference_namewhen uploading models frompreview.llm.rlhf_pipelineinstead of hardcoding value astext-bison@001by @copybara-service in #10321 - test(sdk): add
kfp-kubernetesexecution tests by @connor-mccarthy in #10304 - fix(components): Resolve unique model display name on each
preview.llm.rlhf_pipelinerun instead of reusing cached result by @copybara-service in #10322 - fix(components): Upload the tuned adapter to Model Registry instead of model checkpoint from
preview.llm.rlhf_pipelineby @copybara-service in #10323 - feat(sdk): add local execution output collection #localexecution by @connor-mccarthy in #10325
- chore(sdk): fix use of invalid escape sequence in tests by @connor-mccarthy in #10310
- feat(sdk): add local execution logging #localexecution by @connor-mccarthy in #10326
- chore(components): Fix argument description by @copybara-service in #10327
- feat(backend): preserve querystring in pipeline root (fixes #10318) by @TobiasGoerke in #10319
- feat(components): change output format to allow possible post eval by @copybara-service in #10281
- fix(frontend): Add disableParsingRawHTML option for markdown-to-jsx component by @zijianjoy in #10315
- feat(sdk): add DockerRunner #localexecution by @connor-mccarthy in #10328
- feat(sdk): add special
dsl.OutputPathread logic #localexecution by @connor-mccarthy in #10334 - chore(sdk): write local execution logs to stdout #localexecution by @connor-mccarthy in #10330
- feat(sdk): support local Container Component execution #localexecution by @connor-mccarthy in #10333
- feat(components): Update RLHF env vars to handle empty string by @copybara-service in #10331
- feat(components): Output errors as a separate table from Arbiter by @copybara-service in #10329
- docs(components): Fix AutoSxS docstring formatting by @copybara-service in #10341
- chore(components): add
json_escapeplaceholder util by @copybara-service in #10351 - fix(sdk): fix presentation of strings in local execution #localexecution by @connor-mccarthy in #10353
- fix(sdk): remove redundant newline character in local
DockerRunnerlogs #localexecution by @connor-mccarthy in #10354 - test(sdk): improve KFP SDK local runner test safety #localexecution by @connor-mccarthy in #10336
- feat(backend): Enable logging for KFP components by @DharmitD in #10288
- feat(sdk): remove local execution feature flag #localexecution by @connor-mccarthy in #10355
- fix(sdk): permit empty local execution outputs #localexecution by @connor-mccarthy in #10338
- feat(sdk): support Concat and IfPresent placeholder in local container component execution #localexecution by @connor-mccarthy in #10348
- test(sdk): add test for local execution of
Nonedefault parameter #localexecution by @connor-mccarthy in #10339 - chore(sdk): release KFP SDK 2.5.0 by @connor-mccarthy in #10364
- chore(sdk): depend on protobuf 4 in
kfp-pipeline-specby @connor-mccarthy in #10305 - docs(components): Document AutoML Tables util functions by @copybara-service in #10359
- feat(components): Implement the feature store grounding pipeline by @copybara-service in #10332
- feat(components): update eval pipeline documentation to clarify the required pipeline parameters by @copybara-service in #10366
- docs(components): update BigqueryQueryJobOp docs on container args limit by @copybara-service in #10380
- chore(components): GCPC 2.8.1 release by @copybara-service in #10381
- feat(kubernetes_platform): Update kubernetes_platform go package to include pod labels and annotations by @Tomcli in #10357
- chore(sdk): depend on protobuf 4 in
kfpandkfp-kubernetesby @connor-mccarthy in #10307 - feat(components): Implement new output format of inference component by @copybara-service in #10375
- chore(components): rollback GCPC 2.8.1 release by @copybara-service in #10385
- chore(sdk): release KFP SDK 2.6.0 by @connor-mccarthy in #10386
- chore(components): Add v1.model.ModelGetOp components by @copybara-service in #10226
- chore(sdk): release kfp-kubernetes 1.1.0 by @connor-mccarthy in #10387
- chore(sdk): update
kfp-kubernetesdocs versions and release scripts by @connor-mccarthy in #10388 - docs(sdk): fix
kfp-kubernetesdocs build error by @connor-mccarthy in #10389 - chore(components): Sync AutoML components by @copybara-service in #10372
- fix(components): Update base image for KFP lightweight component for VPC SC compliance by @copybara-service in #10374
- chore: Fix metrics visualization v2 sample by @rimolive in #10399
- docs(sdk): add kfp.local to reference docs #localexecution by @connor-mccarthy in #10395
- chore(components): migrate GCPC to protobuf 4; require KFP>=2.6.0 by @copybara-service in #10401
- feat(components): Support scheduling and labels in utils.build_payload by @copybara-service in #10408
- chore(components): Sync AutoML components by @copybara-service in #10415
- feat(components): support aliases arg in ModelUploadOp by @copybara-service in #10404
- fix(sdk): fix string quotes by @connor-mccarthy in #10413
- docs(components): Update AutoSxS pipeline to use "question_answering" as task name instead of "question_answer", where "question_answer" is still supported, but deprecated by @copybara-service in #10361
- fix(components): Update base image for KFP lightweight component for VPC SC compliance by @copybara-service in #10421
- chore(sdk): local task execution refactor + cleanup #localexecution by @connor-mccarthy in #10420
- fix(backend): MLMD pagination on getting executions of DAG by @hellojunha in #10396
- fix(components): Only run
preview.llm.bulk_inferenceafter tuning third-party models with RLHF by @copybara-service in #10425 - feat(components): Update LLM Evaluation Pipelines to use
text-bison@002model by default by @copybara-service in #10428 - chore(components): GCPC 2.9.0 Release by @copybara-service in #10432
- chore(components): fix GCPC documentation dependencies issue by @copybara-service in #10433
- feat(components): Use a single inference component for AutoSxS by @copybara-service in #10424
- feat(sdk): support local pipeline execution #localexecution by @connor-mccarthy in #10423
- fix(components): Write model resource_name to the output of training pipeline remote runner by @copybara-service in #10426
- feat(sdk): support f-strings in local pipeline execution #localexecution by @connor-mccarthy in #10435
- feat(sdk): support for local execution of
dsl.importer#localexecution by @connor-mccarthy in #10431 - fix(components): Update text generation pipeline input description by @copybara-service in #10434
- feat(sdk): support local execution of pipelines in pipelines #localexecution by @connor-mccarthy in #10440
- fix(backend): correct run field map col names by @HumairAK in #10430
- chore(components): Create module containing AutoSxS and RLHF image tag by @copybara-service in #10445
- fix(components): Remove row_based_metric file path field in Model Evaluation resource's metadata when row based metrics is disabled by @copybara-service in #10444
- feat(sdk): Support dsl.ParallelFor over list of Artifacts by @KevinGrantLee in #10441
- feat(kubernetes_platform): Update kubernetes_platform go package to include ImagePullSecrets by @JosepSampe in #10410
- feat(kubernetes_platform): Update kubernetes_platform go package to include imagePullPolicy. by @revit13 in #10416
- chore(components): update image tags by @copybara-service in #10455
- fix(sdk): fixes type issues for ParallelFor. Fixes #9366 by @rickyxie0929 in #10436
- feat(components): Add RLAIF pipeline to preview by @copybara-service in #10377
- fix(sdk): fix bug where
dsl.OneOfwith multiple consumers cannot be compiled by @connor-mccarthy in #10452 - chore(components): Use new module for looking up ReFINED and AutoSxS image tags by @copybara-service in #10454
- feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ImagePullSecrets by @JosepSampe in #10427
- Fix(backend): fix timeout for internal server error. Fixes #10267 by @abaland in #10439
- fix(backend): get pipeline by name is broken due to version typo, Fixes #9940 by @cybernagle in #10268
- feat(kubernetes_platform): Update kubernetes_platform go package to include ConfigMaps as volumes and as env variables. by @roytman in #10400
- feat(Backend + SDK): Update kfp backend and kubernetes sdk to support pod labels and annotations by @Tomcli in #10393
- chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10472
- feat(kubernetes_platform): Add ActiveDeadlineSeconds(timeout) to the kubernetes platform spec by @Tomcli in #10464
- chore(components): support jinja 3 in GCPC by @copybara-service in #10479
- test(components): Reduce sagemaker component test flakiness by @ananth102 in #10225
- chore(sdk): release KFP SDK 2.7.0 by @connor-mccarthy in #10461
- fix(components): Use PipelineJob location in AutoSxS components, add init file by @copybara-service in #10476
- feat(kubernetes_platform): Add k8s FieldPath as env to the kubernetes_platform by @Tomcli in #10485
- chore: Add Tomcli as a backend approver by @chensun in #10490
- chore(components): Sync AutoML components by @copybara-service in #10492
- chore(README): Update Kubeflow Pipelines on Tekton blog by @Tomcli in #10482
- chore(components): Rename several
_implementation.llmcomponents by @copybara-service in #10474 - feat(kubernetes_platform): Update kubernetes_platform go package to i… by @droctothorpe in #10442
- docs(components): internal by @copybara-service in #10495
- chore(components): bump highest support KFP SDK version in GCPC to KFP SDK 2.7.0 by @copybara-service in #10498
- fix(backend): fixes "cannot save parameter" error message. Fixes #9678 by @hbelmiro in #10459
- fix(rlhf): Supporting adapter only output for reward model training by @copybara-service in #10503
- chore(components): internal by @copybara-service in #10511
- docs(components): internal by @copybara-service in #10505
- feat(components): Added experimental args to batch_prediction_pairwise component by @copybara-service in #10497
- chore(components): release GCPC SDK 2.10.0 by @copybara-service in #10516
- feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations by @droctothorpe in #10471
- feat(components): Add configurable image prefix to llm utility method by @copybara-service in #10513
- chore(components): internal change by @copybara-service in #10512
- feat(components): Add CMEK support to
preview.llm.rlhf_pipelineby @copybara-service in #10456 - feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ConfigMaps as volumes and as env variables by @roytman in #10483
- feat(components): Release Forecasting training pipelines to V1 namespace by @copybara-service in #10523
- fix(sdk): Prevents dsl.ParallelFor over single parameter from compiling. by @KevinGrantLee in #10494
- fix(components): Pass tuned model checkpoint to inference pipeline after RLHF tuning by @copybara-service in #10518
- test: Add ParallelFor compile test over single artifact. by @KevinGrantLee in #10531
- fix(components): rename custom task calibration_score_rubric -> score_rubric by @copybara-service in #10525
- chore(components): Switch default machine type for LLM Text Generation Eval pipeline and components to e2-standard-4 by @copybara-service in #10522
- fix(components): Propagate location to sub-components in AutoSxS by @copybara-service in #10514
- feat(backend + SDK): Add backend and SDK support to use Kubernetes FieldPath as env by @Tomcli in #10496
- chore(components): Change docker image URI used by
preview.llmpipelines by @copybara-service in #10536 - fix(docs): Updated legal info due to migration from CLA to DCO by @hbelmiro in #10501
- feat(backend + SDK): Add Backend and SDK support for timeout in pod spec by @Tomcli in #10481
- fix(components): Return None as sliced feature attribution values for the classes which are not predicted in bp outputs by @copybara-service in #10539
- feat(components): Add CMEK validation to
preview.llm.infer_pipelineby @copybara-service in #10519 - feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ImagePullPolicy by @revit13 in #10417
- docs(components): fix
create_custom_training_job_from_componentdocs rendering by @copybara-service in #10541 - feat(components): Implement the train time evaluation in reward model training. With the train time eval dataset available, the pipeline outputs the accuracy and cross entropy metrics to the log by @copybara-service in #10369
- fix(samples): Updated samples/core to V2 by @PratyushaRavi in #9879
- chore(components): Consolidate validation components in
preview.llm.rlhf_pipelineby @copybara-service in #10543 - feat(backend): Upgrade go version to 1.20 by @Tomcli in #10502
- docs(components): Modify the GetModel documentation by @copybara-service in #10552
- chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10551
- fix(backend): Fixes response status of http error code when uploading duplicate pipeline [Fixes #10311] by @champon1020 in #10546
- chore: Change stalebot rules by @rimolive in #10547
- fix(kubernetes_platform): Add optional field to SecretAsVolume and ConfigMapAsVolume. Fixes #10548 by @revit13 in #10549
- feat(components): Add CMEK support to AutoSxS pipeline by @copybara-service in #10555
- feat(components): Update _LLM_EVAL_VERSION to v0.6 by @copybara-service in #10565
- chore(components): update container image of endpoint batch predict component for vulnerability patch by @copybara-service in #10566
- chore(backend): Update kfp driver and launcher images by @chensun in #10561
- fix(components): Add relevant component and pipeline inputs/outputs to support creating ModelEvaluations as part of the AutoSxS Metrics component by @copybara-service in #10554
- feat(components): Release Forecasting training pipelines to V1 namespace by @copybara-service in #10569
- feat(components): Add location validation to
preview.llm.rlhf_pipelineby @copybara-service in #10542 - chore(components): Testing fix by @copybara-service in #10576
- fix(samples): Update resource_spec, retry, secret samples to v2 pipelines by @junggil in #9876
- fix: Modified the comment/text for pipeline_version_id by @amadhusu in #10581
- chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10582
- chore: Update api-generator to use golang 1.20 by @chensun in #10580
- fix(docs): make full version dropdown show on all KFP SDK docs versions by @connor-mccarthy in #10577
- feat(components): Implement new component to preprocess and validate inputs for rlhf by @copybara-service in #10480
- chore(components): GCPC 2.11.0 Release by @copybara-service in #10573
- fix(components): Fix missing pipeline parameters by @copybara-service in #10587
- chore: Update release image by @chensun in #10599
- fix: Modified the swagger json files according to the modified proto files. by @amadhusu in #10591
- feat(components): Copy text generation eval and text classification evak pipelines from preview to v1 by @copybara-service in #10586
- docs(components): internal by @copybara-service in #10556
New Contributors
- @TobiasGoerke made their first contribution in #10319
- @rimolive made their first contribution in #10399
- @hellojunha made their first contribution in #10396
- @HumairAK made their first contribution in #10430
- @KevinGrantLee made their first contribution in #10441
- @JosepSampe made their first contribution in #10410
- @revit13 made their first contribution in #10416
- @cybernagle made their first contribution in #10268
- @roytman made their first contribution in #10400
- @hbelmiro made their first contribution in #10459
- @PratyushaRavi made their first contribution in #9879
- @champon1020 made their first contribution in #10546
- @amadhusu made their first contribution in #10581
Full Changelog: 2.0.5...2.1.0