github ggml-org/llama.cpp b12550

pre-release31 minutes ago
Details

ggml : add backend scheduler tests (#27258)

  • Adds initial scheduler stress tests

  • Adds new sleep op to simplify scheduler testing

  • Adds testcase addressing #23321

  • Adds experimental vulkan GGML_OP_SLEEP

  • Improves structure and naming of previous tests. Adds new tests,
    including one which detects the nkvo-bug correctly, and validates the
    proposed hotfix.

  • Cleanup: Removes CUDA specific comments and nvtx-markers

  • Fix: forgot to add sleep specific test case

  • Fix oversights identified by CI

  • Drops default prints so that it behaves like other ctests. Adds -v flag
    to toggle this behavior

  • Fix: Early exit in CPU-only settings (cause for graviton CI fails)

  • Improves stress_test_linked_list by removing overlap with
    test_chain_all_backends and test_pair_user_inputs. The latter ones test
    either activations or user inputs in isolation and just once, whereas
    the stress test tests all combinations multiple rounds

  • Exclude ACCEL devices from new tests due to incompatibility

  • Fixes vulkan test failure in llvmpipe

  • Adds asserts, improves comments and some checks

  • Fixes nitpicks in white-space and variable naming

  • Adds comments that certain vk drivers could time OP_SLEEP incorrectly

  • Drops limitiation to CPU/GPU in test-sleep

  • Aligns OP_SLEEP guards in cuda with vk guards (input shape and
    contiguity)

  • Implements feedback for OP_SLEEP to work in place (no copy)

  • Apply non-code suggestions from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler johannesg@5d6.de

  • Removes struct backend_supports; enables tests to return skip, instead
    of only pass/fail

  • Make clang-tidy happy

  • Removes sched_backend_caps, improves create_test_scheduler according to
    feedback (L149)

  • Removes "name" field from struct sched_check in favor of
    ggml_set_name/ggml_get_name

  • Drops unnecessary shader_clock.comp test and guards

  • Replaces sched_ prefixes with helper_ prefix

  • Moves test suite to new RAII types for better memory management

  • Adds sleep option to test_chain_all_backends

  • Adds sleep option to first-run arm of the y-shaped test

  • Fix switchcase order for legibility in stress_test_dag

  • Simplfies stress_test_linked_list via double for-loop

  • Adds pseudo-random input to stress_test_linked_list, and runs all
    transitions both with and without CPU input (next to activation copy as
    input)

  • Fix RPC Proto patch due to GGML_OPS bump

  • Adds ggml-cpp.h treatment to test-sleep.cpp

  • Adds TODO pointers for subsequent additional tests.

  • Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler johannesg@5d6.de

  • Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler johannesg@5d6.de

  • Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler johannesg@5d6.de

  • Fixes node compatibility check to not take buffer locations into account.
    This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

  • Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov ggerganov@gmail.com

  • Completes callsite change of gg-suggestion

  • Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
    to avoid mac clock drift and spurious test failures.
    This makes the test a bit nonsensical on CPU (same clock checks itself),
    but the test still has merit on the other backends where
    independent device clocks are used.


Co-authored-by: Johannes Gäßler johannesg@5d6.de
Co-authored-by: Georgi Gerganov ggerganov@gmail.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.