Skip to content

Performance Benchmark

Performance benchmarks help you evaluate the scalability and responsiveness of Text Generation and Image-Text-to-Text models under varying user loads. This guide shows how to create, run, and review benchmarks for deployed models that use vLLM, TGI, SGLang, TensorRT or any serving framework used with the supported inference tasks.

1. Overview

  • Applicability – Only supported by Text Generation and Image-Text-to-Text models.
  • Goal – Measure how a model performs (latency, throughput, etc.) under simulated real-world concurrency.

2. Accessing Performance Benchmarks

  1. Navigate to the Deployment page for the model you want to test.
  2. Click on the Performance Benchmark tab to see the main view: a list of previously run benchmarks. Main view: a list of performance benchmarks

3. Creating a New Benchmark

  1. From the main view, click the Create Benchmark button.
  2. Enter the benchmark Title in the drawer. Benchmark creation drawer view
  3. You have now created a draft benchmark. Draft benchmark view

4. Completing a Draft Benchmark

Detailed draft benchmark view

  1. Configuration benchmark parameters – Fill out the benchmark details to run a benchmark:
  2. Number of Concurrent Users – The list of user counts to benchmark against.
    • To quickly add a range of user counts, enter the Min, Step, and Max counts in the Quick Generator and click Generate users.
    • To add a new user count to the list, type a number into the User Counts field and click Add user (this will append to the existing list).
  3. Inference parameters – (Optional) Parameters to send to the model server during benchmarking.
    • The available inference parameters vary between model servers.
    • All requests for each number of users will be sent with the selected parameters, except ignore_eos, max_tokens, and min_tokens.
  4. Input size – The number of tokens to send to the model server.

  5. Output size – The number of tokens the model server will generate.

  6. Save – Save the benchmark details.

  7. Run benchmark – Once the benchmark details have been completed, click Run Benchmark to start the run.

Completed benchmark view with run button enabled

5. Monitoring and Results

  • Real-Time Monitoring – View benchmark progress in the Results Dashboard.
  • Completion – Results appear automatically once the test finishes.

Results Dashboard

  • Compare – Selecting past benchmark runs from a list and compare them with the current benchmark.

Compare benchmarks dropdown view

6. Exporting Results

Export benchmark data for deeper analysis or stakeholder reviews:

  • PDF – Create a shareable summary report.
  • Excel – Download raw data for detailed analytics.

Use these insights to optimize model settings and enhance overall performance.

Next Steps