Model Availability Test

Definition of Model Availability Test

The model availability test aims to verify whether the custom inference use cases written by users based on this document pass when executed on the NPU.

How to Perform Model Availability Test

1. Follow the instructions in the model availability test run script below to write the script. Note: Only model adaptation repository administrators with hardware type as "npu" can submit a model availability test request.
2. Click the "Model Evaluation" tab bar to enter the model evaluation page.
3. On the model evaluation page, click the "Availability Test" tab, select the model's weight file, and then click "Run Test" to start the test.
4. Wait for the model availability test case to complete.
5. (Optional) When the model is "Testing", the user can click "Terminate Test" to manually stop the test.
6. After the model evaluation is completed, the execution result will be displayed in the "Availability Test" area. Meanwhile, log download and test result viewing are provided.

Model Availability Test Run Script

The model availability test script starts from deploy.sh. Script writing must strictly follow the specifications in this document.

The model availability test script must include the following two files:

• requirements.txt: The corresponding module required for the script to run (if there are no dependencies to install, it is not necessary to create one).
• deploy.sh: The model evaluation service installs dependencies and starts this model adaptation project based on this script.

File Location

requirements.txt and deploy.sh must be located in the root directory of the repository.

requirements.txt File (Optional)

Set the corresponding dependencies required for running the script on the NPU. Torch_npu, CANN, and Python are provided by the environment according to the selected framework version, so there is no need to add these dependencies again in the requirements.txt file (which may cause dependency installation conflicts). The format example of the library dependency script is as follows:

transformers==4.37.0
accelerate==0.27.2

If no dependencies need to be added, this .txt file can be omitted, and the test task will skip the dependency installation.

deploy.sh File

This file is a shell script that executes the model adaptation inference. There are no strict restrictions on how the inference script runs. Below is the script specification.

Optional Example of Installing Dependencies

python3 -m pip install --upgrade pip setuptools wheel

Constructing the Required Parameters for the Execution Script

The model weights are downloaded by the automated testing side according to the model weight repository selected on the model evaluation page when the test is initiated. In the shell script, if you need to pass the weight path, you can obtain the path of the weight file through the environment variable "$MODEL_PATH". For example, if starting a vllm project, the path of the model is passed as follows:

vllm serve "$MODEL_PATH"  --trust-remote-code --tensor-parallel-size 1 --dtype float16   --max-num-seqs 4  --gpu-memory-utilization 0.95

Adaptation Code Writing Requirements

Adaptation inference code needs to provide a standard openapi inference interface, and the port of the HTTP server service must be: 8000. The model evaluation service will call the corresponding inference interface based on the model weight file task type selected to perform the model evaluation The currently supported evaluation task types are as follows:

Task Type	Task Code	Inference Interface Path
Text Generation	text-generation	/v1/chat/completions
Image to Text	image-text-to-text	/v1/chat/completions
Text to Speech	text-to-speech	/v1/audio/speech
Multimodal	any-to-any	/v1/chat/completions
Speech Recognition	automatic-speech-recognition	/v1/audio/transcriptions
Embedding	embedding	/v1/embeddings

Note: If the adapted framework is started using vllm, this rule can be ignored, as the vllm framework already provides a standard inference interface according to the openapi specification.

Model Weight File Size Limit

• Size Limit: 100GB
• Limit Description: The storage size of the adapted model weight file must not exceed the limit.
• Scope of Impact: If the limit is exceeded, it will trigger a failure in downloading the model weight file, directly leading to the failure of the model evaluation task.

Full Process Code Example

deploy.sh

• vllm Adaptation Verification Example:

#!/bin/sh
set -e
echo "=== MODEL_PATH set to: $MODEL_PATH ==="
vllm serve "$MODEL_PATH"  --trust-remote-code --tensor-parallel-size 1 --dtype float16   --max-num-seqs 4  --gpu-memory-utilization 0.95

Note: The example above is for the vllm startup method, and there is no need to set --served-model-name. The model evaluation service will automatically use the path of the model weight as the served-model-name.