Online Inference

Online Inference：The model performs real-time computations when it receives user requests and immediately returns a single prediction result (usually within milliseconds).It requires low-latency, high-concurrency model serving and is commonly used in applications that require immediate feedback, such as recommendation systems, fraud detection, and chatbot conversations.After creating a new inference endpoint, users call the endpoint using an API key.

Experience Link：Online Inference

Inference endpoint types

• Preset Inference Endpoint: For calls to the KAT model series, the platform provides preset inference endpoints for the corresponding models so users can call them quickly.
• Custom Inference Endpoint: Users create these endpoints themselves; they support a wider range of model choices.

Create Inference Endpoint

Permissions:

• Primary account (i.e., Streamlake‘s account) has access/management rights for all projects under the account and does not need to apply for permissions.
• Sub-accounts: the primary account can grant read-only or admin rights on a per-project basis to its sub-accounts.
• If your account balance is insufficient, please top up first.

After confirming you have the required permissions and sufficient balance:

1. Go to the Online Inference page, choose “Custom Inference Endpoint”, and click “Create Inference Endpoint” to open the creation page.
2. Fill in basic information such as name, application description, and tags. You can configure rate limits for the model connected to the endpoint:
3. • Model Throttling: The maximum overall call frequency allowed for this model under the account. All endpoints calling this model share this total quota. If you wish to increase the overall throttling limit for the model, please submit a support ticket (currently only the primary account can apply).
   • Endpoint Throttling: An independent call frequency limit set for each inference endpoint to control the maximum request volume for that endpoint.

Execution logic:

1. After completing the form, click “Next” to go to the confirmation & submit page. Once submitted successfully you will be redirected automatically to the endpoint details → API Calls page.

Calling an inference endpoint

1. Open the Online Inference list or the endpoint’s detail page, select the inference endpoint you want to call (the endpoint must be in “Running” status), and copy the endpoint ID (e.g., ep-xxx-xxx).
2. On the API Calls page you’ll find the invocation method for the model inference endpoint. Click “View API Key” to obtain the API Key and set it in your environment variables, then use the sample API calls as a reference to invoke the inference endpoint.

Viewing inference services

• The Preset Inference Endpoint page lists the platform’s preset endpoints for the KAT series models. You can modify the endpoint’s rate limit, but other fields are not editable.
• The Custom Inference Endpoint list shows all user-created endpoints under the project. You can edit an endpoint’s name, description, tags, and rate-limiting settings.

Viewing call monitoring

On the Model Monitoring page you can view monitoring records for inference calls. The platform supports aggregation by day/hour/minute and allows custom query periods. Model monitoring is retained for up to one year; the maximum query time span is 31 days.

Chart metrics explained

• Call Volume (calls): Display the total sum of various indicator data within the statistical cycle based on aggregation dimensions: total number of calls, successful calls, failed calls, and prompt cache times.
• Failure Rate (%): Display the call failure rate (call failure/total number of calls) under filtering conditions by aggregation dimension.
• RPM (Requests Per Minute): Display the average RPM value under the filtering conditions based on the aggregation dimension.
• TPM (Tokens Per Minute): Display the data of various indicators under the filtering criteria according to the aggregation dimension: total token count, input token count, output token count.
• First Token Latency (s): Display the average duration of calling the first package based on aggregation dimensions under filtering conditions.
• Non-First Token Latency (s): Display the average duration of calling non first package under filtering conditions based on aggregation dimensions.
• Output Tokens per Second: Display the average speed of calling output tokens under filtering conditions based on aggregation dimensions.
• Duration (s): Average latency under the selected filters by aggregation dimension. API call duration：Time from when a request is sent to when the inference task is completed.
• Total Tokens Consumption: Aggregated totals by dimension (under selected filters): Total tokens consumption, Input tokens, Output tokens, and Prompt cache tokens.