Documentation
POSThttps://api.groq.com/openai/v1/chat/completions
Creates a model response for the given chat conversation.
Request Body
- frequency_penaltynumber or nullOptionalDefaults to 0
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.
- function_callDeprecatedstring / object or nullOptional
Deprecated in favor of
tool_choice
.Controls which (if any) function is called by the model.
none
means the model will not call a function and instead generates a message.auto
means the model can pick between generating a message or calling a function. Specifying a particular function via{"name": "my_function"}
forces the model to call that function.none
is the default when no functions are present.auto
is the default if functions are present. - functionsDeprecatedarray or nullOptional
Deprecated in favor of
tools
.A list of functions the model may generate JSON inputs for.
- logit_biasobject or nullOptionalDefaults to null
This is not yet supported by any of our models. Modify the likelihood of specified tokens appearing in the completion.
- logprobsboolean or nullOptionalDefaults to false
This is not yet supported by any of our models. Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the
content
ofmessage
. - max_tokensinteger or nullOptional
The maximum number of tokens that can be generated in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length.
- messagesarrayRequired
A list of messages comprising the conversation so far.
- modelstringRequired
ID of the model to use. For details on which models are compatible with the Chat API, see available models
- ninteger or nullOptionalDefaults to 1
How many chat completion choices to generate for each input message. Note that the current moment, only n=1 is supported. Other values will result in a 400 response.
- parallel_tool_callsboolean or nullOptionalDefaults to true
Whether to enable parallel function calling during tool use.
- presence_penaltynumber or nullOptionalDefaults to 0
Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.
- response_formatobject or nullOptional
An object specifying the format that the model must output.
Setting to
{ "type": "json_object" }
enables JSON mode, which guarantees the message the model generates is valid JSON.Important: when using JSON mode, you must also instruct the model to produce JSON yourself via a system or user message.
- seedinteger or nullOptional
If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same
seed
and parameters should return the same result. Determinism is not guaranteed, and you should refer to thesystem_fingerprint
response parameter to monitor changes in the backend. - stopstring / array or nullOptionalDefaults to null
Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.
- streamboolean or nullOptionalDefaults to false
If set, partial message deltas will be sent. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a
data: [DONE]
message. Example code. - stream_optionsobject or nullOptionalDefaults to null
Options for streaming response. Only set this when you set
stream: true
. - temperaturenumber or nullOptionalDefaults to 1
What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both
- tool_choicestring / object or nullOptional
Controls which (if any) tool is called by the model.
none
means the model will not call any tool and instead generates a message.auto
means the model can pick between generating a message or calling one or more tools.required
means the model must call one or more tools. Specifying a particular tool via{"type": "function", "function": {"name": "my_function"}}
forces the model to call that tool.none
is the default when no tools are present.auto
is the default if tools are present. - toolsarray or nullOptional
A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.
- top_logprobsinteger or nullOptional
This is not yet supported by any of our models. An integer between 0 and 20 specifying the number of most likely tokens to return at each token position, each with an associated log probability.
logprobs
must be set totrue
if this parameter is used. - top_pnumber or nullOptionalDefaults to 1
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. We generally recommend altering this or temperature but not both.
- userstring or nullOptional
A unique identifier representing your end-user, which can help us monitor and detect abuse.
Returns
Returns a chat completion object, or a streamed sequence of chat completion chunk objects if the request is streamed.
1// Default
2import Groq from "groq-sdk";
3
4const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });
5
6async function main() {
7 const completion = await groq.chat.completions
8 .create({
9 messages: [
10 {
11 role: "user",
12 content: "Explain the importance of fast language models",
13 },
14 ],
15 model: "mixtral-8x7b-32768",
16 })
17 .then((chatCompletion) => {
18 console.log(chatCompletion.choices[0]?.message?.content || "");
19 });
20}
21
22main();
{
"id": "34a9110d-c39d-423b-9ab9-9c748747b204",
"object": "chat.completion",
"created": 1708045122,
"model": "mixtral-8x7b-32768",
"system_fingerprint": null,
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Low latency Large Language Models (LLMs) are important in the field of artificial intelligence and natural language processing (NLP) for several reasons:\n\n1. Real-time applications: Low latency LLMs are essential for real-time applications such as chatbots, voice assistants, and real-time translation services. These applications require immediate responses, and high latency can lead to a poor user experience.\n\n2. Improved user experience: Low latency LLMs provide a more seamless and responsive user experience. Users are more likely to continue using a service that provides quick and accurate responses, leading to higher user engagement and satisfaction.\n\n3. Competitive advantage: In today's fast-paced digital world, businesses that can provide quick and accurate responses to customer inquiries have a competitive advantage. Low latency LLMs can help businesses respond to customer inquiries more quickly, potentially leading to increased sales and customer loyalty.\n\n4. Better decision-making: Low latency LLMs can provide real-time insights and recommendations, enabling businesses to make better decisions more quickly. This can be particularly important in industries such as finance, healthcare, and logistics, where quick decision-making can have a significant impact on business outcomes.\n\n5. Scalability: Low latency LLMs can handle a higher volume of requests, making them more scalable than high-latency models. This is particularly important for businesses that experience spikes in traffic or have a large user base.\n\nIn summary, low latency LLMs are essential for real-time applications, providing a better user experience, enabling quick decision-making, and improving scalability. As the demand for real-time NLP applications continues to grow, the importance of low latency LLMs will only become more critical."
},
"finish_reason": "stop",
"logprobs": null
}
],
"usage": {
"prompt_tokens": 24,
"completion_tokens": 377,
"total_tokens": 401,
"prompt_time": 0.009,
"completion_time": 0.774,
"total_time": 0.783
}
}
POSThttps://api.groq.com/openai/v1/audio/transcriptions
Transcribes audio into the input language.
Request Body
- filestringRequired
The audio file object (not file name) to transcribe, in one of these formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm.
- languagestringOptional
The language of the input audio. Supplying the input language in ISO-639-1 format will improve accuracy and latency.
- modelstringRequired
ID of the model to use. Only
whisper-large-v3
is currently available. - promptstringOptional
An optional text to guide the model's style or continue a previous audio segment. The prompt should match the audio language.
- response_formatstringOptionalDefaults to json
The format of the transcript output, in one of these options:
json
,text
, orverbose_json
. - temperaturenumberOptionalDefaults to 0
The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit.
- timestamp_granularities[]arrayOptionalDefaults to segment
The timestamp granularities to populate for this transcription.
response_format
must be setverbose_json
to use timestamp granularities. Either or both of these options are supported:word
, orsegment
. Note: There is no additional latency for segment timestamps, but generating word timestamps incurs additional latency.
Returns
Returns an audio transcription object
1import fs from "fs";
2import Groq from "groq-sdk";
3
4const groq = new Groq();
5async function main() {
6 const transcription = await groq.audio.transcriptions.create({
7 file: fs.createReadStream("sample_audio.m4a"),
8 model: "whisper-large-v3",
9 prompt: "Specify context or spelling", // Optional
10 response_format: "json", // Optional
11 language: "en", // Optional
12 temperature: 0.0, // Optional
13 });
14 console.log(transcription.text);
15}
16main();
{
"text": "Your transcribed text appears here...",
"x_groq": {
"id": "req_unique_id"
}
}
POSThttps://api.groq.com/openai/v1/audio/translations
Translates audio into English.
Request Body
- filestringRequired
The audio file object (not file name) translate, in one of these formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, or webm.
- modelstringRequired
ID of the model to use. Only
whisper-large-v3
is currently available. - promptstringOptional
An optional text to guide the model's style or continue a previous audio segment. The prompt should be in English.
- response_formatstringOptionalDefaults to json
The format of the transcript output, in one of these options:
json
,text
, orverbose_json
. - temperaturenumberOptionalDefaults to 0
The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit.
Returns
Returns an audio translation object
1// Default
2import fs from "fs";
3import Groq from "groq-sdk";
4
5const groq = new Groq();
6async function main() {
7 const translation = await groq.audio.translations.create({
8 file: fs.createReadStream("sample_audio.m4a"),
9 model: "whisper-large-v3",
10 prompt: "Specify context or spelling", // Optional
11 response_format: "json", // Optional
12 temperature: 0.0, // Optional
13 });
14 console.log(translation.text);
15}
16main();
{
"text": "Your translated text appears here...",
"x_groq": {
"id": "req_unique_id"
}
}
{
"object": "list",
"data": [
{
"id": "gemma-7b-it",
"object": "model",
"created": 1693721698,
"owned_by": "Google",
"active": true,
"context_window": 8192
},
{
"id": "llama2-70b-4096",
"object": "model",
"created": 1693721698,
"owned_by": "Meta",
"active": true,
"context_window": 4096
},
{
"id": "mixtral-8x7b-32768",
"object": "model",
"created": 1693721698,
"owned_by": "Mistral AI",
"active": true,
"context_window": 32768
}
]
}
{
"id": "llama2-70b-4096",
"object": "model",
"created": 1693721698,
"owned_by": "Meta",
"active": true,
"context_window": 4096
}