ChatSambaStudio

This will help you get started with SambaStudio chat models. For detailed documentation of all ChatStudio features and configurations head to the API reference.

SambaNova’s SambaStudio SambaStudio is a rich, GUI-based platform that provides the functionality to train, deploy, and manage models in SambaNova DataScale systems.

Overview

Integration details

Class	Package	Local	Serializable	JS support	Downloads	Version
ChatSambaStudio	langchain-sambanova	❌	❌	❌

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Native async	Token usage	Logprobs
✅	✅	✅	✅	✅	❌	✅	✅	✅	❌

Setup

To access ChatSambaStudio models you will need to deploy an endpoint in your SambaStudio platform, install the langchain_sambanova integration package.

pip install langchain-sambanova

Credentials

Get the URL and API Key from your SambaStudio deployed endpoint and add them to your environment variables:

export SAMBASTUDIO_URL="sambastudio-url-key-here"
export SAMBASTUDIO_API_KEY="your-api-key-here"

import getpass
import os

if not os.getenv("SAMBASTUDIO_URL"):
    os.environ["SAMBASTUDIO_URL"] = getpass.getpass("Enter your SambaStudio URL: ")
if not os.getenv("SAMBASTUDIO_API_KEY"):
    os.environ["SAMBASTUDIO_API_KEY"] = getpass.getpass(
        "Enter your SambaStudio API key: "
    )

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# os.environ["LANGSMITH_TRACING"] = "true"
# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

Installation

The LangChain SambaStudio integration lives in the langchain_sambanova package:

%pip install -qU langchain-sambanova

Instantiation

Now we can instantiate our model object and generate chat completions:

from langchain_sambanova import ChatSambaStudio

llm = ChatSambaStudio(
    model="Meta-Llama-3-70B-Instruct-4096",  # set if using a Bundle endpoint
    max_tokens=1024,
    temperature=0.7,
    top_p=0.01,
    do_sample=True,
    process_prompt="True",  # set if using a Bundle endpoint
)

Invocation

messages = [
    (
        "system",
        "You are a helpful assistant that translates English to French."
        "Translate the user sentence.",
    ),
    ("human", "I love programming."),
]
ai_msg = llm.invoke(messages)
ai_msg

AIMessage(content="J'adore la programmation.", response_metadata={'id': 'item0', 'partial': False, 'value': {'completion': "J'adore la programmation.", 'logprobs': {'text_offset': [], 'top_logprobs': []}, 'prompt': '<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant that translates English to French. Translate the user sentence.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nI love programming.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', 'stop_reason': 'end_of_text', 'tokens': ['J', "'", 'ad', 'ore', ' la', ' programm', 'ation', '.'], 'total_tokens_count': 43}, 'params': {}, 'status': None}, id='item0')

print(ai_msg.content)

J'adore la programmation.

Chaining

We can chain our model with a prompt template like so:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate(
    [
        (
            "system",
            "You are a helpful assistant that translates {input_language} "
            "to {output_language}.",
        ),
        ("human", "{input}"),
    ]
)

chain = prompt | llm
chain.invoke(
    {
        "input_language": "English",
        "output_language": "German",
        "input": "I love programming.",
    }
)

AIMessage(content='Ich liebe das Programmieren.', response_metadata={'id': 'item0', 'partial': False, 'value': {'completion': 'Ich liebe das Programmieren.', 'logprobs': {'text_offset': [], 'top_logprobs': []}, 'prompt': '<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant that translates English to German.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nI love programming.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', 'stop_reason': 'end_of_text', 'tokens': ['Ich', ' liebe', ' das', ' Programm', 'ieren', '.'], 'total_tokens_count': 36}, 'params': {}, 'status': None}, id='item0')

Streaming

system = "You are a helpful assistant with pirate accent."
human = "I want to learn more about this animal: {animal}"
prompt = ChatPromptTemplate.from_messages([("system", system), ("human", human)])

chain = prompt | llm

for chunk in chain.stream({"animal": "owl"}):
    print(chunk.content, end="", flush=True)

Arrr, ye landlubber! Ye be wantin' to learn about owls, eh? Well, matey, settle yerself down with a pint o' grog and listen close, for I be tellin' ye about these fascinatin' creatures o' the night!

Owls be birds, but not just any birds, me hearty! They be nocturnal, meanin' they do their huntin' at night, when the rest o' the world be sleepin'. And they be experts at it, too! Their big, round eyes be designed for seein' in the dark, with a special reflective layer called the tapetum lucidum that helps 'em spot prey in the shadows. It's like havin' a built-in lantern, savvy?

But that be not all, me matey! Owls also have acute hearin', which helps 'em pinpoint the slightest sounds in the dark. And their ears be asymmetrical, meanin' one ear be higher than the other, which gives 'em better depth perception. It's like havin' a built-in sonar system, arrr!

Now, ye might be wonderin' how owls fly so silently, like ghosts in the night. Well, it be because o' their special feathers, me hearty! They have soft, fringed feathers on their wings that help reduce noise and turbulence, makin' 'em the sneakiest flyers on the seven seas... er, skies!

Owls come in all shapes and sizes, from the tiny elf owl to the great grey owl, which be one o' the largest owl species in the world. And they be found on every continent, except Antarctica, o' course. They be solitary creatures, but some species be known to form long-term monogamous relationships, like the barn owl and its mate.

So, there ye have it, me hearty! Owls be amazin' creatures, with their clever adaptations and stealthy ways. Now, go forth and spread the word about these magnificent birds o' the night! And remember, if ye ever encounter an owl in the wild, be sure to show respect and keep a weather eye open, or ye might just find yerself on the receivin' end o' a silent, flyin' tackle! Arrr!

Async

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "human",
            "what is the capital of {country}?",
        )
    ]
)

chain = prompt | llm
await chain.ainvoke({"country": "France"})

AIMessage(content='The capital of France is Paris.', response_metadata={'id': 'item0', 'partial': False, 'value': {'completion': 'The capital of France is Paris.', 'logprobs': {'text_offset': [], 'top_logprobs': []}, 'prompt': '<|start_header_id|>user<|end_header_id|>\n\nwhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', 'stop_reason': 'end_of_text', 'tokens': ['The', ' capital', ' of', ' France', ' is', ' Paris', '.'], 'total_tokens_count': 24}, 'params': {}, 'status': None}, id='item0')

Async Streaming

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "human",
            "in less than {num_words} words explain me {topic} ",
        )
    ]
)
chain = prompt | llm

async for chunk in chain.astream({"num_words": 30, "topic": "quantum computers"}):
    print(chunk.content, end="", flush=True)

Quantum computers use quantum bits (qubits) to process multiple possibilities simultaneously, exponentially faster than classical computers, enabling breakthroughs in fields like cryptography, optimization, and simulation.

Tool calling

from datetime import datetime

from langchain_core.messages import HumanMessage, ToolMessage
from langchain_core.tools import tool


@tool
def get_time(kind: str = "both") -> str:
    """Returns current date, current time or both.
    Args:
        kind: date, time or both
    """
    if kind == "date":
        date = datetime.now().strftime("%m/%d/%Y")
        return f"Current date: {date}"
    elif kind == "time":
        time = datetime.now().strftime("%H:%M:%S")
        return f"Current time: {time}"
    else:
        date = datetime.now().strftime("%m/%d/%Y")
        time = datetime.now().strftime("%H:%M:%S")
        return f"Current date: {date}, Current time: {time}"


tools = [get_time]


def invoke_tools(tool_calls, messages):
    available_functions = {tool.name: tool for tool in tools}
    for tool_call in tool_calls:
        selected_tool = available_functions[tool_call["name"]]
        tool_output = selected_tool.invoke(tool_call["args"])
        print(f"Tool output: {tool_output}")
        messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))
    return messages

llm_with_tools = llm.bind_tools(tools=tools)
messages = [
    HumanMessage(
        content="I need to schedule a meeting for two weeks from today. "
        "Can you tell me the exact date of the meeting?"
    )
]

response = llm_with_tools.invoke(messages)
while len(response.tool_calls) > 0:
    print(f"Intermediate model response: {response.tool_calls}")
    messages.append(response)
    messages = invoke_tools(response.tool_calls, messages)
response = llm_with_tools.invoke(messages)

print(f"final response: {response.content}")

Intermediate model response: [{'name': 'get_time', 'args': {'kind': 'date'}, 'id': 'call_4092d5dd21cd4eb494', 'type': 'tool_call'}]
Tool output: Current date: 11/07/2024
final response: The meeting will be exactly two weeks from today, which would be 25/07/2024.

Structured Outputs

from pydantic import BaseModel, Field


class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")


structured_llm = llm.with_structured_output(Joke)

structured_llm.invoke("Tell me a joke about cats")

Joke(setup='Why did the cat join a band?', punchline='Because it wanted to be the purr-cussionist!')

API reference

For detailed documentation of all SambaStudio features and configurations head to the API reference: docs.sambanova.ai/sambastudio/latest/api-ref-landing.html