Python Library

Learn how to run inference using our Python API.

In this tutorial, we will teach you how to use the Together Python library to run a chat model. We will be querying the mistralai/Mixtral-8x7B-Instruct-v0.1 model to list out fun things to do in New York.

Pre-requisites

• Ensure you have Python installed on your machine.
• Create a free account to obtain a Together API Key, found in your account settings.
• Install the Together library with pip install --upgrade together:

Querying a chat model

The model we are using for this guide is mistralai/Mixtral-8x7B-Instruct-v0.1. You can browse all available models in this list. Now that we’ve chosen our model, let’s query it using the Together Python SDK.

Python

import os
from together import Together

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))

response = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[{"role": "user", "content": "Tell me fun things to do in New York"}],
)
print(response.choices[0].message.content)

Output

Output

Sure, here are some fun things to do in New York:

1. Visit the Statue of Liberty and Ellis Island.
2. Explore Central Park and visit the Central Park Zoo.
3. Go shopping on Fifth Avenue.
4. Visit the Metropolitan Museum of Art.
5. Watch a Broadway show.
6. Take a stroll across the Brooklyn Bridge.
7. Visit the 9/11 Memorial and Museum.
8. Go ice skating at Rockefeller Center.
9. Take a ferry to Staten Island.
10. Visit the American Museum of Natural History.
11. Explore the neighborhoods of SoHo and Greenwich Village.
12. Go on a food tour of New York's diverse culinary scene.
13. Take a helicopter tour of the city.
14. Visit the Empire State Building and take in the views from the observation deck.
15. Go to a concert or sporting event at Madison Square Garden.
16. Take a walking tour of Wall Street and the Financial District.
17. Visit the Guggenheim Museum.
18. Take a day trip to the beaches of Fire Island.
19. Go on a boat ride around Manhattan.
20. Visit the New York Public Library and see the famous Rose Main Reading Room.

These are just a few ideas, but there are countless other fun things to do in New York!

Streaming tokens from a chat model

If you want to stream the response back, simply specify stream=True.

Python

import os
from together import Together

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))
stream = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[{"role": "user", "content": "tell me about new york"}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Async support

If you want to run several calls in parallel, use our async method.

Python

import os, asyncio
from together import AsyncTogether

async_client = AsyncTogether(api_key=os.environ.get("TOGETHER_API_KEY"))
messages = [
    "What are the top things to do in San Francisco?",
    "What country is Paris in?",
]

async def async_chat_completion(messages):
    async_client = AsyncTogether(api_key=os.environ.get("TOGETHER_API_KEY"))
    tasks = [
        async_client.chat.completions.create(
            model="mistralai/Mixtral-8x7B-Instruct-v0.1",
            messages=[{"role": "user", "content": message}],
        )
        for message in messages
    ]
    responses = await asyncio.gather(*tasks)

    for response in responses:
        print(response.choices[0].message.content)

asyncio.run(async_chat_completion(messages))

Querying a completion model

Completions are for code and language models shown here.

Python

import os
from together import Together

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))

response = client.completions.create(
    model="codellama/CodeLlama-34b-Python-hf",
    prompt="def bubbleSort(): ",
)
print(response.choices[0].text)

Querying an image model

To query an image model, use the .images method and specify the image model you want to use.

Python

import os
from together import Together

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))

response = client.images.generate(
    prompt="space robots",
    model="stabilityai/stable-diffusion-xl-base-1.0",
    steps=10,
    n=4,
)
print(response.data[0].b64_json)

For your reference, here is a link to the Python Library.