LiteLLM

Litellm is a popular framework to call LLM APIs using the OpenAI format and is being used by many frameworks to abstract api calls.

We can use litellm to call Imagine Playground APIs.

Pre-requisites

pip install litellm

Chat

Non-streaming

from rich.pretty import pprint

import litellm


# litellm.api_base=""
# litellm.api_key=""

response = litellm.completion(
    model="openai/Llama-3.1-8B",
    messages=[
        {
            "role": "user",
            "content": "Tell me about the new wonders of the world",
        }
    ],
    max_tokens=1024,
)

pprint(response.choices[0].message.content)

This will print something similar to:

You're referring to the New 7 Wonders of the World, a list compiled in 2007 through a worldwide poll. Here are the seven wonders:

1. **The Great Wall of China**: A series of fortifications built across several Chinese dynasties to protect the country from invasions. It stretches over 13,000 miles (21,000 km) and is one of the longest structures ever built.
2. **The Taj Mahal** (India): A mausoleum built by Mughal Emperor Shah Jahan in memory of his wife, Mumtaz Mahal. It's a stunning example of Mughal architecture, with intricate marble inlays and a perfect blend of Indian, Persian, and Islamic styles.
3. **Machu Picchu** (Peru): An Inca citadel built in the 15th century, abandoned before the arrival of the Spanish conquistadors, and rediscovered in the 20th century. It's a testament to the engineering and architectural skills of the Incas.
4. **Chichén Itzá** (Mexico): A pre-Columbian Mayan city on the Yucatán Peninsula, built by the Mayans around 1100 AD. It features the Pyramid of Kukulkan, a 98-foot-tall (30 meters) pyramid that aligns with the movements of the sun and the stars.
5. **The Roman Colosseum** (Italy): An amphitheater built in the 1st century AD, hosting gladiatorial contests, animal hunts, and public spectacles. It's a symbol of the power and engineering prowess of the ancient Romans.
6. **The Christ the Redeemer statue** (Brazil): A massive Art Deco statue of Jesus Christ, built in Rio de Janeiro between 1922 and 1931. It's 98 feet (30 meters) tall, with outstretched arms and a serene expression.
7. **The Pyramids of Giza** (Egypt): The only remaining ancient wonder from the original list, these pyramids are the oldest and only remaining structure from the original list of Seven Wonders of the Ancient World. The largest pyramid, the Great Pyramid of Giza, is an astonishing 481 feet (147 meters) tall.

...

Streaming

import litellm


# litellm.api_base=""
# litellm.api_key=""

stream_response = litellm.completion(
    model="openai/Llama-3.1-8B",
    messages=[
        {
            "role": "user",
            "content": "Tell me about the new wonders of the world",
        }
    ],
    max_tokens=500,
    stream=True,
)

for chunk in stream_response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

This will print something similar to:

You're referring to the New 7 Wonders of the World, a list compiled in 2007 through a worldwide poll. Here are the seven wonders:

1. **The Great Wall of China**: A series of fortifications built across several Chinese dynasties to protect the country from invasions. It stretches over 13,000 miles (21,000 km) and is one of the longest structures ever built.
2. **The Taj Mahal** (India): A mausoleum built by Mughal Emperor Shah Jahan in memory of his wife, Mumtaz Mahal. It's a stunning example of Mughal architecture, with intricate marble inlays and a perfect blend of Indian, Persian, and Islamic styles.
3. **Machu Picchu** (Peru): An Inca citadel built in the 15th century, abandoned before the arrival of the Spanish conquistadors, and rediscovered in the 20th century. It's a testament to the engineering and architectural skills of the Incas.
4. **Chichén Itzá** (Mexico): A pre-Columbian Mayan city on the Yucatán Peninsula, built by the Mayans around 1100 AD. It features the Pyramid of Kukulkan, a 98-foot-tall (30 meters) pyramid that aligns with the movements of the sun and the stars.
5. **The Roman Colosseum** (Italy): An amphitheater built in the 1st century AD, hosting gladiatorial contests, animal hunts, and public spectacles. It's a symbol of the power and engineering prowess of the ancient Romans.
6. **The Christ the Redeemer statue** (Brazil): A massive Art Deco statue of Jesus Christ, built in Rio de Janeiro between 1922 and 1931. It's 98 feet (30 meters) tall, with outstretched arms and a serene expression.
7. **The Pyramids of Giza** (Egypt): The only remaining ancient wonder from the original list, these pyramids are the oldest and only remaining structure from the original list of Seven Wonders of the Ancient World. The largest pyramid, the Great Pyramid of Giza, is an astonishing 481 feet (147 meters) tall.

...

Completion

from rich.pretty import pprint

import litellm


# litellm.api_base=""
# litellm.api_key=""

completion_response = litellm.text_completion(
    prompt="def fibonacci(n):",
    model="text-completion-openai/Llama-3.1-8B",
    max_tokens=100,
)

pprint(completion_response.choices[0].text)

This will print something similar to:

\n    if n <= 0:\n        return "Input should be a positive integer"\n    elif n == 1:\n        return 0\n    elif n == 2:\n        return 1\n    else:\n        a, b = 0, 1\n        for _ in range(2, n):\n            a, b = b, a + b\n        return b\n ...

Embedding

from rich.pretty import pprint

import litellm


litellm.suppress_debug_info = True

# litellm.api_base=""
# litellm.api_key=""

response = litellm.embedding(
    model="openai/BAAI/bge-large-en-v1.5", input=["This is amazing", "Quite good stuff"]
)

pprint(response)