Eric Ma - LlamaBot: a Pythonic interface to Large Language Models | SciPy 2024

LlamaBot is designed as a Pythonic interface to LLMs, focusing on simplicity and composability rather than complex abstractions

The core design philosophy follows the Zen of Python - flat is better than nested, explicit is better than implicit, with intentionally minimal layers of abstraction

Three main bot types are implemented:

• SimpleBot: Basic stateless bot
• ChatBot: Adds conversation memory
• QueryBot: Enables retrieval-augmented generation (RAG)

Key architectural decisions include:

• Separating text generation from retrieval
• Using stateless API calls
• Making prompts configurable and parameterizable
• Enabling composition of different bot types

Default configuration uses OpenAI’s API but supports other providers through liteLLM integration, including local models via Ollama

Practical applications demonstrated include:

• Generating commit messages from git diffs
• Creating social media posts
• Blog post tagging and summarization
• Image generation prompting

Focus on structured text generation through:

• Explicit token masking
• JSON mode validation
• Template-based prompt formatting

Designed for ease of use and maintenance with:

• Single required argument (system prompt)
• Minimal dependencies
• Clear separation of concerns
• Straightforward extension patterns

Built to complement rather than compete with existing tools like LangChain or LlamaIndex, offering a lighter alternative for basic LLM interactions

Open source and targeted at beginners, with emphasis on contributing to documentation and examples