Lessons Learned Building a GenAI Powered App - Marc Cohen & Mete Atamel

LLMs provide powerful quiz generation capabilities but require specific prompt engineering and defensive coding to handle inconsistent outputs and potential failures

Model accuracy has improved significantly over time - from PALM (80%) to Gemini Pro (70%) to Gemini Ultra (94%) for quiz validation

Keep prompts minimal and specific initially, then iterate and version them like code. More detailed prompts don’t always lead to better results

Implement proper error handling and validation since LLM calls are slow and can fail or return unexpected formats. Cache common responses where possible

Consider using higher-level abstractions/frameworks but be aware they add complexity and reduce control over the underlying functionality

Traditional software engineering practices still apply - unit testing, monitoring, logging and defensive coding are even more important with GenAI

Automate testing and validation of LLM outputs. Develop metrics to measure output quality and accuracy

Cost considerations are important - batch requests where possible and implement caching strategies to minimize API calls

Model versions should be pinned/locked to maintain consistency, but also plan for how to evaluate and adopt new improved models

Not everything needs an LLM - consider simpler alternatives when appropriate. GenAI should complement rather than completely replace existing solutions

Real-time applications need special handling due to LLM latency - consider asynchronous processing and appropriate UI feedback