Tech Talks · 1:02:51

Operationalizing LLMs: From Prototype to Production

Yogiraj Awati shares how Instacart moves LLM applications beyond prototypes through retrieval, offline and online evaluation, guardrails, observability, and provider fallbacks. Two case studies cover recipe ingestion and Carebot support workflows with API-backed actions and human handoff.

Yogiraj Awati
Yogiraj Awati
Sep 17, 2026
Xin

Chapters

  1. 00:00 Welcome and talk overview
  2. 00:01:29 LLM fundamentals and production stakes
  3. 00:11:38 Grounding LLM responses with retrieval
  4. 00:12:32 Prompt orchestration for recipe extraction
  5. 00:14:08 Evaluation and observability strategies
  6. 00:18:23 Instacart recipe ingestion case study
  7. 00:24:27 Recipe and image generation pipeline
  8. 00:26:34 Recipe image quality challenges
  9. 00:29:09 Carebot goals and design principles
  10. 00:34:18 Carebot architecture and retrieval
  11. 00:37:11 Production lessons and model fallbacks
  12. 00:40:05 Q&A: Tooling stack question
  13. 00:41:08 Q&A: Internal tooling and frameworks
  14. 00:43:49 Q&A: Code testing question
  15. 00:44:42 Q&A: Unit and integration testing
  16. 00:46:52 Q&A: Governance approach question
  17. 00:47:38 Q&A: Runtime governance layers
  18. 00:49:01 Q&A: Automated payments question
  19. 00:49:26 Q&A: Validating financial actions
  20. 00:51:01 Q&A: Model selection question
  21. 00:51:31 Q&A: Matching models to workloads
  22. 00:52:52 Q&A: Provider switching question
  23. 00:53:31 Q&A: Provider migration and rollout
  24. 00:55:02 Q&A: Caching strategy question
  25. 00:55:12 Q&A: Caching static support content
  26. 00:56:11 Q&A: Local model question
  27. 00:56:51 Q&A: Local embeddings and policy
  28. 00:58:28 Q&A: Additional open models question
  29. 00:58:51 Q&A: Open-weight model experiments
  30. 00:59:40 Q&A: Vector database question
  31. 01:00:02 Q&A: KNN and OpenSearch
  32. 01:00:26 Q&A: Near-term outlook question
  33. 01:00:44 Q&A: Infrastructure standardization outlook
  34. 01:01:45 Closing and upcoming talks

Production LLMs · LLMOps · retrieval-augmented-generation · LLM evaluation · AI guardrails · LLM observability · Model fallback · Instacart Carebot

Summary

Putting an LLM into production requires more than choosing a model. Yogiraj Awati breaks the application into model, retrieval, orchestration, evaluation, safety, and monitoring layers, then shows how those concerns appear in two Instacart use cases. A recipe-ingestion pipeline turns unstructured ingredients into normalized, catalog-compatible data, while Carebot combines help-center retrieval with API-backed actions, clarification, policy controls, and human escalation. The session also covers offline evaluation before model or prompt changes reach live experiments, matching models to workloads, caching stable support content, and preparing tested provider fallbacks.

Watch next

More from the studio

Beyond the DAG: Building Agentic Workflows That Loop, Branch, and Scale1:17:55Tech Talks

Beyond the DAG: Building Agentic Workflows That Loop, Branch, and Scale

Agentic workflows loop, branch, retry, and trigger new work instead of following a fixed DAG. Santosh Kumar Radha explains how AgentField uses typed functions, structured outputs, stopping and escalation patterns, event triggers, and controls for scaling, identity, and authorization.

Santosh Kumar RadhaSantosh Kumar Radha·Sep 17, 2026
Building Trustworthy Financial AI: Governance, Bias, and Mechanistic Insights48:04Tech Talks

Building Trustworthy Financial AI: Governance, Bias, and Mechanistic Insights

Fabrizio Dimino examines why financial LLM recommendations can change when option order changes. He connects positional-bias tests and mechanistic interpretability with finance-specific red teaming, risk-sensitive scoring, model validation, and AI governance.

Fabrizio DiminoFabrizio Dimino·Sep 17, 2026
Securing Code You Didn't Write: The AI Era's New Threat Model1:05:16Tech Talks

Securing Code You Didn't Write: The AI Era's New Threat Model

Coding agents expand the software supply chain to include repositories, configuration, comments, connectors, and tool output. Rome Thorstenson examines reported attacks and outlines practical controls for permissions, secrets, prompt injection, and defense in depth.

Rome ThorstensonRome Thorstenson·Sep 17, 2026