Your LLM issues are really data issues

In this episode of The Stack Overflow Podcast, host Ryan Donovan sits down with Harsha Chintalapani, co-founder and CTO of Collate and co-creator of the open-source metadata platform OpenMetadata. The conversation centers on a critical but often overlooked truth: the real challenges with AI and large language models (LLMs) aren't technical limitations of the models themselves, but rather deep-rooted data issues—especially around structured, real-time production data. Harsha draws from his extensive experience at Yahoo, Hortonworks, and Uber to illustrate how even massive organizations struggle with data semantics, ownership, lineage, and discoverability. He explains that while cloud platforms have solved the problem of data storage and processing, they haven't addressed the human and organizational challenges of understanding what the data actually means. The episode argues that AI’s failure to deliver on promises like 'understanding unstructured data' stems from a lack of explicit, shared business semantics—such as what constitutes a 'customer' or 'customer health'—which are often tacit and inconsistent across teams. The solution, Harsha emphasizes, lies in building robust metadata infrastructure early, using platforms like OpenMetadata to create a centralized, searchable, and semantically rich knowledge graph of data assets. This enables not only better data discovery and governance but also empowers AI to function effectively by giving it context. The episode concludes with practical advice: organizations should start investing in metadata and semantic governance as soon as they form a data team, not after they’ve scaled into chaos.