Problems Product Managers Face When Building Products With Large Language Models (LLMs): And How To Counter

Conversational AI and text UIs are having a moment. But there are headaches in building with the tech. Context is lost over time with long conversations. Chaining is hopeful but not there yet. The current level of latency is unacceptable for some real-time interaction patterns.

Emerging techniques to combat these shortcomings include chain of thought (CoT), few-shot prompting, and think step-by-step.

Shoehorning LLMs into existing products will be hit or miss. The same thing happened with the transition from desktop to mobile. There it became important to play to the platform’s strengths. For mobile, it was the camera, gestures, location, and offline storage. The same approach should be taken for LLMs. Don’t try to take traditional web form design patterns and shoehorn chat UIs into the product.

The key to remember is that LLMs don’t need to be tied to a traditional chat UI. It’s not about forcing users to text-messaging interface. Here are three examples where an app can play to the strength of an LLM.

1. Focus on business documents and coding scenarios where long-form writing already occurs. This makes it more natural to integrate.
2. Remove the human interaction altogether and have your code plus the LLM complete work on their behalf (sending a notification when the work is done).
3. Build in hyper-niche segments where models can be fine-tuned to a high degree of accuracy.

Play to the strength of LLMs, and the immediate opportunities are obvious.