My recipe for solving real world problems with LLMs - Part 1

Today and in the following N weeks I will share with you the end-2-end process I follow with my clients to build real world AI apps that help them move their business metrics.

This is not a theoretical exercise. This is what the actual job of the AI engineer nowadays is, and the way I approach it.

I will use a real time dataset of financial news and follow a scientific/engineering approach to build and deploy a production-ready sentiment analysis service that can help a company like a trading firm, design better trading strategies, and impact their bottom line.

Let's start!

What’s the plan? 

I am building this thing week by week, in this newsletter, following a data-driven approach.

But, what is that?

Hype-based approach vs data-driven approach for building AI software?

Everyone is telling you that "cooler" solutions are better solutions. But that is often not true.

In real world projects the goal is not to build something "cool".

The goal is to build something that works in production, and breaks as rare as possible.

So you start simple (e.g. a single LLM) and

• based on your evaluation metrics, and

• some error analysis,

• You design a better/more complex solution that moves the needle.

You add complexity one step at a time, until your solution performance is above a certain threshold. Once you hit the threshold, you stop.

If adding a twice as slow model, only improves your performance by 0.00001% you don't use it.

Because real world apps, especially in the real time ML world, need to work well and fast.

The plan

Having said all this, the plan for the following weeks will be more or less the following:

1. Problem framing and solution design - Because a good drawing is worth a 324,234 words.

2. Let's build a minimal system that works but does not move the needle.

3. How to generate a high-quality evaluation dataset for your LLM iterations - Because good data is KING.

4. How to evalute your first solution against the golden dataset - Are we done yet?

5. Error analysis to improve the model - What went wrong and what can we do to fix it?

6. First iteration - Let's build something that works.

7. Second iteration - Let's build something that works even better.

8. Third iteration - Let's build something that works even better.

9. Fourth iteration - Let's build something that works even better.

10. Fifth iteration - Let's build something that works even better.

11. Sixth iteration - Let's build something that works even better.

12. Are we done? - I hope so, I need to go to the beach.

Let's start!

Real world example

Imagine you just landed an AI engineer job at Kai & Nil's Crypto Algorithmic Trading Fund, the coolest trading firm in the world.

Kai and Nil happen to be the names of these two boys

 

and these boys happen to be my sons.

Disclaimer
As far as I know, Kai does not trade crypto, and Nil does not know what an algorithm is, so this company is probably made up.

And if it is not, I would be surprised it is still in business hiring experts like you.

Kai (the CEO + Cofunder) comes to you and says:

We just got alpha access to a high-quality real time source of financial news, and we want to build "something" with LLMs to extract sentiment scores in real time and to improve our trading strategies.

When do you think we can have a working prototype?

Wow. It is your first day on the job, and the first project is already on your plate.

And it is not an easy one, I must admit.

Let me help you decipher what your CEO is telling you.

A little bit of help here

CEOs are notoriously bad at explaining things, and more often than often have opinions about everything. I guess that is the life of a CEO. And the life of my son Kai... and my life. I guess we all have to have opinions about absolutely everything without having a clue what we are talking about.

Anyway, let's get back on track.

When a CEO tells you they want to build "some prototype with LLMs", they DON'T mean a bunch of Python files, Jupyter notebooks, and super-fancy Multi Agentic prototypes that "seem" to work on your computer.

The prototype for a CEO is a minimal working system that

• RUNS on their infrastructure, and

• Produces (or at least seems to promise to produce) some positive impact on a key business metric.

CEOs (and hopefully everyone else in the company, in this case my other son Nil, and you) don't care about complex solutions. They only want things that impact bussines metrics.

Ok. So now, what do you do?

First steps

What I usually do is

• I prepare myself a coffee

• Go sit next to the technical guy in the team (or virtual coffee if you work remotely). In this case, we talk to Nil, the guy in charge of the trading bots who happens to be the same guy who does not know what an algorithm is, and

• Ask him to quickly explain the business logic and infrastructre of the company.

• So that I can draw the solution I will start building.

This is what I call "problem framing" and "solution design".

Don’t be afraid to ask stupid questions. There are no stupid questions, only stupid attitudes.

If you don’t know something and you don’t ask, you are the stupid.

Problem framing and solution design

Basically, we need to build a program (in this case a Python service) that

• non-stop, because crypto markets never sleep,

• ingests data from this real-time data source of news your CEO talked about,

• transforms the raw text of the news into a numerical score representing crypto sentiment,

• serves this output** to the trading bots that Nil is in charge of.

In this case, as is the case in 99% of the companies moving data internally in real-time, the company uses an internal message bus (e.g. Kafka/RabbitMQ/Redpanda/NATS) to allow the different components of the system to communicate with each other.

On top of that, Nil tells you the trading bots get all trading signals from a super fast real-time database.

Which means your solution and the services it will need to communicate with will look something like this:

 

Obviously, your service is just another pipeline in the family of pipelines that the company has already built in the past to feed their trading bot crew.

 

Ok, but where are the LLMs and the agents here?

Yes, you are right. I forgot that.

Parsing an unstructured text into a structured output is something LLMs (together with solid pre-processing and post-processing steps) are really good at.

So we will use some LLM to build a first version of the system.

So our first solution will look like this:

 

My advice 

Do not rush during the problem framing and solution design phase.

Take your time to understand the problem.

Remember, there is nothing more frustrating as an engineer to build something complex that does not work... and eventually no one uses it in the company.

What’s next?

Next week, we will start writing some data engineering code to build our first version of the system, without throwing any LLM yet.

And in 3 weeks, I will start showing you how you build iteratively better LLMs apps using a data-centric approach.

Wanna learn to build agentic systems that move business metrics?

On October 6th, a Kubernetes master called Marius and an AI engineer called Pau (that is me) will start teaching live how to build and deploy a production-ready agentic trading system.

Marius loves building and operating Kubernetes clusters where Terabytes of data flow in real time, and I love building AI software that uses this data to impact business metrics. These have been our jobs for years, and that is exactly what we will teach you.

Live.

One step at a time.

This is NOT a course for AI tourists.

This is a course for AI hard-core builders who want to understand both the big picture and the million low-level details to implement, deploy and operate a production-ready cost-effective agentic system on Kubernetes.

So it will be hard. Is that clear?

Wanna know more?

I have taught live real world ML engineering for more than 200 hours this past year, to more than 600 students. One of them is Aparna

 

Early bird 40% discount finishing soon 🎁

As a subscriber to the Real World ML Newsletter you have a 40% discount that you can redeem in the following 15 days. After that, the discount will fall to 35%, 30%, 25%… you got it.

Which means that today is the cheapest it will ever be.

Access all future cohorts of the course today for a one time payment with a 40% discount.

Are you really going to miss this?

Get 40% OFF NOW

See you on the other side,

Pau