3 things I would do today to become an ML engineer

Apr 05, 2024

Here is my recipe for you to land a Machine Learning job, in 3 steps.

The problem

The demand for ML professionals has never been higher than today. However, the competition for these roles has intensified to the point that landing an ML job has never been tougher.

In my opinion, landing a job is like a game, where you need to play your cards differently than the rest, to stand out, and boost your chances to land a job.

Which means, you need to stop

trying to read every new paper that comes out every day,
scrolling your favourite AI influencer posts on Twitter, or
watching every tutorial that Youtube recommends you

You cannot be a consumer. You need to be a producer.

Because,

In Machine Learning, you are as good as the last thing you have built. Not just read about.

Today I wanna share with you the recipe I would follow to become an ML engineer again.

3 steps to break into ML

Step 1 → Get your feet wet with Kaggle 👣

Machine Learning is intimidating for everyone starting in the field. Yes. EVERYONE.

No matter how many Master programs or Online course certificates you have collected in the past.

Every dataset and problem is a new challenge, you need to

understand,
struggle with, and
finally win

Every. Single. Time. No matter how experienced you are.

Which means that, beyond specific tools or models, you need to be and feel like a problem solver.

And the only way to become a problem solver is to…. SOLVE PROBLEMS.

So here is my suggestion

My suggestion ✨

Start your ML journey by finding a dataset and problem you are interested in Kaggle and train a predictive model using Python.

Do not try to learn every possible ML model at this point. Instead, find the model that solves the job

for tabular data → XGBoost, LightGBM, CatBoost

for NLP → a pre-trained LLM from Hugging Face

for CV → a pre-trained Convolutional Network

and fit the model parameters to get a decent test metric.

Your output at this stage are messy all-in-one Jupyter notebooks you upload to your public Github profile.

Step 2 → Level up your Software development skills 👨🏻‍💻

Once you have solved one problem yourself, you are ready to take the next step. And that is, you need to start writing ML software that others can easily read and use.

Start by learning how to organize your project code. This video I made a couple of weeks ago will help you.

Then, go back to your messy Jupyter notebooks and refactor them following the project template shown in the video.

How?

Extract common functionality into reusable functions.

Add type hints, to make your code more readable. For example:

def get_data_from_store(
    from_date: datetime,
    to_date: datetime
) -> pd.DataFrame:
    """
    Connects to the Feature Store and fetches features ranging from
    `from_date` to `to_date`    
    """
    # your code goes here

Add a linter and formatter, to automatically fix code style issues. There are several tools that will help you (e.g. black, isort…). My personal choise is ruff.

My suggestion ✨

Apart from writing software, you also need to be familiar with versioning software, aka Git. In real-world projects you will be constantly:

creating branches

pushing to remote branches

merging branches, and

often solving merge conflicts.

Practice these 4 steps while working on your projects, so you are prepared for the real job.

Step 3 → Put your ML prototypes to work 🚀

So far you know how to write professional code to train ML models.

The next step is to learn how to put your ML models to work, which means model deployment.

When it comes to deploying, the best approach is to put a Serverless Feature Store as the central component of your system (e.g. Hopsworks) and break-down your system into 3 types of pipelines:

Feature pipelines, that transform raw data into model features, that are saved in the Feature Store.
Training pipelines, that ingest historical features and produce a model, that gets pushed to a model registry
Inference pipelines, that read a batch of recent features from the store, and load the model from the registry, to generate a new batch of fresh predictions.

My suggestion ✨

In this repository you will find a list of ML apps built using Feature-Training-Inference pipelines

→ https://github.com/logicalclocks/hopsworks-tutorials

And if you need more examples to inspire you, check out this video ↓

That’s it for today.

I hope today is the beginning of a new beginning.

Well, it always is, isn’t it?

Rest, enjoy the weekend, and talk to you next week