Reading time: 5 minutes

What are the latest innovations in the field of Data Engineering announced at the Snowflake Summit in San Francisco? In this interview, Laurent Letourmy, Head of Data at Devoteam Data Driven, speaks with Frédéric Adet, Data Architecture Practice Leader, about using notebooks and Snowpark with Pandas on Snowflake.

How are these new approaches revolutionizing the way data science teams and Python developers can create complex data pipelines?

A new approach to Data Engineering

Laurent: Hello Frédéric, thank you for taking the time to share your expertise with us. You attended the Snowflake Summit in June in San Francisco with Devoteam. What do you remember from this event?

Frederick: Hi Laurent! An idea that comes to mind, coming out of the annual Snowflake conference in which we participated, is to talk about new approaches in Data Engineering, in particular through the use of notebooks and Snowpark with Pandas .

This topic will be of particular interest to those coming from the world of data science and python developers, including those using other Data platforms. The idea is to show how we can create data engineering pipelines, collaborate via notebooks, and do incredible things, including integrating machine learning and artificial intelligence, all without leaving the computer for a moment. Snowflake platform and without the need for SQL language.

The advantages of notebooks and Snowpark

Laurent: Very interesting! How do these new methods differ from traditional approaches with SQL or dbt?

Frederick: The big difference is flexibility and collaboration. With notebooks, you can mix SQL, Python, and even Markdown to document your work directly in the same file. Integration with GitHub or GitLab also allows notebooks to be synchronized, making collaboration, versioning, and deployment to production easier via tools like GitHub Actions.

In comparison, more traditional SQL pipelines can quickly become difficult to read and maintain, especially when they become complex, or when you are not SQL-native. With notebooks, you simplify experimentation and iteration, while maintaining clear visibility into your pipeline for developers who are used to managing code.

Laurent: It seems really powerful! Does this allow you to create pipelines as complex as those built with SQL?

Frederick: Absolutely, and even more! With notebooks, you can create even more complex pipelines without losing readability. Where a classic SQL pipeline could quickly become indigestible, here you can structure your code in a more modular and understandable way.

In addition, with the integration of the Pandas API, you have access to the entire data science ecosystem, which facilitates the observability, testing, and industrialization of your pipeline.

And what?

Laurent: We talk a lot about dbt at the moment. Does this approach compete with dbt?

Frederick: Yes, it’s true that dbt is very popular, but it must be said, it is mainly reserved for SQL transformation. With the notebook and Snowpark approach, this is no longer a requirement, and the most important thing is that you can capitalize on your python, data science, etc. skills.

For companies with data science teams, this approach is much more natural and intuitive. They can create secure and sophisticated pipelines without having to change paradigms or train their teams on dbt.

How to get started?

Laurent: What would be your advice for those who would like to get started with these new features?

Frederick: I always recommend starting with the quickstarts, especially those focused on data engineering with Snowpark and the Pandas API. Notably this one and this one. It’s an excellent way to discover everything that this approach allows in terms of data engineering and time-to-market. It is also a way to see to what extent this method can make life easier for teams while increasing their efficiency.

I would like to take this opportunity to admit that personally, I systematically use the quickstarts to discover the many features of Snowflake. ChatGPT & Stack Overflow is very good, but we must not forget the formidable effectiveness of learning by example.

What this changes for Data Science

Laurent: Thank you Frédéric, it’s really fascinating. We feel that this subject is close to your heart, it must be linked to your background in data science, right?

Frederick: Exactly. When I first started with Snowflake, I wondered how to do machine learning with data directly in Snowflake, which was quite complicated at the time.

Today, data scientists find themselves at the heart of the platform, and can carry out the entire data chain, from experimentation to industrialization, without having to change technologies. This is a real plus for them, and it allows them to play a central role in the company’s data strategy, in the core Data team and to be more involved in the production of data platforms and products.

And now, what are the prospects?

Laurent: To close these discussions, what are your impressions a few months after the Snowflake Summit?

Frederick: This was a first for me in San Francisco and at Snowflake Summit. Although I had prepared for it, this week immersed in conferences and workshops with enthusiasts from all over the planet made me vibrate with emotions and allowed me to crystallize and validate Business perspectives in terms of organization, methodology and technique that I envisaged.

A company that uses a Snowflake platform can, with an organized core team, experiment, deploy and industrialize services, applications (business processes, BI, ML/DL, AI, etc.) simply and by offering value quickly, thus easily differentiating itself from its competitors.

Last subject, barely three months later, the vast majority of the features announced are available on the platform, and their commitment to deliver is kept, as announced by the new CEO.