Given the increase in popularity of technologies such as Snowflake, SQL is more prevalent than ever. That’s excellent news; SQL makes data manipulation a lot easier. Still, a lot of practitioners seem to perceive it as an outdated language. This blog post motivates SQL as the language for data science and provides some practical tips for using it effectively. Why use SQL over Python/R? SQL is a declarative language. Instead of writing a recipe with exact steps (like we do in imperative languages such as Python or R), we describe what we want to accomplish.

tl; dr; This post shows how you can include data documentation in a parquet file by decorating the function that generated it Documenting code makes it more readable to us and others. The same applies to data; it is a good practice to include relevant information to make it more accessible. A straightforward approach is to have a separate file along with the data file. For example, if we go to UCI’s ML Repository to download the iris data set, we’ll see two download options: Data Folder and Data Set Description.

Introduction Operationalizing Data Science projects is no trivial task. At the very least, data analysis workflows have to run on a regular basis to produce up-to-date results: a report with last week’s data or re-training a Machine Learning model due to concept drift. In some cases, the output of such workflows needs to be exposed as an API, for example, a trained Machine Learning model that generates predictions by hitting a REST endpoint.