Learn Microsoft Fabric: Streamline Your Data Analytics Workflow

You're about to see the first few videos of our introduction to Microsoft Fabric course. If these topics seem interesting to you, use the link in the description to access the full course. In the full course, we'll give you access to a real fabric account where you can practice the topics that are covered in this video. Enjoy in this video, we'll learn what Microsoft Fabric is used for and the Personas at a company that typically use it. Imagine a data workflow at a video streaming company keen on analyzing customer viewing habits to make informed decision about future content. A traditional workflow might look like this. Raw data about viewer habits such as views, likes, or watch duration is collected and stored in an unstructured format. Data engineers then process this raw data through pipelines to store it in tables for efficient querying and analysis. Data scientists then use these tables to analyze and build machine learning models using tools like TensorFlow or Pyspark. The insights are then communicated via dashboards and reports using a tool like Power bi. Data flows through various platforms and tools at every stage of this process, which could result in many issues. For example, some team members might not have access to the tools they need, or the tools that they do have access to might not be compatible with their preferred programming language. In short, this process is messy and complicated. This is the problem that Fabric tries to solve. Fabric is an end to end data analytics platform that integrates all parts of a data workflow into a single unified experience. A data engineer might set up the pipelines to move data into fabric. Database administrators can secure and govern this data regardless of what team is using it. Data scientists can use SQL or Python inside of Fabric to create models. BI developers can use Power BI inside of fabric to create models, dashboards, or reports. As seen in this image. The experience is unified across all of these functions. Fabric can be used by various Personas and it's divided into seven experiences. Data engineers might work primarily in the data factory experience, while data scientists will find relevant tools in the data science experience. We'll explore these experiences in more detail in upcoming videos, but it's essential to understand that each experience uses the same data source. OneLake data stored in OneLake is in a format called Delta Parquet. All experiences use the same data source. For example, a data engineer using a data warehouse might write SQL to interact with that warehouse. Their SQL is passed through an engine named T SQL and their code is compatible with OneLake. Meanwhile, a data scientist might write Python code using the data engineering experience. This Python code is passed through a different engine named Spark that allows Python to interact with the Delta parquet format as well. The key takeaway is no matter what tool you'll use in Fabric, you'll interact with the same data as everyone else. Different teams can choose the tool that best fits their use case or skillset. The data can be managed in a single unified experience. This single unified experience is made possible through OneLake and the Delta Parquet file format. Let's explore the three data engineering experiences in Data Factory, Data Engineering and Data Warehouse Fabric organizes tools into distinct experiences based on their functions, making it easy to find the right tool for your task. Data Factory, for instance, focuses on ingesting and transforming data. If your task involves data ingestion, you can easily find all relevant tools in Data Factory, which without sifting through unrelated ones. Let's begin by exploring tools within Data Factory. The primary Tools here are Dataflow Gen 2 and Data Pipeline. Dataflow Gen 2 ingests data from various sources into the fabric ecosystem. You can select specific data from sources like a snowflake database and load it into Fabric's onelake through a storage solution such as a lakehouse or a warehouse. Dataflow's interface is user friendly, resembling Power Query in Excel or Power Bi, making it accessible to those unfamiliar with programming languages. Data Pipeline, on the other hand, allows for the creation of data workflows. This is a tool ideal for setting up a regular or logic based data ingestion process. For example, you can refresh data in fabric every Monday using a workflow that runs a notebook at the start of every week. Dataflow Gen2 is great for manual data ingestion, while Data Pipeline is suited for continuous data ingestion processing. Next, we have the Data Warehouse experience, which contains the SYNAPSE Data Warehouse. A data warehouse is one of Fabric's main storage options, the other being a lakehouse. Warehouses store data in structured databases that are queryable by SQL. As we saw earlier, you can use tools like Data Pipeline or Dataflow Gen 2 to load data into a warehouse. Warehouses support fully transactional operations with SQL, meaning you can execute both read and write queries using SQL. Remember, data in a warehouse is stored as a parquet file within onelake, allowing all fabric tools to interact with the same data. This means data engineers using Python or analysts using SQL can work seamlessly with the same dataset. Finally, let's discuss the data engineering experience, focusing on the lakehouse. Like a warehouse, a lakehouse is a storage option within Fabric, but unlike a warehouse, lakehouses support both structured and unstructured data. Developers typically interact with lakehouses using notebooks and Python code, using packages like pyspark to transform and analyze data. While SQL can be used to read data in a lakehouse, write operations are not supported, requiring SPARK for data modification. Developers familiar with Python and Spark tend to use lakehouses, while heavy SQL users might prefer to use a warehouse similar to warehouses. Data in lakehouses is stored in the onelake as parquet files, making it accessible to other users in the fabric workspace. Let's dive back into exploring the seven fabric experiences. So far, we've explored three of the seven experiences. Now let's look at the remaining Data Science, Real Time Intelligence, Power bi, and Data Activator. Our three experiences in the last video lived much more in the data engineering world. These experiences are much more focused on analysis and reporting. Let's start with Data Science. The main tools in data science are focused on building models and experiments. Data scientists can use notebooks to write SPARK code for initial data exploration and manipulation. Once data is prepared for experimentation, the ML model and experiment tools can be used to create machine learning models, evaluate their relative strengths, and prepare insights for a tool like Power bi. In short, the Data Science experience contains all the tools that data scientists would need to implement an end to end machine learning workflow. These data science tools primarily work with lakehouses. Remember, lakehouses are typically used when working with notebooks and spark, whereas warehouses are used when working primarily with SQL. Next, we'll look at the Power BI experience. Power BI is a Microsoft product that was released in 2011. It's a product built to create dashboards and visualizations to drive business impact. With Power BI's integration into fabric, the entire data workflow now lives within the same ecosystem. BI developers now have direct access to the data stored in lakehouses and warehouses within Fabric. The next experience we'll look at is Real Time Intelligence. This experience has a suite of tools built around ingesting, processing, analyzing, and monitoring events in real time. This experience has a new data storage the Event House. An Event house is similar to a lakehouse or warehouse, but specifically designed to handle real time data streams. Like with other fabric storage solutions, event houses use OneLake, meaning data in your event house can be accessed across other areas of fabric. The Real Time Intelligence Experience also contains the Event Stream tool. This tool ingests real time data, transforms it, and routes it to the appropriate storage option. You can think of it as the real time equivalent to dataflow. It's a tool where you define how to manipulate your data before sending it to its final destination. Finally, we'll look at the Data Activator experience. This experience lets users define actions to take when certain patterns or conditions are detected in data. It can connect to Power BI reports or event streams. For example, let's say you're collecting data from sensors monitoring a number of items in a warehouse. When the number of items passes a certain threshold, an event could be triggered to send an email to the manager of that warehouse. These events and triggers are configured using the Reflex tool within Data Activator. This video has been just a taste of what you can find in our full fabric course. Use the link in the description to start the course for free, where you can try some of our interactive exercises. In these exercises, you'll gain access to a fabric account right in your browser to try some of these topics you've learned. Thanks.