• Search for:

    Category Archives: Azure Monitoring Overview

    1. Home  - 
    2. Archive by category "Azure Monitoring Overview"
    8 Aug, 2024
    Optimize Pipelines for Analytical or Transactional Purposes – Troubleshoot Data Storage Processing-5
    Azure Monitoring Overview, Develop a Batch Processing Solution, Exams of Microsoft, Handle Skew in Data, Microsoft DP-203, Tune Queries by Using Cache

    The other option in the Compute Size drop‐down is Custom, which enables the Compute Type and Core Count boxes, allowing you to select up to 256 (+ 256 driver cores) for either Basic or Standard type. In addition to allowing

    Continue ReadingOptimize Pipelines for Analytical or Transactional Purposes – Troubleshoot Data Storage Processing-5

    8 Jul, 2024
    Optimize Pipelines for Analytical or Transactional Purposes – Troubleshoot Data Storage Processing-4
    Azure Monitoring Overview, Microsoft DP-203

    Both data flow performance and diagnostic metric sources just presented help you determine where to start with your optimization effort. From both a source and sink perspective, recall the configurations available through their numerous tabs, for example, Projection and Optimize,

    Continue ReadingOptimize Pipelines for Analytical or Transactional Purposes – Troubleshoot Data Storage Processing-4

    30 Oct, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-3
    Azure Monitoring Overview, Develop a Batch Processing Solution, Exams of Microsoft, Microsoft DP-203

    For the Apache Spark pool perspective, refer to the “Design a Partition Strategy” section in Chapter 3. Methods like spark.conf.set(“spark.sql.shuffle.partitions”, 200) and df.rdd.getNumPartitions() provide insights and features for managing data partitions and shuffling. One final note concerning the number of

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-3

    14 Sep, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-2
    Azure Monitoring Overview, Develop a Batch Processing Solution, Handle Skew in Data, Microsoft DP-203

    It would also make sense to identify the specific columns that are required from the READING table instead of using the wildcard character. This will reduce the size of the retrieved dataset and reduce latency. This action would not reduce

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-2

    22 Aug, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-1
    Azure Monitoring Overview, Develop a Batch Processing Solution, Microsoft DP-203, Tune Queries by Using Cache

    As you learned in the previous section, data shuffling occurs when the data required to successfully execute a SQL command must be retrieved from more than a single node. There are numerous reasons for this. One reason is skewing. When

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-1

    8 Jul, 2023
    Handle Data Spill – Troubleshoot Data Storage Processing
    Azure Monitoring Overview, Develop a Batch Processing Solution, Exams of Microsoft, Microsoft DP-203, Tune Queries by Using Cache

    A data spill is the same as a data leak, which happens when protected data is transferred to an unauthorized person or stored in an unauthorized location. Data can be spilled by copying a protected file to a publicly accessible

    Continue ReadingHandle Data Spill – Troubleshoot Data Storage Processing