• Search for:

    Yearly Archives: 2023

    1. Home  - 
    2. 2023
    30 Oct, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-3
    Azure Monitoring Overview, Develop a Batch Processing Solution, Exams of Microsoft, Microsoft DP-203

    For the Apache Spark pool perspective, refer to the “Design a Partition Strategy” section in Chapter 3. Methods like spark.conf.set(“spark.sql.shuffle.partitions”, 200) and df.rdd.getNumPartitions() provide insights and features for managing data partitions and shuffling. One final note concerning the number of

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-3

    14 Sep, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-2
    Azure Monitoring Overview, Develop a Batch Processing Solution, Handle Skew in Data, Microsoft DP-203

    It would also make sense to identify the specific columns that are required from the READING table instead of using the wildcard character. This will reduce the size of the retrieved dataset and reduce latency. This action would not reduce

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-2

    22 Aug, 2023
    Tune Shuffle Partitions – Troubleshoot Data Storage Processing-1
    Azure Monitoring Overview, Develop a Batch Processing Solution, Microsoft DP-203, Tune Queries by Using Cache

    As you learned in the previous section, data shuffling occurs when the data required to successfully execute a SQL command must be retrieved from more than a single node. There are numerous reasons for this. One reason is skewing. When

    Continue ReadingTune Shuffle Partitions – Troubleshoot Data Storage Processing-1

    8 Jul, 2023
    Handle Data Spill – Troubleshoot Data Storage Processing
    Azure Monitoring Overview, Develop a Batch Processing Solution, Exams of Microsoft, Microsoft DP-203, Tune Queries by Using Cache

    A data spill is the same as a data leak, which happens when protected data is transferred to an unauthorized person or stored in an unauthorized location. Data can be spilled by copying a protected file to a publicly accessible

    Continue ReadingHandle Data Spill – Troubleshoot Data Storage Processing

    28 May, 2023
    Optimize Resource Management – Troubleshoot Data Storage Processing-2
    Develop a Batch Processing Solution, Microsoft DP-203, Tune Queries by Using Cache

    Even more important than managing costs is security. Security is the most important aspect to consider when running any data analytics solution. You learned a lot about this in Chapter 8, “Keeping Data Safe and Secure,” which covered topics such

    Continue ReadingOptimize Resource Management – Troubleshoot Data Storage Processing-2

    16 Apr, 2023
    Optimize Resource Management – Troubleshoot Data Storage Processing-1
    Azure Monitoring Overview, Microsoft DP-203, Tune Queries by Using Cache

    As previously mentioned in this book, you should delete unneeded Azure products. Doing so will reduce costs and enable you to efficiently optimize and manage your Azure resources. Removing items that are no longer needed, especially when they generate a

    Continue ReadingOptimize Resource Management – Troubleshoot Data Storage Processing-1