Download Implementing Data Engineering Solutions Using Azure Databricks.DP-750.DumpsBase.2026-06-23.20q.vcex

Box 1: Set deletedFileRetentionDuration to 90.The most important property to set to 90 is deletedFileRetentionDuration.The delta.deletedFileRetentionDuration table property explicitly dictates the safety threshold used by the VACUUM command. Setting this to 90 days ensures that VACUUM will physically remove data files that have been logically deleted or unreferenced in the transaction log for more than 90 days to reclaim storage.Incorrect:UntilArchivedThis is not a Delta table retention property. It is a conceptual flag or cloud lifecycle strategy but does not directly govern the behavior of the VACUUM command.logRetentionDurationThe delta.logRetentionDuration property controls how long the table history and transaction log entries are kept, which defaults to 30 days. While you should ideally keep it equal to or greater than the deleted file retention (e.g., matching it at 90 days), it governs metadata cleanup rather than the physical removal of unreferenced data files targeted by the VACUUM command.Box 2: Run the VACUUM command.Execute the VACUUM table_name command. This deletes the actual physical files from your cloud storage to reclaim space. Tables remain online and fully queryable during this process.Reference: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-vacuum

To create a new schema with a specific managed location in Azure Databricks Unity Catalog, use the CREATE SCHEMA command.CREATE SCHEMA catalog_name.schema_nameMANAGED LOCATION 'abfss://[email protected]/data';Catalog Name: Replace catalog_name with your existing catalog.Schema Name: Replace schema_name with your desired new schema name.Quotes: The storage path must be enclosed in single quotes.Privileges: You must have CREATE SCHEMA privileges on the parent catalog and ownership (or adequate permissions) on the external location.Reference: https://dev.to/encorepartners/creating-your-first-catalog-schema-and-tables-in-databricks-20p3

Box 1: A Serverless clusterTo capture automated data lineage via Unity Catalog for both Databricks notebooks and jobs with low administrative overhead, you should use Serverless compute (or Shared access mode clusters if serverless is unavailable).Recommended Compute OptionsServerless Compute (Highly Recommended):This option completely minimizes administrative overhead. It requires no infrastructure management, scales automatically, starts instantly, and captures lineage inherently for both Databricks Notebooks and Jobs.Box 2: Catalog ExplorerTo review data lineage in an Azure Databricks workspace enabled for Unity Catalog, you should use Catalog Explorer.Because you are using Unity Catalog and Serverless compute, table-level and column-level data lineage is automatically captured for notebooks and jobs without any manual configuration or administrative overhead.View Lineage in Catalog ExplorerYou can visually explore how data flows between tables by navigating through the Azure Databricks user interface:Open the Catalog icon in the sidebar.Select the desired catalog, schema, and table.Click the Lineage tab.Click See lineage graph to view an interactive map of upstream and downstream dependencies.Reference:https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practiceshttps://kanerika.com/blogs/databricks-data-lineage/

Box 1: USAGE, CREATE TABLEUSAGE, CREATE TABLE allows entering the schema and creating new tables.Box 2: TO ROLE group1Reference: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/access-control/privileges-reference

Box 1: Liquid ClusteringTo optimize query performance, implement:To optimize query performance for a rapidly growing, frequently updated managed Delta table filtered by customer and date in Unity Catalog, you must implement Liquid Clustering on both the customer and date columns, while running regular OPTIMIZE and VACUUM maintenance.Liquid Clustering replaces traditional partitioning and Z-Ordering, drastically improving performance for tables with high-frequency write, update, and delete workloads.Box 2: Enable deletion vectors.To optimize updates and deletions:Write and Execution StrategiesUse Deletion Vectors: Ensure Deletion Vectors are enabled (default in Databricks Runtime 12.1+). They record deletes and updates in a separate lightweight file instead of rewriting the entire data file immediately.Leverage MERGE INTO: Execute updates and deletions in a single atomic batch using the MERGE command. Ensure your ON search condition explicitly includes filters for both the customer and date columns to trigger file pruning.Reference: https://blog.devgenius.io/liquid-clustering-in-databricks-4002fddbf0beSecure and govern Unity Catalog objectsQuestion Set 1

Correct:* You create a foreign catalog in Catalog Explorer.You should create a Foreign Catalog using Lakehouse Federation.Data Copying: Lakehouse Federation queries data directly in the source SQL Server without moving or copying it.Seamless Integration: The database schemas and tables appear right inside Unity Catalog alongside your other data objects. Real-time Access: It provides immediate access to live SQL Server data.Incorrect:* You create a Databricks access connector.* You create a Lakeflow Connect pipeline and connect it to DB1.Data Copying: Lakeflow Connect is an ingestion tool that physically replicates and copies data into Databricks-managed storage (Delta tables).Storage Costs: It violates your requirement to keep data out of Databricks storage. * You create a new native catalog in Unity Catalog. Note:To expose the external SQL Server database in Unity Catalog without copying the data, you must use Lakehouse Federation.Here are the step-by-step actions you need to take:1. Create a ConnectionCreate a securable object in Unity Catalog that specifies the path and credentials to access the SQL Server database.Go to Catalog Explorer or use SQL.Select External Data > Connections.Create a connection using the SQL Server connection details (URL, host, port, and database credentials). *-> 2. Create a Foreign CatalogCreate a specific type of catalog in Unity Catalog that mirrors the external database.Use the CREATE FOREIGN CATALOG SQL command or the Catalog Explorer UI.Link this foreign catalog directly to the connection you created in step 1.3. Query the DataOnce the foreign catalog is created, Unity Catalog automatically syncs the schemas andtables from SQL Server.Reference: https://docs.databricks.com/gcp/en/database-objects/

To register an Azure Storage account (ADLS Gen2) as an external location using a managed identity and an Access Connector under the principle of least privilege, you must complete the following configuration workflow:Step 1: Create a Databricks access ConnectorAction 1: Create an Access Connector for Azure Databricks Navigate to the Azure Portal.Create a new resource and search for Access Connector for Azure Databricks.Configure it in the same Azure region as your ADLS Gen2 storage account.Ensure the System-assigned managed identity status is turned On (or attach a specific user-assigned managed identity). Once deployed, navigate to its properties and copy the Resource ID.Step 2: Assign Workspace1 the Storage Blob Data Contributor role for account1.Action 2: Configure Least-Privilege IAM Roles on the Storage Account Navigate to your hierarchical namespace-enabled Azure Storage Account. Open Access Control (IAM) and click Add role assignment. *-> Assign the Storage Blob Data Contributor role.Note on Least Privilege: To enforce true least privilege, avoid assigning this role at the entire storage account level. Instead, scope this role assignment specifically to the target container inside the storage account.Select Managed identity as the assignee type and select your Access Connector.Step 3: Register the access connector as a storage credential in metastore1Action 3: The Databricks access connector must be registered in the metastore (via a Unity Catalog Storage Credential), not in the individual workspace.Centralized Security: Unity Catalog manages security at the account/metastore level.Cross-Workspace Access: Registering it in the metastore allows the credential to be safely shared across multiple workspaces attached to that same metastore.Object Hierarchy: In Unity Catalog, an External Location relies on a Storage Credential. Both are metastore-level securable objects.Reference: https://community.databricks.com/t5/data-governance/unity-catalog-error-creating-table-errorclass-invalid-state/td-p/10009

To ensure a service principal can query data in two specific catalogs while maintaining the principle of least privilege, you must assign the following permissions for each catalog:Unified Catalog PermissionsUSE CATALOG on the specific catalog: This is a prerequisite that allows the principal to see the catalog and its child objects.USE SCHEMA on the schemas containing the data: This allows the principal to interact with the schemas within that catalog.SELECT on the specific tables or views: This provides the actual read access required to query the data.Reference: https://learn.microsoft.com/en-us/purview/data-governance-roles-permissions

Box 1: A Key Vault-backed secret scopeTo securely expose the storage Key to your Databricks notebook while adhering to the principle of least privilege, use an Azure Key Vault-backed secret scope.This approach acts as a read-only bridge to your Key Vault, ensuring the security team manages the key centrally, while the notebook always fetches the latest version dynamically.Secret Exposure: Notebooks reference the secret by an alias name.Always Updated: Databricks fetches the secret directly from Key Vault at runtime, ensuring it always gets the latest version updated by your security team.Least Privilege: Access can be restricted to specific users or groups using Databricks Access Control Lists (ACLs).Box 2: dbutils.secrets.getUse an Azure Key Vault-backed secret scope in combination with the dbutils.secrets.get function.This approach centralizes the secret's management for your security team, follows the principle of least privilege through Azure IAM, and ensures your notebook always fetches the latest runtime value.Run this specific command inside your Databricks Python notebook:storage_key = dbutils.secrets.get(scope="your-scope-name", key="storageKey")Reference: https://medium.com/@techgeorge/how-to-integrate-databricks-with-azure-key-vault-using-rbac-instead-of-access-policies-972e8c1c2971

Enabling Photon acceleration on a job compute cluster is the best option. It significantly speeds up data engineering pipelines, handles JSON file ingestion seamlessly, and optimizes cost-efficiency for bursty production workloads, which are key for processing rapidly arriving IoT sensor events.Enable Photon acceleration for a job compute cluster:Databricks Auto Loader is designed for low-latency, high-volume file ingestion. Running Auto Loader on a job compute cluster provides the lowest operational cost because job clusters are billed at a much lower rate than all-purpose clusters.Enabling Photon acceleration heavily optimizes the ingestion, parsing, and processing of raw JSON data strings, allowing the cluster to handle bursty workloads faster and auto-scale more efficiently.Scenario, Data Environment, Contoso ingests the following operational and business data: Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.Contoso identifies the following data ingestion and processing requirements:-> Auto-scale ingestion pipelines to handle bursty workloads.Handle schema drift for the maintenance and telemetry data.-> Ingest file-based telemetry data by using minimal operational effort.Reference: https://medium.com/@krthiak/10-days-of-data-engineering-interview-qna-day-5-db1c0b58bf86