Lilly Tech Systems
Databricks Workspace
Learn how to set up and manage Databricks workspaces, configure clusters, work with notebooks, and administer environments for enterprise teams.
Workspace Overview
A Databricks workspace is the primary environment where teams collaborate on data and AI projects. It provides a unified interface for notebooks, clusters, jobs, data assets, and administrative controls.
Each workspace is deployed into your cloud account (AWS, Azure, or GCP), ensuring your data never leaves your infrastructure while Databricks manages the control plane.
Cluster Management
Clusters are the compute backbone of Databricks. Understanding cluster types and configuration is essential:
| Cluster Type | Use Case | Key Features |
|---|---|---|
| All-Purpose | Interactive development | Shared, auto-scaling, notebook-attached |
| Job Clusters | Automated workloads | Ephemeral, cost-efficient, per-job |
| SQL Warehouses | SQL analytics | Photon-powered, serverless option |
| Serverless | On-demand compute | Zero management, instant startup |
Notebooks
Databricks notebooks support Python, SQL, Scala, and R within the same notebook using magic commands:
# Python cell - default language
df = spark.read.table("catalog.schema.my_table")
df.display()
-- SQL cell (use %sql magic command)
-- %sql
-- SELECT * FROM catalog.schema.my_table LIMIT 10
# Notebooks support:
# - Real-time co-authoring
# - Version control with Git integration
# - Widgets for parameterization
# - Automated scheduling as jobsJobs & Workflows
Databricks Workflows lets you orchestrate multi-step data pipelines:
- Task orchestration: Chain notebooks, Python scripts, JARs, and SQL queries with dependency management
- Scheduling: Cron-based scheduling with support for triggers and manual runs
- Monitoring: Built-in alerts, retry policies, and run history for operational visibility
- Parameters: Dynamic parameterization for reusable pipeline templates
Workspace Administration
Enterprise workspace management includes:
- Identity management: SCIM provisioning, SSO with SAML/OIDC, and group-based access
- Network security: VPC peering, private link, IP access lists, and customer-managed keys
- Cluster policies: Restrict instance types, enforce auto-termination, and control costs
- Audit logging: Comprehensive audit logs shipped to your cloud storage for compliance
- Workspace folders: Organize assets with folder-level permissions and Git-backed repos