Principal Databricks Data Engineer

Principal Databricks Data Engineer​ Experience Required: 12–18 Years Key Requirements • 12–18 years of overall Data Engineering experience • 8+ years of experience with enterprise Data Warehouse and Data Lake platforms • 5+ years of hands-on experience with Databricks and Apache Spark at scale • Strong experience modernizing legacy Cloudera platforms including: • CDH/CDP • Hive • HBase • Impala • Spark • Modernize Cloudera platforms to Databricks Lakehouse architecture • Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake • Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines • Ensure data reconciliation, audit integrity, and consistency during migration • Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures • Implement layered data architecture including: • Raw / Landing Layer • Curated / Conformed Layer • Semantic / Consumption Layer • Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures • Strong experience with finance and risk data models including: • General Ledger • Sub-ledger • Financial Hierarchies • Credit Risk • Liquidity Risk • Market Risk • Enable reporting capabilities including: • Aggregation • Drill-down • Drill-back • Build and manage semantic and consumption layers for BI, reporting, and analytics • Define business metrics, dimensions, hierarchies, and KPIs • Experience with: • Databricks SQL • Delta Tables • dbt or similar frameworks • Develop and optimize large-scale data pipelines using: • PySpark • Spark SQL • Delta Lake • Implement Medallion Architecture including: • Bronze Layer • Silver Layer • Gold Layer • Optimize workloads using: • Z-ORDER • OPTIMIZE • Caching • Cluster Configuration Tuning • Implement: • Data Governance • Data Quality Frameworks • Reconciliation Controls • Exception Handling • Establish data lineage and metadata management • Ensure data security, access control, and compliance standards • Experience with AWS or Azure cloud platforms • Experience with CI/CD pipelines using: • Git • Terraform • Jenkins • Azure DevOps • Familiarity with: • Apache Airflow • Databricks Workflows • Experience with dbt is an advantage • Act as a technical authority and lead enterprise architecture decisions • Mentor senior engineers and establish engineering standards • Collaborate with finance, risk, analytics, and governance stakeholders • Translate complex data structures into business-ready insights Nice-to-Have Skills • Experience in BFSI, Capital Markets, or Regulatory Reporting • Exposure to: • SAP Finance • Oracle Financials • SAP S/4HANA • Experience supporting AI/ML workloads • Databricks and Cloud Certifications Key Responsibilities • Lead Cloudera to Databricks transformation initiatives • Shape enterprise finance and risk data platforms • Support regulatory, management, and analytical reporting systems Essential Skills • Databricks • Apache Spark • PySpark • Spark SQL • Delta Lake • Lakehouse Architecture • Cloudera (CDH/CDP) • Hive • HBase • Impala • Data Warehouse & Data Lake • Medallion Architecture • Databricks SQL • dbt • AWS / Azure • Airflow / Databricks Workflows • CI/CD • Terraform • Jenkins • Git • Finance & Risk Data Models • Enterprise Data Architecture