Intermediate Data Engineer

REQUIREMENTS Minimum education (essential): Diploma in Computer Science, Information Systems, Data Engineering or a related field. Minimum Education (desirable) Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field. Relevant Microsoft Fabric and/or Azure data certification. Minimum applicable experience (years): 4+ years of practical data engineering experience, including strong recent hands-on experience with Microsoft Fabric.

Required nature of experience: Strong hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2. Strong SQL Server and T-SQL capability, including complex query development, schema design, indexing and performance optimisation. Practical experience developing, maintaining and supporting production ETL/ELT pipelines.

Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources. Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices. Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution. Experience with troubleshooting pipeline failures, data quality issues and performance bottlenecks, including the ability to restore service efficiently.

Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools. Experience supporting Power BI and other downstream analytical or reporting requirements. Demonstrated ability to take ownership of an existing technical environment with limited hand-holding. Strong documentation, communication and stakeholder engagement skills. Skills and Knowledge (essential): Microsoft Fabric: OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.

SQL Server / T-SQL Python / PySpark REST/SOAP APIs and structured/unstructured data ingestion. ETL/ELT, incremental loading, orchestration and scheduling. Dimensional modelling, medallion architecture, schema evolution and data warehousing. Power BI integration and understanding of downstream analytical requirements. Git / Azure DevOps, CI/CD and environment deployment practices. Monitoring, data quality, performance optimisation, security and operational support.

Other: Proficient in Afrikaans and English. Own transport and valid driver’s license. KEY PERFORMANCE AREAS AND OBJECTIVES Fabric Data Engineering and Pipeline Development Take ownership of the existing Microsoft Fabric data engineering environment and become productive quickly following handover. Design, develop, maintain and orchestrate reliable batch and near-real-time data ingestion pipelines using appropriate Microsoft Fabric capabilities.

Extract and ingest data from structured and unstructured sources, including REST APIs, SOAP APIs, databases and flat files. Develop robust data transformation logic using SQL, Python/PySpark, Fabric notebooks and Dataflows Gen2, as appropriate. Implement incremental loading, retry mechanisms, logging, monitoring and alerting to support data integrity and pipeline reliability. Troubleshoot and resolve pipeline failures and data processing issues efficiently.

Optimise data pipelines and processing workloads for performance, scalability and cost-effectiveness. Data Architecture and Platform Management Design, manage and evolve scalable data architectures using Microsoft Fabric, OneLake, Lakehouse, Warehouse and SQL Server. Maintain appropriate data-layering and medallion architecture principles, where applicable, with clear movement from raw to curated data.

Develop and maintain robust schema designs, indexes, partitioning and query strategies to support analytical and operational workloads. Manage schema evolution and version control to maintain consistency and minimise disruption to downstream consumers. Maintain metadata, data d