Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : AWS Glue Good to have skills : NA Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary As a Senior AWS Glue Data Engineer / AWS Glue Architect, you will design, develop, and implement enterprise-scale ETL and data integration solutions using AWS Glue and AWS cloud-native data services.
The role requires strong hands-on expertise in AWS Glue ETL development, PySpark programming, AWS Lambda, IAM security, VPC networking, and serverless data engineering architectures. The engineer will build scalable data pipelines, metadata-driven frameworks, and cloud-native integration solutions using AWS Glue, Glue Data Catalog, Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and other AWS services.
The role involves data ingestion, transformation, performance optimization, security implementation, monitoring, and production support following AWS best practices. Roles & Responsibilities 1. Design, develop, and maintain enterprise-scale ETL pipelines using AWS Glue. 2. Develop and optimize AWS Glue Spark jobs using PySpark for large-scale data processing. 3. Create and manage AWS Glue components including Jobs, Crawlers, Data Catalog, Workflows, Triggers, Connections, and Job Bookmarks.
4. Build metadata-driven ETL frameworks using AWS Glue Data Catalog and configuration-based processing approaches. 5. Develop data ingestion and transformation pipelines from multiple sources including databases, APIs, files, and cloud platforms. 6. Integrate AWS Glue with AWS services including Lambda, S3, Athena, Redshift, Step Functions, EventBridge, and SQS. 7. Develop AWS Lambda functions for data processing, automation, event-driven workflows, and Glue job orchestration.
8. Configure and manage AWS IAM Roles, Policies, Permissions, and Security Controls for Glue jobs and AWS services. 9. Design and configure AWS VPC components including VPC endpoints, subnets, security groups, route tables, and network connectivity for AWS Glue workloads. 10. Configure AWS Glue connections for databases using JDBC connectivity. 11. Implement secure data integration solutions using JDBC, REST APIs, SFTP, and event-driven architectures.
12. Perform AWS Glue job tuning using Spark optimization techniques, partitioning, compression, caching, and performance best practices. 13. Implement data quality checks, validation frameworks, error handling, and reconciliation processes. 14. Monitor and troubleshoot AWS Glue jobs using Amazon CloudWatch, CloudWatch Logs, and AWS monitoring services. 15. Implement CI/CD pipelines for AWS Glue deployments using Git, Jenkins, GitHub Actions, AWS CodePipeline, or AWS CodeBuild.
16. Develop Infrastructure as Code (IaC) solutions using AWS CloudFormation or AWS CDK. 17. Support production incidents, troubleshooting, performance issues, and operational improvements. 18. Collaborate with data architects, data engineers, application teams, and business stakeholders to deliver AWS data solutions. 19. Mentor team members on AWS Glue development standards, PySpark optimization, and AWS data engineering best practices.
Professional & Technical Skills 1. Strong hands-on experience with AWS Glue for ETL development, data integration, and data pipeline implementation. 2. Strong programming skills in Python, PySpark, and SQL for data transformation and processing. 3. Experience with AWS services including Lambda, S3, Athena, Redshift, Step Functions, and EventBridge. 4. Strong knowledge of AWS IAM Roles, Policies, Security Configuration, and VPC setup for secure data solutions.
5. Experience designing and optimizing AWS Glue ETL pipelines, Spark jobs, and data processing wor