Job at a glance
We’re hiring a talented Data Engineer and Big Data enthusiast to work in our platform to help ensure that our data quality is flawless. As a company, we have millions of new data points every day that come into our system. You will be working with a passionate team of engineers to solve challenging problems and ensure that we can deliver the best data to our customers, on-time. You will be using the latest cloud data warehouse technology to build robust and reliable data pipelines.
Duties/Responsibilities Include: Develop expertise in the different upstream data stores and systems across Numerator. Design, develop and maintain data integration pipelines for Numerators growing data sets and product offerings. Build testing and QA plans for data pipelines. Build data validation testing frameworks to ensure high data quality and integrity. Write and maintain documentation on data pipelines and schemas Requirements : BS or MS in Computer Science or related field of study 3 + years of experience in the data warehouse space Expert in SQL, including advanced analytical queries Proficiency in Python (data structures, algorithms, object oriented programming, using API’s) Experience working with a cloud data warehouse (Redshift, Snowflake, Vertica) Experience with a data pipeline scheduling framework (Airflow) Experience with schema design and data modeling Exceptional candidates will have: Amazon Web Services (EC2, DMS, RDS) experience Terraform and/or ansible (or similar) for infrastructure deployment Airflow -- Experience building and monitoring DAGs, developing custom operators, using script templating solutions.
Experience supporting production systems in an on-call environment