Sr Engineer, Data Processing/DevOps
We are looking for a Senior Engineer who specializes in Data Engineering and DevOps to take us to a new level of data processing. In this role, you will join a fast-paced, challenging environment and work on data pipelines that powers Gartner for Marketers analytics and machine learning teams, making a large impact on our architecture and the organization. The main challenge for this role is to architect and enhance data processing pipelines that integrate with various machine learning models. This is a foundational role that will establish new capability and practices for our team and is a great opportunity to apply your experience and grow in new directions.
Here is a sample of projects you will be working on:
Pipelines that process data streams from scraping half a million pages a day.
Pipelines that ingest and process data from our data partners.
Integrating machine learning models into data pipelines.
Application to manage training data for various ML models
Operationalizing development and deployment workflows
What we’d like to see in you:
Expertise in Python with multiple production deployments.
Experience with Spark and opinions on use-cases, dev workflow, testing, performance.
Familiarity with Airflow.
Strong SQL knowledge.
Experience with AWS in production settings.
Comfortable with complexity and drive to make things simpler.
Passion for learning and self-improvement, being receptive to feedback.
Ability to quickly learn a new vendor API, new tool or a technology concept.
Experience with modern web and application development practices: continuous integration, scrum or agile development, distributed version control systems, test-driven development, automated deployment and provisioning.
Ability to support product team on planning, scope breakdown, task estimation.
Strong communication skills, ability to self-motivate and self-direct.
Active Github, Bitbucket or StackOverflow profile.
We believe in tackling difficult questions/problems as a team in order to get to the best solution and we don’t hesitate to change our approach when a better idea is on the table.
We are a cross-functional team of engineers (front-end, back-end, QA, and data), product managers, and data scientists. We practice code reviews, unit and integration testing, continuous integration and delivery, agile 2 week sprints, and take pride in seeing our work through from conception to deployment and to the next better iteration.
Primary data stores include S3, Postgres, Redshift, Athena, Snowflake, SQL is our lingua franca.
AWS (EC2, Lambda, EKS, Kinesis, Fargate, Cloudwatch, DynamoDB, etc).
Docker and Kubernetes deployments triggered by GitLab CI.
Airflow for workflow management.
dbt and more for ETL and data pipelines.
Job Requisition ID:29699
By submitting your information and application, you confirm that you have read and agree to the country or regional recruitment notice linked below applicable to your place of residence.
Gartner Applicant Privacy Link: https://jobs.gartner.com/applicant-privacy-policy
For efficient navigation through the application, please only use the back button within the application, not the back arrow within your browser.