Skip to main content

Sr Engineer, Data Processing/DevOps

New York, New York
Apply

Job Description:


What makes Gartner Research & Advisory a GREAT fit for you? When you join Gartner, you’ll be part of a leading-edge team that values expert insights, bold ideas and intellectual courage. You’ll deliver must-have research through compelling client interactions, and contribute to our double-digit growth by providing clients the unbiased insight and advice they need to make the right decisions every day. Through constant learning, discovery and collaboration, you’ll help clients deliver on their mission-critical priorities, grow your career and increase your industry impact. We value hard workers — and reward you with unlimited opportunity. If you’re looking to explore what’s next in technology and business, Gartner is looking for you.

We are looking for a Senior Engineer who specializes in Data Engineering and DevOps to take us to a new level of data processing. In this role, you will join a fast-paced, challenging environment and work on data pipelines that powers Gartner for Marketers analytics and machine learning teams, making a large impact on our architecture and the organization. The main challenge for this role is to architect and enhance data processing pipelines that integrate with various machine learning models. This is a foundational role that will establish new capability and practices for our team and is a great opportunity to apply your experience and grow in new directions.

Here is a sample of projects you will be working on:

  • Pipelines that process data streams from scraping half a million pages a day.

  • Pipelines that ingest and process data from our data partners.

  • Integrating machine learning models into data pipelines.

  • Application to manage training data for various ML models

  • Operationalizing development and deployment workflows

What we’d like to see in you:

  • Expertise in Python with multiple production deployments.

  • Experience with Spark and opinions on use-cases, dev workflow, testing, performance.

  • Familiarity with Airflow.

  • Strong SQL knowledge.

  • Experience with AWS in production settings.

  • Comfortable with complexity and drive to make things simpler.

  • Passion for learning and self-improvement, being receptive to feedback.

  • Ability to quickly learn a new vendor API, new tool or a technology concept.

  • Experience with modern web and application development practices: continuous integration, scrum or agile development, distributed version control systems, test-driven development, automated deployment and provisioning.

  • Ability to support product team on planning, scope breakdown, task estimation.

  • Strong communication skills, ability to self-motivate and self-direct.

  • Active Github, Bitbucket or StackOverflow profile.

About us:

We believe in tackling difficult questions/problems as a team in order to get to the best solution and we don’t hesitate to change our approach when a better idea is on the table.

We are a cross-functional team of engineers (front-end, back-end, QA, and data), product managers, and data scientists. We practice code reviews, unit and integration testing, continuous integration and delivery, agile 2 week sprints, and take pride in seeing our work through from conception to deployment and to the next better iteration.

Our Stack:

  • Our primary languages are Python and modern JavaScript (React, ES6, Redux, Babel/Webpack, SASS, Express, Node).

  • Primary data stores include S3, Postgres, Redshift, Athena, Snowflake, SQL is our lingua franca.

  • AWS (EC2, Lambda, EKS, Kinesis, Fargate, Cloudwatch, DynamoDB, etc).

  • Docker and Kubernetes deployments triggered by GitLab CI.

  • Airflow for workflow management.

  • dbt and more for ETL and data pipelines.

Job Requisition ID:29699

By submitting your information and application, you confirm that you have read and agree to the country or regional recruitment notice linked below applicable to your place of residence.

Gartner Applicant Privacy Link: https://jobs.gartner.com/applicant-privacy-policy

For efficient navigation through the application, please only use the back button within the application, not the back arrow within your browser.

Apply

Similar Jobs

View All of Our Available Opportunities

Sign Up for Job Alerts

Interested InSearch for a category and select one from the list of suggestions. Search for a location and select one from the list of suggestions. Finally, click “Add” to create your job alert.