Big Data Engineer - Global Consumer Goods Brand

Big Data Engineer required for one of the worlds leading Consumer Brands who are embracing Big Data technologies to enable data-driven decisions.

This hire will be a driving force for our clients Digital arm. You will be working on development projects related to consumer behavior, commerce, and web analytics.

Responsibilities:

Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem. Ability to design and implement end to end solution.

Build utilities, user defined functions, and frameworks to better enable data flow patterns.

Research, evaluate and utilize new technologies/tools/frameworks centered around Hadoop and other elements in the Big Data space.

Define and build data acquisitions and consumption strategies

Build and incorporate automated unit tests, participate in integration testing efforts.

Work with teams to resolving operational & performance issues

Work with architecture/engineering leads and other teams to ensure quality solutions are implements, and engineering best practices are defined and adhered to.

Qualification:

MS/BS degree in a computer science field or related discipline

6+ years' experience in large-scale software development

1+ year experience in Hadoop or big data technologies.

Strong Java programming, Python, Shell Scripting, and SQL

Strong development skills around Hadoop, Spark, Hive, and Pig

Good understanding of file formats including JSON, Parquet, Avro, and others

Experience with performance/scalability tuning, algorithms and computational complexity

Ability to understand relational database schemas

Proven ability to work cross functional teams to deliver appropriate resolution

Experience with AWS components and services, particularly, EMR, S3, and Lambda

Automated testing, Continuous Integration/Continuous Delivery

NICE TO HAVE:

Experience with open source NOSQL technologies such as HBase, DynamoDB, Cassandra

Experience with messaging & complex event processing systems such as Kafka and Storm

Scala

Machine learning frameworks

Statistical analysis with Python, R or similar

Experience (at least familiarity) with data warehousing, dimensional modelling and ETL development

This is an excellent long-term assignment for a talented Data Engineer. Please apply to (see below) or call Adam.