Big Data Engineer Top Client

FOR ONE OF OUR MAJOR CLIENT COMPANIES WE ARE LOOKING FOR A BIG DATA SPECIALIST WHO IS PRIMARILY EXPERIENCED WITH, JAVA, AWS, HADOOP AND NOSQL.

RESPONSIBILITIES:

- Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem. Ability to design and implement end-to-end solution.
- Build utilities, user defined functions, and frameworks to better enable data flow patterns.
- Research, evaluate and utilize new technologies/tools/frameworks centered on Hadoop and other elements in the Big Data space.
- Define and build data acquisitions and consumption strategies
- Build and incorporate automated unit tests, participate in integration testing efforts.
- Work with teams to resolving operational & performance issues
- Work with architecture/engineering leads and other teams to ensure quality solutions are implements, and engineering best practices are defined and adhered to.

QUALIFICATION:

- MS/BS degree in a computer science field or related discipline
- 6+ years' experience in large-scale software development
- 1+ year experience in Hadoop or big data technologies.
- Strong Java programming, Python, Shell Scripting, and SQL
- Strong development skills around Hadoop, Spark, Hive, and Pig
- Good understanding of file formats including JSON, Parquet, Avro, and others
- Experience with performance/scalability tuning, algorithms and computational complexity
- Ability to understand relational database schemas
- Proven ability to work cross functional teams to deliver appropriate resolution
- Experience with AWS components and services, particularly, EMR, S3, and Lambda
- Automated testing, Continuous Integration/Continuous Delivery

NICE TO HAVE:

- Experience with open source NOSQL technologies such as HBase, DynamoDB, Cassandra
- Experience with messaging & complex event processing systems such as Kafka and Storm
- Scala
- Machine learning frameworks
- Statistical analysis with Python, R or similar
- Experience (at least familiarity) with data warehousing, dimensional modelling and ETL development

WE WOULD BE VERY GLAD TO FIND CANDIDATES WHO KNOW THEIR WAYS AROUND THE ABOVE REQUIREMENTS..