Senior Data Quality Engineer

SENIOR DATA QUALITY ENGINEER

My client is embracing Big Data technologies to enable data-driven decisions. We're looking to expand our Data Engineering team to keep pace. The Enterprise Information Management team brings broad experience in the practices of Master Data Management (MDM), Data Quality, Metadata Management, Data Stewardship, and Data Governance.

Your primary accountability will be to maintain and mature our Data Quality Services platform. This includes Data Quality, Data Services (Virtualization) and Master Data Management. You will collaborate with data stewards, information and data architects, subject matter experts, BSAs, and other IT teams to establish the technical vision and analyze tradeoffs between usability, supportability and performance needs.

RESPONSIBILITIES:

- Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem. Ability to design and implement end to end solution.
- Build utilities, user defined functions, and frameworks to better enable data flow patterns.
- Research, evaluate and utilize new technologies/tools/frameworks centered around Hadoop and other elements in the Big Data space, be a strategic thinker
- Build and incorporate automated unit tests, participate in integration testing efforts.
- Work with teams to resolving operational & performance issues, be a effective communicator
- Work with architecture/engineering leads and other teams to ensure quality solutions are implements, and engineering best practices are defined and adhered to.
- focus on Data Analysis
- be Results oriented

QUALIFICATION:

- MS/BS degree in a computer science field or related discipline
- 5+ years programming/analysis/design/administration
- 1+ year experience in Hadoop or big data technologies.
- Strong Java programming, Python, Shell Scripting, and SQL
- Strong development skills around Hadoop, Spark, Hive, and Pig
- Good understanding of file formats including JSON, Parquet, Avro, and others
- Experience with performance/scalability tuning, algorithms and computational complexity
- Ability to understand relational database schemas
- Proven ability to work cross functional teams to deliver appropriate resolution
- Experience with AWS components and services, particularly, EMR, S3 AND SPLUNK
- Automated testing, Continuous Integration/Continuous Delivery
- Comfortable working in a fast-paced, results-oriented environment
- Ability to work on multiple projects at any given time
- Ability to adapt to changing needs, adjusted processes and ambiguity

NICE TO HAVE:

- Experience with Informatica Data Services platform
- Experience with Alteryx
- Statistical analysis with Python, R or similar