As a Data Engineer, you will develop, maintain, evaluate, and test big data solutions.You will be responsible to develop and deliver solutions that enable the collection, transformation, and management of information from one or more sources.
Responsibilities : Responsible to ingest data from files, streams, and databases; process the data with Python, Spark and ScalaDevelop programs in Spark and Scala as part of data cleaning and processingResponsible to develop distributed, high volume, high velocity multi-threaded event processing systemsDevelop efficient software code for multiple use cases leveraging Spark and Big Data technologies for various use cases built on the platformProvide high operational excellence guaranteeing high availability and platform stabilityImplement scalable solutions to meet the ever-increasing data volumes, using big data / cloud technologies.
You will also : Design and implement processes and jobs for managing sales and marketing data.Oversee and manage the data pipeline, from the retrieval for of the raw data to the transformation and aggregation of multiple data streams : Develop monitoring tools and reports at all pipeline levels,Ensure high data quality throughout the pipeline.
Develop production-ready software for data retrieval, storage, ETL, and analysis.Work in a distributed multiprocessing environment.
Onboard new data for both historical testing and real-time systems.Aid in the design of real-time data acquisition systems.
Perform data analysis to validate data quality.Collaborate closely with data scientists, the portfolio and research teams.