Big Data Engineer

Responsibilities:

• Design, develop, install, run, test, and manage data management and processing systems.
• Build scalable, robust, and fault-tolerant big data systems.
• Develop and maintain ETL (Extract-Transform-Load) processes.
• Develop system architecture according to organizational business needs.
• Explore and identify new ways to leverage existing data.
• Propose improvements for data quality, reliability, and overall system performance.
• Create complete integration solutions using a wide range of big data tools and programming languages.
• Implement fault recovery and data recovery technologies.

Requirements:

• Experience with big data tools such as HDFS, Hive, NiFi, Spark, Kafka, and Trino.
• Experience with programming languages such as Python, Scala, Java, and proficiency in at least one of them.
• Experience with SQL queries.
• Experience working with relational databases such as Oracle.
• Familiarity with working with Apache tools such as Flink and Airflow.
• Familiarity with NoSQL and non-relational databases.
• Familiarity with Unix-based environments.
• Familiarity with software development tools such as SonarQube, Jenkins, and Git.
• Familiarity and interest in statistical and machine learning concepts and tools such as Spark MLlib, Scikit-learn, and TensorFlow.