Data Engineer

Job DescriptionRed are currently partnered with an IT Consulting Giant supporting them in finding a Lead Big Data Engineer to work on a long term project. My client requires somebody who is available to be in the office 3 days a week and to join the project on a freelance basis. Details are below:• Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data factory, Databricks, Python and PySpark.• Experience with at least one of the following technologies: Scala/Java or Python• Deep understanding of the software development craft, with focus on cloud based (Azure), event driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalake houses using medallion architecture. Knowledge of DataMesh principles is added plus.• Ability to debug using tools Ganglia UI, expertise in Optimizing Spark Jobs• The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.• Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake• Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data factory, Informatica)• Experience in data modelling at least one database technology such as: - Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL) - NoSQL (MongoDB, Cassandra, Neo4J, CosmosDB, Gremlin)• Understanding of Information Security principles to ensure compliant handling and management of data• Ability to clearly communicate complex solutions.• Strong problem solving and analytical skills.• Working experience in Agile methodologies (SCRUM, Kanban)• A proven team player with strong leadership skills, who can work in a collaborative way across business units, teams and regions
#J-18808-Ljbffr