Data Engineer

We are seeking a skilled Databricks Data Engineer to support the development of a data product and analytics-driven culture. You will work closely with data experts, DBAs, and data consumers/scientists to establish and maintain a robust Databricks environment. Your expertise will play a critical role in shaping and ensuring the success of this new platform and strategy.

Responsibilities:

• Build and maintain scalable data architectures and data lakes on the Databricks platform.
• Implement monitoring and alerting systems to proactively identify and address data processing issues.
• Stay updated with the latest advancements and best practices in Databricks and big data technologies.
• Collaborate with data engineers to ensure scalability, reliability, and security of Databricks clusters.
• Design and configure data pipelines using Databricks to ingest, transform, and load data from various sources.
• Develop and optimize SQL queries and Spark jobs for efficient data processing and analysis.
• Troubleshoot and resolve performance issues and data quality problems in Databricks environments.
• Provide technical guidance and support to data scientists and analysts using the Databricks platform.
• Document solutions, architectures, and processes related to Databricks implementations.

Requirements

• Strong knowledge of big data technologies and concepts, including Apache Spark and distributed computing.
• Experience with cloud platforms, preferably AWS Databricks.
• Deep understanding of data integration, data modeling, and data warehousing.
• Expertise in data ingestion, ETL processes, and both streaming and batch data.
• Proven experience in building and optimizing data pipelines using Databricks Notebooks, Databricks Jobs, and Databricks Delta Lake.
• Proficiency in SQL with expertise in database concepts, query optimization, and performance tuning.
• Programming skills in Python, Scala, or SQL.
• Knowledge of exposing data for dashboarding and data analytics.
• Familiarity with data governance, data security, and compliance requirements in a big data environment.
• Strong problem-solving skills for troubleshooting and resolving data processing and performance issues.
• Installation and configuration of Databricks clusters and associated components (network connectivity, access controls, etc.) is an advantage.

Your expertise in Databricks will drive impactful decisions and innovations in agricultural research and development.