- Implement new data ingestion jobs utilizing Terraform or other infrastructure as code tools to automate workflows and improve overall data processing efficiency.
- Optimize existing data pipelines for performance, scalability, and reliability using best practices in distributed computing.
- Contribute to the continuous integration and delivery of high-quality software by collaborating with team members on Agile methodologies.
- Document and maintain technical documentation related to data engineering processes, tools, and infrastructure.
- Collaborate with DevOps engineers to ensure that CI/CD pipelines are functioning effectively and efficiently, and that deployment processes are well-defined and efficient.
- Provide support for critical production systems as needed.
Qualifications:
- We are looking for a candidate with a bachelor’s degree in a technology field and 4+ years of experience in a Data Engineer role.
- 4+ years of experience working with PySpark or other distributed computing frameworks (Python preferred).
- Proficiency in Python and Scala programming languages for data engineering tasks.
