Lead – DataOps

About us:

We are proudly part of a privately owned global group of insurance and online comparison related businesses, with approximately 50 offices and 10,000 employees. In Australia, the Group includes trusted and well-known brands Budget Direct, Ozicare, Chasing Cars, iSelect and Compare the Market. In late 2024, iSelect and Compare the Market formed a new Aggregation business within the group and now support millions of Australians to compare and buy their personal finance and household products such as insurance, energy, and loans.

What you do:

The Lead - DataOps is responsible for designing, implementing, and maintaining highly available, scalable, and secure data pipelines and platforms. This role ensures efficient CI/CD processes, infrastructure automation, and monitoring to support the organization's data ecosystem.

The role requires a strong background in DataOps, DevOps, cloud data platforms, and infrastructure automation. They will work closely with Data Engineering, Security, IT Operations, and Analytics teams to streamline data workflows, enhance system reliability, and optimize cloud-based data environments.

The primary focus areas include infrastructure as code (IaC), CI/CD for data pipelines, monitoring & observability, cost optimization, and cloud security.

 Data Platform & Pipeline Automation

• Automate data pipeline deployments, monitoring, and scaling to ensure reliability and performance.
• Design, implement, and maintain CI/CD pipelines for data engineering workflows.
• Ownership of the platforms data operations — specifically, end-to-end troubleshooting of failed pipelines, adding more observability/robustness/scalability there whether related to "CI/CD" OR "data" workflows.
• Develop and enforce infrastructure-as-code (IaC) practices using tools like Terraform, AWS CloudFormation, or Azure Bicep.
• Work with Data Engineers and IT teams to optimize ETL/ELT processes and cloud data platforms.

Cloud Infrastructure & Security

• Oversee data infrastructure in cloud environments (AWS, Azure, GCP), ensuring high availability and security.  
• Implement role-based access control (RBAC), encryption, and network security best practices within data environments.
• Collaborate with Security and Compliance teams to ensure regulatory compliance
• Optimize cloud storage, compute, and networking costs while maintaining performance

Monitoring, Observability & Incident Response

• Implement monitoring solutions for real-time alerting, anomaly detection, and performance optimization.
• Utilize logging and observability tools such as Prometheus, Grafana, Datadog, Splunk, or CloudWatch.
• Establish incident response processes, including automated rollback strategies and disaster recovery plans.
• Work with Site Reliability Engineering (SRE) teams to ensure high system availability and minimal downtime.

DataOps Best Practices

• Advocate for DataOps methodologies to improve collaboration between Data Engineering, IT, and Analytics teams.
• Implement version control, automated testing, and release management for data pipelines.
• Continuously improve CI/CD workflows to enhance deployment frequency and reduce failures.
• Enforce best practices for data governance, lineage tracking, and auditability.

Collaboration & Stakeholder Engagement

• Work closely with Data Engineers, Analytics, Security, and IT Operations to streamline data infrastructure.
• Act as the subject matter expert (SME) on Data DevOps, providing training and mentorship to team members.
• Engage with business leaders and data users to align infrastructure improvements with business goals.

What you need:

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Cloud Computing, or a related field/relevant experience.
• Experience & Skills:
• 3-5+ years of experience in DataOps, Data Engineering, or Cloud Infrastructure roles.
• Hands-on experience with CI/CD tools (e.g., GitHub Actions, Jenkins, AWS).
• Proficiency in containerization and orchestration (Docker, Kubernetes, ECS, EKS, AKS) as plus.
• Experience implementing infrastructure as code (IaC) using Terraform
• Expertise in cloud data platforms (AWS, Azure, GCP) with services like AWS Lambda, S3, RDS, Redshift, Glue, EMR, Databricks, BigQuery, etc.
• Strong understanding of cloud networking, security policies, and identity management (IAM, VPC, KMS, Secrets Manager, etc.).
• Experience setting up monitoring and alerting using tools like Prometheus, Grafana, CloudWatch, Datadog etc.
• Ability to troubleshoot performance issues and optimize system reliability.
• Strong experience in Python, Bash, or PowerShell for automation and scripting.
• Ability to write automation scripts for infrastructure provisioning, deployments, and data pipeline management.
• Ability to lead a small team and mentor junior engineers and promote DataOps best practices across teams.
• Strong communication skills to work with both technical and non-technical stakeholders

 What's in it for you:

• Enjoy flexible work arrangements, including the option of one day working from home each week or a 9-day fortnight. 
• Enjoy additional leave days - ‘ME’ leave and ‘Volunteer Day’ leave. 
• Employee discounts on Car, Home, Travel insurance. 
• Vibrant and social community who have annual celebrations, family fun days and regular events. 

If you are someone who fosters team spirit, has resilience and aspires to continue learning and developing their skills to drive commercial outcomes, you will have tremendous success in helping us achieve our progressive growth agendas. High energy levels and a love of having fun are also important to working in this energetic business.

Interested? We’d love to hear from you! 

At Compare the Market and iSelect and we’re proud to be an Equal Opportunity Employer and we embrace diversity.  

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.  

Compare the Market and iSelect are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside the Auto & General Holdings Group without your consent.