Data Engineer - Databricks & PySpark

Sydney, NSW
Permanent
Full-time

23 days ago

I am working with an organization that's ready to take a product from proof-of-concept (PoC) to a fully operational, production-grade managed service. This is a hands-on role for a Data Engineer who thrives in fast-paced environments and knows how to turn prototypes into scalable, reliable solutions.What You'll Be Doing

Take ownership of an existing PoC and lead its evolution into a production-ready service.
Build, refine, and scale data pipelines and outputs to support ongoing monitoring and management.
Collaborate closely with developers and stakeholders to ensure smooth delivery within a tight 6-month timeline.
Apply best practices in CI/CD and production deployment to ensure stability and maintainability.

What We're Looking ForMust-Have Skills

Python: Strong to advanced proficiency, especially working with PySpark.
Databricks: Solid experience with Unity Catalog and Delta Live Tables (intermediate to advanced).
RESTful APIs: Advanced experience managing client credential requests using Python.
CI/CD: Deep understanding of best practices; confident in setting up and maintaining pipelines.
End-to-End Development: Experience pushing solutions into production environments.

Nice-to-Have Skills

SQL: Useful for data manipulation and validation.
Microsoft Graph API: Great if you've worked with it, but general RESTful API experience is fine

if this is an opportunity that you are interested in please reach out and we can hop on a call!Sign up for job alertsIf you are a human, ignore this field Create alertYou agree to ourShare this jobData Engineer - Databricks & PySparkSydney CBD Posted 25 minutes agoQuick LinksConnectWe are committed to Building Better Futures for the organisations we work for, the professionals we help and our own employees.

Ambition

Apply Now