Senior Databricks Data Engineer
- Location: Toronto (Hybrid — minimum 2 days per week onsite at the client’s downtown Toronto office)
- Contract Length: 2 Months
We at Raise are hiring a Senior Databricks Data Engineer for one of our top clients. After establishing themselves as an industry leader, they’re now expanding their team to meet rising demand. We’re hiring right now; if you’re interested, apply below for your chance to join a great place to work.
Key Responsibilities
Databricks Engineering and Migration
- Migrate and refactor existing ETL/ELT pipelines, scripts and processing workloads into Databricks.
- Develop production-quality data pipelines using Python, PySpark and SQL.
- Translate existing processing and business logic into scalable Databricks-native implementations.
- Develop and maintain ingestion, transformation and metric-calculation logic across multiple enterprise data sources.
- Modernize existing workflow and orchestration processes, including scheduled jobs and Airflow/DAG-based processing.
- Implement reusable engineering patterns to accelerate migration of additional workloads.
Data Integration
- Integrate data from enterprise applications, APIs, databases, files and other source systems.
- Work with source-system SMEs to understand existing data structures, interfaces and processing logic.
- Implement source-to-target mappings based on approved requirements and target data models.
- Support integration with enterprise platforms such as ServiceNow and cybersecurity data sources.
- Troubleshoot connectivity, ingestion, schema and data-quality issues.
Data Quality, Testing and Reconciliation
- Develop automated developer tests and data-validation routines.
- Validate source-to-target transformations and data movement through bronze, silver and gold processing layers where applicable.
- Perform reconciliation between legacy processing outputs and Databricks-generated results.
- Identify, document and resolve data defects, processing exceptions and reconciliation discrepancies.
- Produce clear validation evidence to support business and technical acceptance.
Engineering Quality and DevOps
- Develop maintainable, reusable and well-documented production code.
- Follow enterprise coding, security, architecture and release-management standards.
- Use Git-based repositories and established CI/CD and deployment processes.
- Participate in peer reviews and contribute to improving engineering standards and reusable patterns.
- Support code check-in, deployment readiness, defect remediation and production-transition activities.
- Monitor and troubleshoot pipeline execution and processing failures.
Agile Delivery and Collaboration
- Work within an Agile delivery team executing against two-week sprint commitments.
- Participate in sprint planning, backlog refinement, technical design sessions, defect triage and sprint reviews.
- Estimate engineering effort and identify technical dependencies, blockers and risks.
- Work closely with architects and technical leads to ensure alignment with enterprise architecture and Databricks standards.
- Collaborate directly with client engineering teams and transfer knowledge throughout the engagement.
- Produce technical documentation, source-to-target mappings, implementation notes and handover materials.
Required Skills and Experience
- 7+ years of experience in data engineering, software engineering or data-platform development.
- Strong hands-on experience building production workloads on Databricks.
- Advanced development skills in:
- Python
- PySpark / Apache Spark
- SQL
- Experience designing and implementing ETL/ELT and large-scale data-processing pipelines.
- Experience migrating legacy data pipelines or processing applications to modern cloud data platforms.
- Strong understanding of data ingestion, transformation, modelling, source-to-target mapping, data quality and reconciliation.
- Experience with workflow orchestration technologies such as Apache Airflow or equivalent.
- Experience integrating data through APIs, databases, files and enterprise applications.
- Experience with Git, CI/CD and enterprise software-development practices.
- Strong troubleshooting skills across data, code, integration and environment issues.
- Ability to understand existing code and processing logic and efficiently refactor it into a modern architecture.
- Experience working within structured Agile/Scrum delivery environments.
- Strong written and verbal communication skills with the ability to work directly with client technical teams.
- Must be based in the Greater Toronto Area and available to work onsite in downtown Toronto a minimum of two days per week.
Preferred Qualifications
- Databricks certification such as Databricks Certified Data Engineer Associate or Professional.
- Experience delivering data solutions within financial services, banking or another highly regulated industry.
- Experience with cybersecurity, IT controls, compliance or operational-risk datasets.
- Experience integrating platforms such as ServiceNow, CrowdStrike, BeyondTrust, FireEye, CyberArk, Tenable or similar enterprise security platforms.
- Experience with cloud platforms such as Azure or AWS.
- Experience modernizing Airflow DAGs or other legacy orchestration frameworks.
- Familiarity with enterprise data-governance, metadata, lineage and auditability requirements.
- Experience working in environments with strict security, architecture and release-governance processes.
Looking for meaningful work? We can help!
Raise is an established hiring firm with over 65 years of experience. We believe strongly in making the world a better place through work, which is why we’re a certified B Corporation and donate 10% of our profits to charity.
We strive to build teams that reflect the diversity of the communities we work in. We encourage all qualified applicants to apply, including people from traditionally underrepresented groups such as women, visible minorities, Indigenous peoples, people identifying as LGBTQ2SI, veterans, and people with visible/nonvisible disabilities.
We have a dedicated webpage for accommodations where you can learn more about what we offer and request accommodations: https://raise.jobs/accommodations/
In order to submit candidates for roles, our clients will sometimes require personal information to confirm the identity of applicants and their legal status to work. Raise will never ask you for personal or banking information unless you have been selected for a job. If you are ever unsure about the legitimacy of this or another job posting by Raise (or have any other questions), please contact us at +1 800-567-9675 or hello@raiserecruiting.com