ABOUT THE ROLE

We’re looking for a hands-on Data Platform Engineer engineer who thrives at the intersection of streaming data, scalable ETL, and robust infrastructure. In this role, you’ll architect, build, and optimize real-time ingestion services and PySpark pipelines on Azure Databricks, ensuring low-latency, reliable, and secure data movement from diverse clients and systems. You’ll also own containerized APIs, CI/CD workflows, and IaC for reproducible environments.

WHAT YOU’LL DO

• Design real-time client integrations for ingestion, including hardware–software             wrappers and edge connectivity.
• Build low latency streaming pipelines from clients and services using Event             Hubs/Kafka, Databricks Autoloader, and Delta Lake best practices.
• Implement large file handling strategies (chunking, parallel processing), sync jobs,       and storage lifecycle management across Azure Storage.
• Solve complex software and data engineering problems, implementing reliable        mechanisms, logging, and error-handling logic.
• Configure and leverage Unity Catalog for advanced search, metadata indexing,      lineage, and access control.
• Develop and operate Azure Databricks/PySpark ETL pipelines for batch and streaming workloads.
• Orchestrate jobs with Azure Data Factory (ADF), integrating triggers, dependencies, and alerting.
• Containerize and deploy APIs (e.g., FastAPI/Flask) with CI/CD pipelines (GitHub Actions/Azure DevOps) and robust release strategies.
• Use IaC (ARM/Terraform) for reproducible infrastructure across environments.
• Embed observability (logging, metrics, tracing) and performance tuning for streaming and batch jobs.
• Collaborate with data platform, product, and security teams to meet SLAs and compliance requirements.

WHAT YOU WILL BRING

• Proven experience with real-time client ingestion and hardware–software wrapper design.
• Hands-on expertise in low-latency streaming using Event Hubs/Kafka and Databricks Autoloader.
• Strong skills in large file processing, parallelization, synchronization, and storage management in Azure (Blob/Data Lake).
• Solid software engineering fundamentals: structured logging, retry/backoff, circuit breakers, idempotency, and stateful/stateless designs.
• Practical experience with Unity Catalog (advanced search, metadata indexing, data lineage, policies).
• Containerization & CI/CD experience (Docker, GitHub Actions/Azure DevOps), deploying APIs to Azure services (AKS, App Service, Container Apps).
• Working knowledge of IaC: Terraform and/or ARM templates for environment provisioning.

WHO ARE WE?

We love doing things together and strive for a good balance between work and leisure. This could be after-work hangouts, events, sports activities, ski trips, or inspirational talks. We've been on many fun trips together, which have been fantastic, and something we want to continue doing.

Skills development has always been important to us, and in addition to growth opportunities within your consulting assignments and various technical trainings, we also have internal programs for personal development.

WE ARE PART OF THE SIGMA GROUP

As part of the Sigma Group, we offer the best of both worlds: stability and entrepreneurial spirit. Sigma consists of several companies within different technical competence areas and currently has over 5,000 employees and operations in many countries. This makes us Sweden’s largest privately-owned consulting group.