Senior Data Engineer

We are seeking a Senior Data Engineer to design, build, and deploy a Streamlit application within a Databricks environment, serving as a user-friendly interface for real-time data analysis and pipeline orchestration.

You’ll work at the intersection of data engineering, analytics, and automation, connecting a modern UI (Streamlit) with Databricks workflows, Azure Data Factory (ADF) pipelines, and machine learning experiments.
This is a highly cross-functional role that combines backend development, data orchestration, and cloud integration to empower business teams with intuitive, real-time data insights.

Responsibilities:

• Design and develop an interactive Streamlit application hosted in Databricks.

• Build input forms to capture parameters and trigger analytical processes.

• Ensure a seamless, intuitive experience for non-technical business users.

• Connect the Streamlit frontend with Databricks backend services and APIs.

• Trigger and monitor Azure Data Factory (ADF) pipelines based on user input.

• Handle real-time execution feedback and error reporting within the app.

• Query and process pipeline outputs from Databricks for visualization.

• Enable dynamic filtering, exploration, and visual representation of data.

• Collaborate with data scientists to manage ML experiments and results.

• Use MLflow or similar tools for tracking and monitoring model outputs.

• Work closely with data engineers, data scientists, and business stakeholders to refine requirements.

• Document architecture, workflows, and user instructions for long-term maintainability.

Qualifications:

• Proven experience building Streamlit applications for production use.

• Strong knowledge of Databricks (PySpark, SQL, Delta Lake).

• Hands-on experience with Azure Data Factory (ADF) pipeline orchestration.

• Solid background in Python and API/service integrations.

• Understanding of machine learning processes and ability to support ML experiments.

• Familiarity with cloud data ecosystems (Azure preferred).

• Experience with CI/CD pipelines for Databricks and Streamlit deployments.

• Knowledge of Azure DevOps or GitHub Actions for automation.

• Understanding of authentication and access control in Streamlit apps.

• Excellent problem-solving and communication skills.

• Remote start: you will begin working fully remotely for the first 3–4 months, with transition to a hybrid format afterward.

• Opportunity to influence architecture and product direction directly.

• Collaboration with a highly skilled, cross-functional global team.

• Exposure to a modern Azure + Databricks ecosystem and hands-on ML workflow management.

• Flexible working hours (core working hours: 8:00 p.m. – 11:00 p.m. (GMT+5). The remaining 5 working hours can be completed flexibly before 8:00 p.m. (GMT+5) )

• Competitive compensation package and long-term engagement potential.

• A culture of trust, respect, and camaraderie, focused on excellence and innovation.