Collaborate with BI Solution Architect and Business Analyst to convert requirements into technical specifications
Establish both batch and real-time data pipelines, primarily using Kafka-Spark Streaming for ETL processes
Transfer data from multiple sources to a centralized area for data scientists, engaging in discussions with data scientists to determine the preferred data format
Move modeling results to the target front-end application, consulting with Business Units and Business Analysts for specific requirements
Generally, load data into a database or designated storage; in rare cases, develop REST API services for front-end applications to access real-time data
Document source-target mappings within the created pipelines
Maintain and monitor data pipelines to ensure correct, available, and restricted access to authorized personnel
Adhere to the SCB SDLC process by collaborating with various teams, including Architects, IT security, Quality Assurance, and Change
Qualifications
Preferred academic background: BSc or MSc in computer science, software engineering, or information technology
Proficient in Spark, SQL, Python, or Scala
Experience in software development life cycles: coding, unit testing, using git or other SCM, CI/CD pipelines, and monitoring logs or service resource usage
Fundamental knowledge in computer engineering: systems, networks, and security
Strong communication and interpersonal skills to collaborate effectively with both technical teams (scientists/engineers) and non-technical business units
Preferred Qualifications
Prior work experience in roles such as ML Engineer, Data Engineer, Fullstack or Backend Software Engineer, Solution/Cloud Architect, or related positions.
Experience with Kubernetes, Kafka, Hadoop, Hive, and sFTP
Experience with cloud platforms, with a preference for Azure: Azure Storage, Azure Data Factory, and Databricks
Experience in developing data warehouse, big data, and business intelligence applications, particularly in a competitive service industry, preferably in banking, financial services, or telecom
