UFCEQB 15 M

UFCEQB-15-M Scaling and Orchestration

Overview

This module is designed to provide a foundation in modern data engineering, from building solutions capable of handling peta-scale data loads, to designing deployment pipelines that allow data science deliverables to be incorporated into industrial technology platforms

This module aims to introduce key concepts and design considerations for operational data architectures, by looking at examples and the range of demands on modern services.

By providing hands-on experience with representative tools across the range of data engineering activities, students should gain confidence in what design features and maintenance functions look like in practice.

Objectives

1. Investigate data engineering problems and design robust, future-proof and compliant solutions.
2. Demonstrate in-depth knowledge of how to seamlessly scale operations from small standalone data applications into complex, high throughput and storage-demanding services.
3. Show competence in the building of modular pipelines and distributed architectures enabling the deployment, integration and monitoring of data products in production environments.

Curriculum

Topics may include, but are not limited to:

• Characteristics of big data and data engineering challenges
• Architectural principles for modularisation and scalability
• Balancing costs, compliance, security and reliability assurance
• Cloud service platforms, functions and features
• Edge computing and federated architectures
• Machine learning model deployment and evaluation
• Large Language and image model (LLM) deployments and Artificial Intelligence (AI) agent architectures
• Microservices and Application Programming Interface (API) integration
• Containerisation and container deployment and maintenance
• Continuous integration and deployment
• Comparison and use of contemporary tools for data engineering, operations and monitoring

Assessment

Presentation (100%)