Data Integration and Large-scale AnalysisModule description #41112 / Version 1
- 6 CP
- Written exam
- Graded
- Winter semester
- English
This module description is not valid in the current semester (Winter semester 2026/27).
Current version (v2)
| Module / Version | #41112 / #1 |
|---|---|
| Validity | Winter semester 2023/24 – Winter semester 2025/26 |
| Available language(s) | English |
| Credit points | 6 |
| Person responsible for the module | Böhm, Matthias |
| Grading | Graded |
| Exam format | Written exam |
| Language(s) of instruction | English |
Affiliation
| Faculty | Fakultät IV |
|---|---|
| Institute | Institut für Softwaretechnik und Theoretische Informatik |
| Academic chair | 34352900 FG Big Data Engineering |
Contact
| Office | Not specified |
|---|---|
| Contact person | Böhm, Matthias |
| E-mail address | matthias.boehm@tu-berlin.de |
| Website | Not specified |
Learning outcomes
Common data and system characteristics include heterogeneous data sources and formats, often in different representations and with data quality issues (which require integration and cleaning), as well as large data collections (which require distributed data storage and analysis). Accordingly, the goals of this module are to convey a basic understanding and practical applicability of
O1 Major data integration architectures
O2 Key techniques for data integration and cleaning
O3 Methods for large-scale data storage and analysis
O1 Major data integration architectures
O2 Key techniques for data integration and cleaning
O3 Methods for large-scale data storage and analysis
Content
This course covers major data integration architectures, key techniques for data integration and cleaning, as well as methods for large-scale, i.e., distributed, data storage and analysis. In detail, the module covers the following topics which also reflect the lecture calendar (with a separate 90-120min lecture per topic):
A: Data Integration and Preparation
01 Introduction and Overview
02 Data Warehousing, ETL, and SQL/OLAP
03 Message-oriented Middleware, EAI, and Replication
04 Schema Matching and Mapping
05 Entity Linking and Deduplication
06 Data Cleaning and Data Fusion
07 Data Provenance (and Blockchain)
B: Large-Scale Data Management and Analysis
08 Cloud Computing Fundamentals
09 Cloud Resource Management and Scheduling
10 Distributed Data Storage
11 Distributed, Data-Parallel Computation
12 Distributed Stream Processing
13 Distributed Machine Learning Systems
The module contains both lectures and exercises/programming projects which are offered as separate courses in order to increase flexibility.
A: Data Integration and Preparation
01 Introduction and Overview
02 Data Warehousing, ETL, and SQL/OLAP
03 Message-oriented Middleware, EAI, and Replication
04 Schema Matching and Mapping
05 Entity Linking and Deduplication
06 Data Cleaning and Data Fusion
07 Data Provenance (and Blockchain)
B: Large-Scale Data Management and Analysis
08 Cloud Computing Fundamentals
09 Cloud Resource Management and Scheduling
10 Distributed Data Storage
11 Distributed, Data-Parallel Computation
12 Distributed Stream Processing
13 Distributed Machine Learning Systems
The module contains both lectures and exercises/programming projects which are offered as separate courses in order to increase flexibility.
Module components
Compulsory area
The following courses are mandatory for this module:
| Courses | Type | Number | Frequency | Language | Hours per week | Course catalog |
|---|---|---|---|---|---|---|
| Data Integration and Large-scale Analysis | Lecture | 41112 | Winter semester | [en] | 3 | WS 2026/27 (1) ISIS |
| Data Integration and Large-scale Analysis | Exercise | 41112 | Winter semester | [en] | 2 | ISIS |
Courses in the course catalog
Courses created from the teaching units of this module.
Winter semester 2026/27 (1)
Data Integration and Large-scale Analysis (Lecture) 1
- Data Integration and Large-scale Analysis (Vorlesung) Termingruppe 1 41112
Workload and credit points
Data Integration and Large-scale Analysis (Lecture)
| Workload description | Multiplier | Hours | Total |
|---|---|---|---|
| Attendance Lectures | 15 | 3 h | 45 h |
| Pre/post-processing Lectures | 15 | 1 h | 15 h |
| Exam Preparation | 1 | 30 h | 30 h |
| 90 h | |||
Data Integration and Large-scale Analysis (Exercise)
| Workload description | Multiplier | Hours | Total |
|---|---|---|---|
| Attendance Discussion Rounds | 3 | 2 h | 6 h |
| Exercise Prototype Implementation | 1 | 84 h | 84 h |
| 90 h | |||
The workload of the module adds up to 180 hours. The module therefore comprises 6 credit points.
Description of teaching and learning methods
This module combines traditional lectures (on data integration and large-scale data engineering and analysis), supporting examples of open source systems, as well as practical exercises/projects on related topics. For the practical part, teams of 1-3 students can pick one of two alternatives:
a) Contribution of a unique feature to an open-source data-centric ML system (from a list of project proposals), or
b) Alternative exercise on large-scale data analysis pipelines using Apache Spark
a) Contribution of a unique feature to an open-source data-centric ML system (from a list of project proposals), or
b) Alternative exercise on large-scale data analysis pipelines using Apache Spark
Requirements for participation and examination
Desirable prerequisites for participation in the courses:
Completed basic courses on applied machine learning, data management, and distributed systems
Mandatory requirements for the module examination registration:
The examination requirements could not be retrieved.
Completion of the module
| Grading | Graded |
|---|---|
| Exam format | Written exam |
| Language | English |
| Duration/Extent | 90 |
Duration of the module
The following number of semesters is estimated for taking and completing the module: 1 semester(s).
This module can be started in the following semesters: Winter semester.
Maximum number of participants
This module is not limited to a number of students.
Registration procedures
Registration in ISIS and exercise/project selection within the first 4 weeks of the semester.
Recommended reading, lecture notes
Lecture notes
unavailable
Electronic lecture notes
available – https://mboehm7.github.io/teaching/ws2122_dia/index.htm
Recommended literature
No recommended literature given
Assigned programs
This module version is used in the following programs:
| Program / program version | Semester(s) | Curriculum component |
|---|---|---|
| Computer Engineering Computer Engineering (B. Sc.) - StuPO 2015 | SS 2025 | Wahlpflichtbereich › Informatik |
| WS 2025/26 | Wahlpflichtbereich › Informatik | |
| Computer Engineering Computer Engineering (M. Sc.) - StuPO 2015 | SS 2025 | Wahlpflicht Studiengebiete Fak. IV › Verteilte Systeme und Netze / Distributed Systems and Networks |
| WS 2025/26 | Wahlpflicht Studiengebiet B › Verteilte Systeme und Netze / Distributed Systems and Networks | |
| Computer Science Computer Science (B. Sc.) - StuPO 2015 | SS 2025 | Wahlpflichtbereich › Wahlpflichtbereich Katalog Informatik |
| WS 2025/26 | Wahlpflichtbereich › Wahlpflichtbereich Katalog Informatik | |
| Computer Science (Informatik) Computer Science (Informatik) (M. Sc.) - StuPO 2015 | SS 2025 | Wahlpflicht Studiengebiete Fak. IV › Kognitive Systeme / Cognitive Systems |
| WS 2025/26 | Studiengebiete › Data and Software Engineering / Data and Software Engineering | |
| Electrical Engineering Electrical Engineering (M. Sc.) - StuPO 2015 | SS 2025 | Wahlpflicht Studiengebiete Fak. IV › Verteilte Systeme und Netze / Distributed Systems and Networks |
| WS 2025/26 | Wahlpflicht Studiengebiete Fak. IV › Data and Software Engineering / Data and Software Engineering | |
| Information Systems Management Information Systems Management (B. Sc.) - StuPO 2015 | SS 2025 | Wahlpflichtbereich › Informatik |
| WS 2025/26 | Wahlpflichtbereich › Informatik | |
| Information Systems Management Information Systems Management (B. Sc.) - StuPO 2021 | SS 2025 | Wahlpflichtbereich › Informatik |
| WS 2025/26 | Wahlpflichtbereich › Informatik | |
| Information Systems Management (Wirtschaftsinformatik) Information Systems Management (Wirtschaftsinformatik) (M. Sc.) - StuPO 2017 | SS 2025 | Wahlpflichtbereich › Fachstudium Informatik › Verteilte Systeme und Netze / Distributed Systems and Networks |
| WS 2025/26 | Wahlpflichtbereich › Fachstudium Wirtschaft › Business, Economics and Management |
Miscellaneous
Not specified