IPE 03-2023 Master Thesis or Internship: Data Partitioning for Large Scientific Databases

Organizational unit

Institute for Data Processing and Electronics (IPE)

Job description

With large-scale experiment generating data at an unprecedented rate, we are witnessing continuous time-series data being pushed into the SQL database without many considerations. To improve the query performance, the database would be partitioned occasionally based on the principle that older data will not be used that often. However, partitioning the database as such is cumbersome, and even worse, this approach doesn’t guarantee performance improvement if users are querying data across the partitioned databases.

The focus of the project is to develop an efficient and robust database partitioning strategy for managing vast amounts of data generated by large-scale scientific experiments. Notably, the KATRIN experiment (Karlsruhe Tritium Neutrino Experiment) serves as an ideal use case to conduct the database evaluations. You will be involved in one of the research scopes below:

• Understand the complexities of large-scale scientific data and the challenges it poses to traditional database systems. Evaluate the performance difference between vertical partitioning and horizontal partitioning.
• Investigate and evaluate existing database partitioning techniques in the context of scientific data.
• Design and implement novel database partitioning algorithms tailored to the specific requirements of scientific experiments.
• Optimize data access and retrieval for faster query performance and enhanced data management.

This project offers a remarkable opportunity for skill development and knowledge enhancement in the domain of advanced database management where you will learn to handle large volumes of scientific data through cutting-edge database partitioning techniques.

Starting date

by appointment

Personal qualification

• Familiarity with programming languages such as Python or C++ or C#.
• Basic knowledge of database system.
• Practical experience with SQL database is advantageous but not mandatory.