Project Group: Benchmarking and Optimizing RAG Systems (SS26)

In this PG, we are going to gain hands-on experience in systems for retrieval augmented generation (RAG) technology for LLMs and in particular the understanding and optimization of RAG performance on modern mobile devices. For a quick overview of this PG, please check this introductory slides deck. 

Course In­form­a­tion

Learning Goals

This PG aims to train on both scientific and academic soft skills. After taking this PG, students will be able to 

  • Read scientific papers and perform some basic literature study
  • Get familiar with the mobile RAG pipeline and systems
  • Propose a research idea with a clear motivation
  • Design and implement a system to realize the proposed research idea
  • Demonstrate your research prototype to your peers
  • Write a scientific report summarzing your ideas and findings

 

Organization

We will have weekly meetings for progress monitoring and feedback. The PG is organized in three phases:

Phase 1: Literature Study and Proposal

  • You will be introduced to the topic (in the first appointment) and will be given a list of papers to start
  • Read these papers and other relevant ones you find thoroughly and think about potential directions for your work
  • Propose a clear research idea your would like to pursue in the PG and write a one-pager proposal with clear motivation
  • Receive feedback and approval about your proposal from the PG coordinator

Phase 2: Design and Implementation

  • Get familiar with our mobile RAG benchmarking system
  • Sketch the design and clarify tools and open-source software that can be used for implementation
  • Implement your design
  • Perform a case study to validate your design and implementation

Phase 3: Demonstration, Presentation, and Report

  • Prepare a demo for your project
  • Demonstrate and present your project to your peers (in a poster/demo format)
  • Write a report to summarize your ideas and findings
  • (Optional) Submit a research paper to a scientific conference

 

Assessment

The seminar will be assessed based on three components:

  • Scientific quality (20%): Ideally, your research idea should be original and goes beyond the state-of-the-art in some way. The motivation should be clearly stated.
  • Implementation quality (50%): Your implementation should be solid and be able to demonstrate your research ideas. Your code should be well organized and documented.
  • Demo and report quality (30%): You should be able to explain your project to others and summarize your work in a report with clarity. 

You pass the course if you receive no less than 50% overall. Partitipation in each and every component is strictly required; getting a zero for any component means failing the course. 

 

Important Dates 

Note: The following dates and tentative and are subject to changes.

  • Proposal deadline: TBA
  • Implementation midterm: TBA
  • Demo: TBA
  • Final report submission: TBA