Compact Course: Prompting and Evaluating Multimodal AI Models with AMMICO
Instructor: PD Dr. Delia Dumitrescu, Political Sciences, IPW / Dr. Inga Ulusoy, Research Software Engineer, Scientific Software Center
Prerequisites
Participants should have basic familiarity with handling structured data such as CSV files. In addition, basic familiarity with Python and Jupyter notebooks, as well as with generative AI and large language models, is useful but not required. Participants are required to bring their own laptops to work on during the course. They need to have a registered ORCID account to be able to use the Jupyter4NFDI service.
Summary
Multimodal AI models can analyze text, images, video, and combinations thereof, enabling researchers to extract structured information from increasingly large and heterogeneous datasets. However, the results of these models depend strongly on the selected model and model parameters, the dataset, and - particularly - the way a task is formulated through prompts.
In this hands-on course, participants will use AMMICO, an open-source Python package for AI-based media and content analysis, to systematically explore these factors. We will apply vision-language and audio models to different datasets and compare different model backends for tasks such as content description, feature extraction, and visual question answering.
A particular focus will be on prompt design. Participants will experiment with different ways of formulating the same research question and investigate how prompt specificity, context, output constraints, and examples influence model responses. We will develop prompts iteratively and discuss strategies for obtaining more consistent, interpretable, and reusable results.
The course will discuss practical approaches for evaluating outputs and documenting model and prompt configurations to make AI-assisted analyses more transparent and reproducible.