AI for security applications (WP4)
WP4 focuses on the reliability, security and resilience of artificial intelligence systems in critical applications. The aim is to enhance the trustworthiness of AI models, protection against manipulation, and the overall cyber security of both hardware and software infrastructure. The research covers three main areas. The first is the adaptation of large and small language models for professional domains such as pharmacology, biotechnology, law and the media, including the detection of disinformation and fraudulent content.
The second area involves investigating the two-way interaction between users and machine learning algorithms with the aim of increasing the transparency, controllability and security of decision-making processes. The third area involves analysing the computational complexity of AI models, including exploring the potential of quantum computing and developing computationally efficient alternatives to existing approaches. The outcome of WP4 is safer, more robust and more trustworthy AI systems suitable for deployment in critical and highly regulated sectors.
What problem does the WP solve, and why is it important?
01
AI systems remain vulnerable to errors, uncertainty in input data and deliberate manipulation
02
There is a growing risk of models being misused (e.g. disinformation, fraud, adversarial attacks)
03
There is a lack of sufficiently robust and interpretable models for use in critical and regulated areas
04
The computational demands of modern models limit their scalability and practical application
05
Interactions between users and AI systems are not always safe, transparent and controllable
Examples of use / areas of application and benefits
Pharmacology and Biotechnology
- greater reliability of AI when working with sensitive scientific data
- reducing the risk of misinterpretation in the analysis of biological and chemical information
- safer use of AI in environments where high levels of accuracy are required
Law and the regulatory environment
- greater transparency and explainability of AI decisions
- safer handling of legal documents and knowledge management systems
- minimising errors in the interpretation of complex textual sources
Média a informační prostor
- The media and the information landscape
- increasing resilience to cyber-attacks and manipulation of the public sphere
- promoting a trustworthy digital environment
Cyber security and critical IT infrastructure
- increased resilience of AI systems to adversarial attacks
- protection of models, data and decision-making processes against misuse
- strengthening the security of critical digital services
Computationally intensive AI and future architectures (including quantum research)
- more efficient and scalable AI models
- reducing the computational demands of existing approaches
- paving the way for a new computing paradigm in tackling complex problems
Key people
| FEL CTU
Technology transfer and the validation of research findings in practical applications
Participating institutions
Used technologies and procedures
- Multimodal AI models linking text, images, audio and video with structured data
- Methods for evaluating AI systems for the automatic assessment of the quality, reliability and safety of outputs
- Prompt engineering and meta-prompting for controlling the behaviour and quality of generative models
- Temporal analysis of text data and summarisation for the detection of trends and patterns
- A model of compression techniques (distillation, pruning, efficient RNN–Transformer architectures)
- Fine-tuning and adapting models (e.g. QLoRA) for domain-specific and efficient deployment.
- Black-box optimisation and kernel methods for robust analysis and anomaly detection in cybersecurity
- Analysis of quantum algorithms (NISQ) in terms of robustness and practical applicability
Achieved and planned results
2026
Abductive explanations of decisions made by artificial intelligence methods
R- software
2027
Tools for text cluster analysis with support for temporal data and network structures
R- software
Evolutionary software for prompt engineering
R- software
A library for interaction between the user and a machine learning system
R- software
Knihovna pro interakci mezi uživatelem a systémem strojového učení.
R- software
Methods for detecting hallucinations and improving the factual accuracy of texts generated by large language models
R- software
Pipelines for automated fact-checking
R- software
Privacy-preserving quantum machine learning algorithms for threat detection
R- software
2028
Pipelines and tools for fact extraction, document set summarisation and structured summarisation
R- software
Methods for the efficient training and inference of large language models (LLMs) with limited computational resources, supported by additional modalities (e.g. image data for OCR)
R- software
Small, security-focused language models for critical equipment
Gprot – prototyp
2029
Dual-agent offensive-defensive co-evolution
Gfunk – funkční vzorek
A semi-supervised quantum machine learning algorithm for anomaly detection
R- software
2030
A multi-LLM agent-based architecture for AI defence and attack simulation
R- software
2031
Agent-based AI architectures for attacking AI agents
Gfunk – funkční vzorek