Welcome to AI4Europe-Benchmarking-Guide!

Benchmarking is a crucial aspect of Artificial Intelligence (AI). In this document, you will find guidelines for the creation of useful, reproducible, and trustworthy benchmark of your AI applications.

Contents

• Home
• Why Benchmarking?
  • Motivations
  • Relevance
  • Challenges
    • Task-specific Evaluations and Large-Language Models
• Benchmarking Basics
  • Preliminaries
  • Workflow
    • AI Asset Selection
    • Hardware Resources
    • Hyperparameters
      • Parameters vs. Hyperparameters in Large Language Models (LLMs)
    • Datasets
    • Metrics
    • Benchmark Output
• Metrics
  • Choice of metrics to be measured
    • Primary versus Secondary metrics
    • Examples of metrics
    • Influence of Metric Choice on Model Selection
    • How Metric Choice Influences the End Result
    • Examples of different benchmarking tasks and their metrics
• Benchmark Output
  • Metadata Companion
    • Metadata Example
• API
  • Configuration files
  • Datasets
  • Hardware platforms (WIP)
  • Examples
    • Response for /config (GET):
    • cURL examples
      • Download (GET)
      • Upload (POST)
      • Delete (DELETE)
• Benchmarking for Automation
  • Importance of Benchmarking
  • Challenges in AI Adoption
  • Automated Matching of an AI Asset to Hardware Resources
• Benchmarking Process Considerations
• Reproducibility in Benchmarking: Challenges and Opportunities
  • Challenges in Reproducibility
  • Existing Solutions and Limitations
  • The Path Forward: Federated Computing Infrastructures
  • Reproducible Benchmarks in the AIoD Platform
    • REANA as a Reproducibility Tool
  • Data Reproducibility Checklist
• BonsAPPs Benchmarking Service
  • How it works
    • Features
    • Supported Metrics
    • User Workflow
    • Integration with Bonseyes AI Marketplace
  • Documentation

• Index

• Module Index

• Search Page