Benchmarking Process Considerations

There are a series of aspects that need to be taken into considerations when performing benchmarks.

Starting from the cost, there are financial costs (such as acquiring or renting hardware resources to run benchmarks), and energy costs, as running AI algorithms consumes computational resources and power.

Sustainability is of paramount importance and different aspects should be considered. The environmental sustainability should aim at a responsible use of resources, minimizing carbon emissions, and reducing ecological footprints. The data collection has also sustainable implications as well, and the use of efficient techniques to minimize environmental impact should be fostered, together with strategies to reduce unnecessary data points and optimize data storage to lower energy consumption and emissions. Finally, all-around energy-efficient practices should be adopted, such as preferring cloud-based or energy-efficient high-performance computing resources, and using energy-efficient infrastructure for storing benchmarking results.

Connected to the previous issues, the actual benchmarking necessity should be taken into account, to avoid redundant benchmarking if AI assets are already benchmarked in sufficient detail. Finally, collaboration and standardization actions should be taken, so as to promote collaboration among organizations to develop standardized benchmarking methodologies. This would allow to foster transparency and reduce duplication of efforts. Additionally, the usage of standardized benchmarks allows for comparing environmental performance and drive sustainable practices.