ARIA: Scaling Compute - Benchmarking

ARIA welcome applications from across the R&D ecosystem, including individuals, universities, research institutions, small, medium and large companies, charities and public sector research organisations.

This Request For Proposals (RFP) is derived from the programme thesis Unlocking AI compute hardware at 1/1000th the cost and Nature computes better opportunity space.

As described in the solicitation for Technical Areas (TA) 1-3, the programme is designed to demonstrate that:

• It is possible to drop the hardware costs required to train large AI models by >1000x
• It is possible to do this without primarily relying on leading-edge fabrication facilities All programme activities will be anchored around reducing the hardware costs required to train large-scale AI models

All programme activities will be anchored around reducing the hardware costs required to train large-scale AI models.

The initial programme targets are defined with targeted time/cost pareto frontier to train three specific workloads from the MLPerf benchmark (to the quality level described in the benchmark).

During the delivery phase, all programme activities in TA 1-3 will be evaluated based on ability to meet these targets.

This TA 4 RFP is designed to find a team who can help track these moving targets and publish their findings to the research community. The purpose is to provide an accurate (and open) source of ground truth of where the overall programme targets should be in a fast-moving industry.

View the full call for proposals (PDF) with technical details.

As outlined above, TA 4 represents the benchmarking component of the programme. Successful applicants for TA 4 will be asked to monitor and select relevant workloads representing the state-of-the-art AI models, profile their performance on commercial AI accelerators, create estimates of cost and runtime, and share their findings (and code) with ARIA and the broader R&D community.

We are looking to fund this TA with up to £2m (inclusive of VAT and all other costs including overheads). We expect to fund one award in this TA. Applicants recognise and accept that it will be at ARIA’s sole discretion as to which, if any, proposal is accepted.

We are looking for a team who can periodically poll the current landscape of best-in-class AI models (workloads with available code/data and which can fit on modest-sized commercial hardware). The TA 4 team will be expected to estimate performance/cost of pre-training, fine-tuning, and inference of the selected models. The successful TA 4 team will be evaluated on how best they can: (1) identify models and implement known algorithmic optimisations which are representative of the existing state-of-the-art, (2) perform baseline benchmarking using commercially available hardware and profiling tools, and (3) extrapolate performance for fully scaled-out systems where appropriate.

If you have any questions relating to the RFP, please submit your question to clarifications@aria.org.uk. Clarification questions should be submitted no later than 3 days prior to the application deadline date. Clarification questions received after this date will not be reviewed. Any questions or responses containing information relevant to all applicants will be provided to everyone that has started a submission within the application portal.

If you would like help from Innovate UK Business Connect to find a collaboration partner, contact our Digital or Electronics teams.