aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.

https://aws-samples.github.io/foundation-model-benchmarking-tool/

aws-samples/foundation-model-benchmarking-tool Issues

KeyError: "Column(s) ['GPUMemoryUtilization', 'GPUUtilization'] do not exist" for non-GPU instances
Closed a month ago5
Add Time to first token (TTFT), time per output token (TPOT) and time to last token (TTLT)
Updated a month ago
Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker.
Updated 2 months ago1
Support Python 3.10 for SageMaker Studio notebooks
Closed 2 months ago1
Error: need to escape, but no escapechar set
Closed 2 months ago1
Loosen dependencies for more flexible installation
Updated 2 months ago1
Reports only show information per instance instead of per endpoint
Updated 2 months ago
No such file or directory: '/tmp/fmbench-read/tokenizer'
Closed 2 months ago2
To integrate the Open-Orca HF dataset to FMBench
Closed 2 months ago1
To add support for bring your own 'SageMaker' pre existing endpoint
Closed 2 months ago1
FMBench needs to be configurable so that no S3 access is needed
Closed 2 months ago1
config_filepath is incorrect
Closed 2 months ago1
FMBench to support an REST Predictor
Closed 3 months ago
FMbench support for different AWS Region
Closed 4 months ago1
Support for custom datasets and custom inference scripts
Closed 4 months ago
FMBench to support benchmarking for embedding models
Updated 4 months ago
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
Updated 5 months ago
Add support for different payload formats for bring your own datasets for that might be needed for different inference containers
Updated 5 months ago1
Emit live metrics
Updated 5 months ago2
Compare different models for the same dataset
Updated 6 months ago
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute
Updated 6 months ago
Provide config file for FLAN-T5 out of the box
Updated 6 months ago
Allow different experiments in the same config file to use different regions
Updated 6 months ago
Add Autoscaling functionality to FMBench
Updated 6 months ago
Containerize FMBench and provide instructions for running the container on EC2
Updated 6 months ago
The business summary plot in the report needs to have a caption for disclaimer
Updated 6 months ago
The README gives an impression that this tool is comparing instance types when it should be serving stacks
Updated 6 months ago
Add Bedrock benchmarking to this tool
Updated 6 months ago
Merge model accuracy metrics also into this tool
Updated 6 months ago
Add support for models hosted on platforms other than SageMaker
Updated 6 months ago
Add documentation for the config file on readthedocs.io
Updated 6 months ago
Per notebook run support via py package repo
Closed 6 months ago1
Add a AWS CloudFormation template and provide some canned datasets to get started
Closed 6 months ago1
Code cleanup needed to replace the notebooks with regular python files
Updated 6 months ago
Add support for a default tokenizer if no tokenizer is specified
Closed 6 months ago
Assign cost per run for FMBT
Closed 6 months ago
Add code to determine the cost of running an entire experiment and include it in the final report
Closed 6 months ago
Add support for a custom token counter
Closed 6 months ago
Convert 'FMBT' into a python packaged version
Closed 6 months ago
[Highest priority] Add support for reading and writing files (configuration, metrics, bring your own model scripts) from Amazon S3.
Closed 6 months ago