{"api":"v1","id":"chips.inference-gains","title":"AI chips on the inference benchmark: each chip’s best result per chip in two compatible rounds","section":"chips","row":"chip, benchmark and scenario in two rounds","page":"https://jesseahawkins.com/chips/inference/","fields":[{"name":"benchmark","type":"string","tier":null},{"name":"benchmark_name","type":"string","tier":null},{"name":"scenario","type":"string","tier":null},{"name":"units","type":"string","tier":null},{"name":"chip","type":"string","tier":null},{"name":"from_round","type":"string","tier":null},{"name":"from_result_id","type":"string","tier":null},{"name":"from_submitter","type":"string","tier":null},{"name":"from_system","type":"string","tier":null},{"name":"from_chips","type":"number","tier":"derived"},{"name":"from_result","type":"number","tier":"derived"},{"name":"from_per_chip","type":"number","tier":"derived"},{"name":"to_round","type":"string","tier":null},{"name":"to_result_id","type":"string","tier":null},{"name":"to_submitter","type":"string","tier":null},{"name":"to_system","type":"string","tier":null},{"name":"to_chips","type":"number","tier":"derived"},{"name":"to_result","type":"number","tier":"derived"},{"name":"to_per_chip","type":"number","tier":"derived"},{"name":"ratio","type":"number","tier":"derived"},{"name":"compatibility_row","type":"string","tier":null},{"name":"per_chip_status","type":"string","tier":null},{"name":"page","type":"string","tier":null},{"name":"license","type":"string","tier":null},{"name":"trademark","type":"string","tier":null}],"tiers":["derived"],"provenance":"Each chip’s best figure per chip in two rounds, and their ratio, are this site’s arithmetic on MLCommons results; the rounds are paired only where the MLCommons compatibility table marks both.","freshness":{"state":"current","asOf":"2026-11-12","oldestCheck":"2026-11-10","next":"2026-12-25","late":"2027-01-01","status":"https://jesseahawkins.com/status/","sources":["mlperf"],"notice":null},"rowCount":64,"terms":["MLCommons results repository, Apache License 2.0"],"checked":"2026-11-10","nextCheckBy":"2026-12-25","links":{"rows":"https://jesseahawkins.com/data/v1/datasets/chips.inference-gains.json","csv":"https://jesseahawkins.com/chips/inference-gains.csv","releases":"https://jesseahawkins.com/data/v1/datasets/chips.inference-gains/releases.json"},"latestRelease":{"id":"2026-11-10","date":"2026-11-10","sha256":"bd27b514175d7338058bb6f7928db1648840241bd2712ac0e0fed01da89e5bd2","json":"https://jesseahawkins.com/data/v1/datasets/chips.inference-gains/releases/2026-11-10.json","csv":"https://jesseahawkins.com/data/releases/chips/inference-gains/2026-11-10.csv"},"csvSha256":"bd27b514175d7338058bb6f7928db1648840241bd2712ac0e0fed01da89e5bd2","matchesLatestRelease":true,"rows":[{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Server","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0069","from_submitter":"NVIDIA","from_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":18592.2,"from_per_chip":2324.025,"to_round":"6.0","to_result_id":"6.0-0083","to_submitter":"Nebius","to_system":"Nebius B200 n1 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":51692.9,"to_per_chip":6461.6125,"ratio":2.7803541269994945,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB300","from_round":"5.1","from_result_id":"5.1-0072","from_submitter":"NVIDIA","from_system":"(72x GB300-288GB_aarch64, TensorRT)","from_chips":72,"from_result":209328,"from_per_chip":2907.3333333333335,"to_round":"6.0","to_result_id":"6.0-0082","to_submitter":"Nebius","to_system":"Nebius GB300 NVL72 (4x GB300-288GB_aarch64, TensorRT)","to_chips":8,"to_result":64510,"to_per_chip":8063.75,"ratio":2.7735897729878465,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0071","from_submitter":"NVIDIA","from_system":"(72x GB200-186GB_aarch64, TensorRT)","from_chips":72,"from_result":167578,"from_per_chip":2327.472222222222,"to_round":"6.0","to_result_id":"6.0-0015","to_submitter":"CoreWeave","to_system":"CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)","to_chips":64,"to_result":311531,"to_per_chip":4867.671875,"ratio":2.0913984830944394,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0071","from_submitter":"NVIDIA","from_system":"(72x GB200-186GB_aarch64, TensorRT)","from_chips":72,"from_result":289712,"from_per_chip":4023.777777777778,"to_round":"6.0","to_result_id":"6.0-0015","to_submitter":"CoreWeave","to_system":"CoreWeave GB200 NVL72 (64x GB200-186GB_aarch64, TensorRT)","to_chips":64,"to_result":468665,"to_per_chip":7322.890625,"ratio":1.8199043360302645,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0069","from_submitter":"NVIDIA","from_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":33379,"from_per_chip":4172.375,"to_round":"6.0","to_result_id":"6.0-0083","to_submitter":"Nebius","to_system":"Nebius B200 n1 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":58581.6,"to_per_chip":7322.7,"ratio":1.7550435902813146,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"deepseek-r1","benchmark_name":"DeepSeek-R1","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB300","from_round":"5.1","from_result_id":"5.1-0072","from_submitter":"NVIDIA","from_system":"(72x GB300-288GB_aarch64, TensorRT)","from_chips":72,"from_result":420659,"from_per_chip":5842.486111111111,"to_round":"6.0","to_result_id":"6.0-0017","to_submitter":"CoreWeave","to_system":"CoreWeave GB300 NVL72 (8x GB300-288GB_aarch64, TensorRT)","to_chips":8,"to_result":78564,"to_per_chip":9820.5,"ratio":1.68087690980105,"compatibility_row":"DeepSeek R1","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"whisper","benchmark_name":"Whisper","scenario":"Offline","units":"Samples/s","chip":"NVIDIA L40S","from_round":"5.1","from_result_id":"5.1-0022","from_submitter":"Dell","from_system":"Dell PowerEdge XE7740 (8x L40S, TensorRT)","from_chips":8,"from_result":9272,"from_per_chip":1159,"to_round":"6.0","to_result_id":"6.0-0090","to_submitter":"RedHat","to_system":"IBM Cloud gx3-48x240x2l40s instance,2xL40S-PCIE-48GB,vLLM 0.15.1","to_chips":2,"to_result":3646.91,"to_per_chip":1823.455,"ratio":1.5733002588438307,"compatibility_row":"Whisper","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB300","from_round":"5.1","from_result_id":"5.1-0072","from_submitter":"NVIDIA","from_system":"(72x GB300-288GB_aarch64, TensorRT)","from_chips":72,"from_result":12248.4,"from_per_chip":170.11666666666667,"to_round":"6.0","to_result_id":"6.0-0078","to_submitter":"NVIDIA","to_system":"NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)","to_chips":72,"to_result":18627.6,"to_per_chip":258.71666666666664,"ratio":1.5208190457529145,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0071","from_submitter":"NVIDIA","from_system":"(72x GB200-186GB_aarch64, TensorRT)","from_chips":72,"from_result":9921.02,"from_per_chip":137.79194444444445,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":14010.1,"to_per_chip":194.58472222222224,"ratio":1.4121632654706875,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0070","from_submitter":"NVIDIA","from_system":"(4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":29745.6,"from_per_chip":7436.4,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":754855,"to_per_chip":10484.097222222223,"ratio":1.40983503068988,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0070","from_submitter":"NVIDIA","from_system":"(4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":29745.6,"from_per_chip":7436.4,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":754855,"to_per_chip":10484.097222222223,"ratio":1.40983503068988,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":36052.3,"from_per_chip":4506.5375,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":44087.3,"to_per_chip":5510.9125,"ratio":1.2228706628980675,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0071","from_submitter":"NVIDIA","from_system":"(72x GB200-186GB_aarch64, TensorRT)","from_chips":72,"from_result":11614.3,"from_per_chip":161.3097222222222,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":14133.9,"to_per_chip":196.30416666666667,"ratio":1.2169394625590868,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB300","from_round":"5.1","from_result_id":"5.1-0072","from_submitter":"NVIDIA","from_system":"(72x GB300-288GB_aarch64, TensorRT)","from_chips":72,"from_result":16104.1,"from_per_chip":223.66805555555555,"to_round":"6.0","to_result_id":"6.0-0078","to_submitter":"NVIDIA","to_system":"NVIDIA GB300 NVL72 (72x GB300-288GB_aarch64, TensorRT)","to_chips":72,"to_result":19511.7,"to_per_chip":270.99583333333334,"ratio":1.2115982886345713,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Server","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":5348.45,"from_per_chip":1337.1125,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":6293.89,"to_per_chip":1573.4725,"ratio":1.1767689704493827,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":26205.8,"from_per_chip":3275.725,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":29908.4,"to_per_chip":3738.55,"ratio":1.1412893328957712,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":26205.3,"from_per_chip":3275.6625,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":29900.1,"to_per_chip":3737.5125,"ratio":1.140994378999668,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":26005.9,"from_per_chip":3250.7375,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":28727.8,"to_per_chip":3590.975,"ratio":1.104664710700264,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":26001,"from_per_chip":3250.125,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":28717.1,"to_per_chip":3589.6375,"ratio":1.104461366870505,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0062","from_submitter":"Lenovo","from_system":"ThinkSystem SR780a V3 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":146960,"from_per_chip":18370,"to_round":"6.0","to_result_id":"6.0-0062","to_submitter":"Lambda_SIT","to_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":160403,"to_per_chip":20050.375,"ratio":1.0914738704409364,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"whisper","benchmark_name":"Whisper","scenario":"Offline","units":"Samples/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0069","from_submitter":"NVIDIA","from_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":45333.4,"from_per_chip":5666.675,"to_round":"6.0","to_result_id":"6.0-0040","to_submitter":"Google","to_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":49327.2,"to_per_chip":6165.9,"ratio":1.0880983998552942,"compatibility_row":"Whisper","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":3009.38,"from_per_chip":752.345,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":3270.66,"to_per_chip":817.665,"ratio":1.0868218702855736,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":3009.38,"from_per_chip":752.345,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":3270.66,"to_per_chip":817.665,"ratio":1.0868218702855736,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":46841.8,"from_per_chip":5855.225,"to_round":"6.0","to_result_id":"6.0-0052","to_submitter":"HPE","to_system":"HPE ProLiant DL385 Gen11 (2x NVIDIA RTX PRO 6000 Blackwell Server Edition-600W, TensorRT)","to_chips":2,"to_result":12495.2,"to_per_chip":6247.6,"ratio":1.0670127962631666,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"mixtral-8x7b","benchmark_name":"Mixtral 8x7B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":30134.3,"from_per_chip":3766.7875,"to_round":"6.0","to_result_id":"6.0-0046","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":32151,"to_per_chip":4018.875,"ratio":1.0669237380659249,"compatibility_row":"Mixtral-8x7b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":46060,"from_per_chip":5757.5,"to_round":"6.0","to_result_id":"6.0-0047","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":48794.3,"to_per_chip":6099.2875,"ratio":1.0593638732088582,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"whisper","benchmark_name":"Whisper","scenario":"Offline","units":"Samples/s","chip":"NVIDIA H200","from_round":"5.1","from_result_id":"5.1-0049","from_submitter":"HPE","from_system":"HPE Cray XD670 (8x H200-SXM-141GB)","from_chips":8,"from_result":34450.7,"from_per_chip":4306.3375,"to_round":"6.0","to_result_id":"6.0-0093","to_submitter":"RedHat","to_system":"IBM Cloud qx3d-160x1792x8h200,NVIDIA 8xH200-SXM-141GB,vLLM 0.15.1","to_chips":8,"to_result":36395.7,"to_per_chip":4549.4625,"ratio":1.0564574885270837,"compatibility_row":"Whisper","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0086","from_submitter":"Supermicro","from_system":"AS-4126GS-NBR-LCC (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":122269,"from_per_chip":15283.625,"to_round":"6.0","to_result_id":"6.0-0062","to_submitter":"Lambda_SIT","to_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":128750,"to_per_chip":16093.75,"ratio":1.0530060767651652,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0011","from_submitter":"Cisco","from_system":"Cisco UCS C845A M8 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":50776.8,"from_per_chip":6347.1,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":53193.6,"to_per_chip":6649.2,"ratio":1.0475965401521954,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Offline","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":6472.37,"from_per_chip":1618.0925,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":6762.3,"to_per_chip":1690.575,"ratio":1.044795028714366,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":2110.05,"from_per_chip":527.5125,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":2199.5,"to_per_chip":549.875,"ratio":1.0423923603706073,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"Intel Arc Pro B60 Dual 48G","from_round":"5.1","from_result_id":"5.1-0056","from_submitter":"Intel","from_system":"1-node-4x-BMG-Pro-B60-Dual","from_chips":4,"from_result":2110.05,"from_per_chip":527.5125,"to_round":"6.0","to_result_id":"6.0-0060","to_submitter":"Intel","to_system":"1-node-4x-BMG-Pro-B60-Dual","to_chips":4,"to_result":2199.5,"to_per_chip":549.875,"ratio":1.0423923603706073,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"mixtral-8x7b","benchmark_name":"Mixtral 8x7B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA RTX PRO 6000 Blackwell","from_round":"5.1","from_result_id":"5.1-0051","from_submitter":"HPE","from_system":"HPE ProLiant Compute DL380a Gen12 (8x RTX Pro 6000 Blackwell Server Edition)","from_chips":8,"from_result":32738.8,"from_per_chip":4092.35,"to_round":"6.0","to_result_id":"6.0-0046","to_submitter":"HPE","to_system":"HPE ProLiant Compute DL380a Gen12 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition, TensorRT)","to_chips":8,"to_result":33637.3,"to_per_chip":4204.6625,"ratio":1.0274445001038524,"compatibility_row":"Mixtral-8x7b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"AMD Instinct MI300X","from_round":"5.1","from_result_id":"5.1-0066","from_submitter":"MangoBoost","from_system":"Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)","from_chips":8,"from_result":24860.5,"from_per_chip":3107.5625,"to_round":"6.0","to_result_id":"6.0-0069","to_submitter":"MangoBoost","to_system":"MangoBoost Mi300X (8x MI300X, LLMBoost)","to_chips":8,"to_result":25463.1,"to_per_chip":3182.8875,"ratio":1.0242392550431407,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"AMD Instinct MI300X","from_round":"5.1","from_result_id":"5.1-0066","from_submitter":"MangoBoost","from_system":"Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)","from_chips":8,"from_result":24860.5,"from_per_chip":3107.5625,"to_round":"6.0","to_result_id":"6.0-0069","to_submitter":"MangoBoost","to_system":"MangoBoost Mi300X (8x MI300X, LLMBoost)","to_chips":8,"to_result":25463.1,"to_per_chip":3182.8875,"ratio":1.0242392550431407,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":31267.4,"from_per_chip":3908.425,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":32004,"to_per_chip":4000.5,"ratio":1.0235580828594637,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":31267.4,"from_per_chip":3908.425,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":32004,"to_per_chip":4000.5,"ratio":1.0235580828594637,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0073","from_submitter":"Nebius","from_system":"Nebius B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":101611,"from_per_chip":12701.375,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":103635,"to_per_chip":12954.375,"ratio":1.0199191032466957,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0073","from_submitter":"Nebius","from_system":"Nebius B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":101611,"from_per_chip":12701.375,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":103627,"to_per_chip":12953.375,"ratio":1.0198403716133095,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0028","from_submitter":"Dell","from_system":"Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":128794,"from_per_chip":16099.25,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":131270,"to_per_chip":16408.75,"ratio":1.0192244980356229,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0061","from_submitter":"Lenovo","from_system":"ThinkSystem SR680a V3 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":102909,"from_per_chip":12863.625,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":104572,"to_per_chip":13071.5,"ratio":1.0161599082684702,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0061","from_submitter":"Lenovo","from_system":"ThinkSystem SR680a V3 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":102909,"from_per_chip":12863.625,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":103954,"to_per_chip":12994.25,"ratio":1.010154602610073,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"whisper","benchmark_name":"Whisper","scenario":"Offline","units":"Samples/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0052","from_submitter":"HPE","from_system":"HPE ProLiant DL385 Gen11 (2x H200-NVL-141GB)","from_chips":2,"from_result":7925.55,"from_per_chip":3962.775,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":31969.2,"to_per_chip":3996.15,"ratio":1.008422128432727,"compatibility_row":"Whisper","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0074","from_submitter":"Nebius","from_system":"Nebius GB200 (4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":855.82,"from_per_chip":213.955,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":15461.5,"to_per_chip":214.74305555555554,"ratio":1.003683277116943,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":29070.3,"from_per_chip":3633.7875,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":29084.9,"to_per_chip":3635.6125,"ratio":1.0005022307991318,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":29070.3,"from_per_chip":3633.7875,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":29084.9,"to_per_chip":3635.6125,"ratio":1.0005022307991318,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-8b","benchmark_name":"Llama 3.1 8B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0080","from_submitter":"Quanta_Cloud_Technology","from_system":"D75E-4U_H200-NVL-141GBx4","from_chips":4,"from_result":25753.8,"from_per_chip":6438.45,"to_round":"6.0","to_result_id":"6.0-0013","to_submitter":"Cisco","to_system":"Cisco UCS X210 M8 - X580P PCIe Node (4x H200-NVL-141GB, TensorRT)","to_chips":4,"to_result":25731.6,"to_per_chip":6432.9,"ratio":0.9991379912867228,"compatibility_row":"Llama 3.1 8B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":16419.2,"from_per_chip":2052.4,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":16343.8,"to_per_chip":2042.975,"ratio":0.9954078152406938,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA H200 NVL","from_round":"5.1","from_result_id":"5.1-0023","from_submitter":"Dell","from_system":"Dell PowerEdge XE7745 (8x H200-NVL-141GB, TensorRT)","from_chips":8,"from_result":16419.2,"from_per_chip":2052.4,"to_round":"6.0","to_result_id":"6.0-0021","to_submitter":"Dell","to_system":"PowerEdge XE7740 (8x H200-NVL-141GB, TensorRT)","to_chips":8,"to_result":16343.8,"to_per_chip":2042.975,"ratio":0.9954078152406938,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"AMD Instinct MI325X","from_round":"5.1","from_result_id":"5.1-0045","from_submitter":"GigaComputing","from_system":"G893-ZX1-AAX2","from_chips":8,"from_result":32139.4,"from_per_chip":4017.425,"to_round":"6.0","to_result_id":"6.0-0070","to_submitter":"MangoBoost","to_system":"MangoBoost Mi325X (8x MI325X, LLMBoost)","to_chips":8,"to_result":31888.9,"to_per_chip":3986.1125,"ratio":0.9922058283602059,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"AMD Instinct MI325X","from_round":"5.1","from_result_id":"5.1-0045","from_submitter":"GigaComputing","from_system":"G893-ZX1-AAX2","from_chips":8,"from_result":32139.4,"from_per_chip":4017.425,"to_round":"6.0","to_result_id":"6.0-0070","to_submitter":"MangoBoost","to_system":"MangoBoost Mi325X (8x MI325X, LLMBoost)","to_chips":8,"to_result":31888.9,"to_per_chip":3986.1125,"ratio":0.9922058283602059,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"AMD Instinct MI300X","from_round":"5.1","from_result_id":"5.1-0066","from_submitter":"MangoBoost","from_system":"Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)","from_chips":8,"from_result":27854.4,"from_per_chip":3481.8,"to_round":"6.0","to_result_id":"6.0-0069","to_submitter":"MangoBoost","to_system":"MangoBoost Mi300X (8x MI300X, LLMBoost)","to_chips":8,"to_result":27598.7,"to_per_chip":3449.8375,"ratio":0.9908201217760928,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"AMD Instinct MI300X","from_round":"5.1","from_result_id":"5.1-0066","from_submitter":"MangoBoost","from_system":"Supermicro AS-8125GS-TNMR2 (8x MI300X-192GB MangoBoost-LLMBoost)","from_chips":8,"from_result":27854.4,"from_per_chip":3481.8,"to_round":"6.0","to_result_id":"6.0-0069","to_submitter":"MangoBoost","to_system":"MangoBoost Mi300X (8x MI300X, LLMBoost)","to_chips":8,"to_result":27598.7,"to_per_chip":3449.8375,"ratio":0.9908201217760928,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"whisper","benchmark_name":"Whisper","scenario":"Offline","units":"Samples/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0077","from_submitter":"Oracle","from_system":"BM.GPU.GB200.4","from_chips":4,"from_result":22272.6,"from_per_chip":5568.15,"to_round":"6.0","to_result_id":"6.0-0041","to_submitter":"Google","to_system":"NVIDIA GB200 NVL72 (4x GB200-186GB_aarch64, TensorRT)","to_chips":4,"to_result":22051.4,"to_per_chip":5512.85,"ratio":0.9900685146772269,"compatibility_row":"Whisper","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0046","from_submitter":"GigaComputing","from_system":"G894-SD1","from_chips":8,"from_result":62851.2,"from_per_chip":7856.4,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":61644.7,"to_per_chip":7705.5875,"ratio":0.9808038669110534,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"AMD Instinct MI325X","from_round":"5.1","from_result_id":"5.1-0003","from_submitter":"AMD","from_system":"QuantaGrid D74A-7U","from_chips":8,"from_result":34555,"from_per_chip":4319.375,"to_round":"6.0","to_result_id":"6.0-0070","to_submitter":"MangoBoost","to_system":"MangoBoost Mi325X (8x MI325X, LLMBoost)","to_chips":8,"to_result":33880.4,"to_per_chip":4235.05,"ratio":0.9804774996382579,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"AMD Instinct MI325X","from_round":"5.1","from_result_id":"5.1-0003","from_submitter":"AMD","from_system":"QuantaGrid D74A-7U","from_chips":8,"from_result":34555,"from_per_chip":4319.375,"to_round":"6.0","to_result_id":"6.0-0070","to_submitter":"MangoBoost","to_system":"MangoBoost Mi325X (8x MI325X, LLMBoost)","to_chips":8,"to_result":33880.4,"to_per_chip":4235.05,"ratio":0.9804774996382579,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Interactive","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0046","from_submitter":"GigaComputing","from_system":"G894-SD1","from_chips":8,"from_result":62851.2,"from_per_chip":7856.4,"to_round":"6.0","to_result_id":"6.0-0048","to_submitter":"HPE","to_system":"HPE ProLiant Compute XD685 (8x NVIDIA B200 180GB, TensorRT)","to_chips":8,"to_result":61300.4,"to_per_chip":7662.55,"ratio":0.975325848989359,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0070","from_submitter":"NVIDIA","from_system":"(4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":51736.9,"from_per_chip":12934.225,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":888054,"to_per_chip":12334.083333333334,"ratio":0.9536004927495334,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0008","from_submitter":"Azure","from_system":"ND_GB200_v6","from_chips":4,"from_result":52061.6,"from_per_chip":13015.4,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":888054,"to_per_chip":12334.083333333334,"ratio":0.9476530366591372,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99","benchmark_name":"Llama 2 70B, 99% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0070","from_submitter":"NVIDIA","from_system":"(4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":49359.6,"from_per_chip":12339.9,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":810104,"to_per_chip":11251.444444444445,"ratio":0.9117938106827806,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama2-70b-99.9","benchmark_name":"Llama 2 70B, 99.9% accuracy","scenario":"Server","units":"Tokens/s","chip":"NVIDIA GB200","from_round":"5.1","from_result_id":"5.1-0070","from_submitter":"NVIDIA","from_system":"(4x GB200-186GB_aarch64, TensorRT)","from_chips":4,"from_result":49359.6,"from_per_chip":12339.9,"to_round":"6.0","to_result_id":"6.0-0075","to_submitter":"NVIDIA","to_system":"NVIDIA GB200 NVL72 (72x GB200-186GB_aarch64, TensorRT)","to_chips":72,"to_result":810104,"to_per_chip":11251.444444444445,"ratio":0.9117938106827806,"compatibility_row":"Llama2-70b","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Offline","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0028","from_submitter":"Dell","from_system":"Dell PowerEdge XE9685L (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":1660.71,"from_per_chip":207.58875,"to_round":"6.0","to_result_id":"6.0-0040","to_submitter":"Google","to_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":1350.24,"to_per_chip":168.78,"ratio":0.8130498401286197,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."},{"benchmark":"llama3.1-405b","benchmark_name":"Llama 3.1 405B","scenario":"Server","units":"Tokens/s","chip":"NVIDIA B200","from_round":"5.1","from_result_id":"5.1-0073","from_submitter":"Nebius","from_system":"Nebius B200 (8x B200-SXM-180GB, TensorRT)","from_chips":8,"from_result":1279.53,"from_per_chip":159.94125,"to_round":"6.0","to_result_id":"6.0-0040","to_submitter":"Google","to_system":"NVIDIA DGX B200 (8x B200-SXM-180GB, TensorRT)","to_chips":8,"to_result":983.626,"to_per_chip":122.95325,"ratio":0.7687400842496854,"compatibility_row":"Llama 3.1 405B","per_chip_status":"A figure per chip is this site’s division of a result by the system’s count of accelerators. It is not a primary metric of MLPerf® Inference, and it is not verified by MLCommons Association.","page":"https://jesseahawkins.com/chips/inference/#gains","license":"MLCommons results repository, Apache License 2.0","trademark":"The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information."}]}