DEVOPSdigest

OctoML announced early access to Octomizer.

Octomizer brings the power and potential of Apache TVM, an open source deep learning compiler project that is becoming a de facto industry standard, to machine learning engineers challenged by model deployment timelines, inferencing and throughput performance issues or high inferencing cloud costs.

OctoML has demonstrated the potential of the Octomizer with early customer engagements across model architectures and hardware targets. OctoML’s early partners include Computer Vision (CV) and Natural Language Processing (NLP) machine learning teams focused on improving model performance on various targets such as NVIDIA’s V100, K80, and T4 GPU platforms, Intel’s Cascade Lake, Skylake, and Broadwell x86 CPUs, and AMD’s EPYC Rome x86 CPUs. Model performance improvements were at an order-of-magnitude level - for example, a Computer Vision based team worked with OctoML to decrease model latency from 95 milliseconds to 10 milliseconds, unlocking higher throughput and enabling new product feature development.

Accessible through both a SaaS platform and API, the Octomizer accepts serialized models, enables users to select specific hardware targets, and losslessly optimizes and packages models for the selected hardware. By making use of TVM’s state-of-the-art technical performance capabilities, the Octomizer can deliver up to 10 times model performance improvements, enabling deep learning teams to improve model performance, cut inferencing costs, and reduce time and effort for model deployment.

The Octomizer currently makes available all cloud-based CPU and GPU as well as ARM A-class hardware targets, with additional hardware targets identified for early 2021.

As part of its enterprise offerings, OctoML also provides customer-specific hardware target onboarding, which enables internal performance testing and benchmarking and vendor-specific model optimization.