AI Frameworks#
Open Edge Platform provides its own components but also enables and integrates other industry-standard solutions. Here are some frameworks you may find crucial for your AI systems.
OpenVINO toolkit#
OpenVINO toolkit is an extensive collection of software for developing deep learning applications. It enables you to easily deploy entire industry-grade systems that are highly scalable and optimized for Intel hardware.
Based on OpenVINO inference engine and complimented by tools such as Model Converter and Neural Network Compression Framework, it is a complete software development solution for both conventional and generative AI use cases.
PyTorch#
PyTorch is an optimized tensor library for deep learning using GPUs and CPUs. This industry-standard deep learning framework is fit for both research and production deployment thanks to its flexibility, modular nature, wide integration capabilities, stability, and a rich open source ecosystem.
If you need to build, train, or just tweak your custom model, PyTorch is a good choice.
OpenVINO Model Server#
OpenVINO Model Server enables you to use inference engine capabilities remotely, over standard network protocols. It can be set up to host models and run them based on requests coming from multiple client systems.
Based on OpenVINO inference engine and part of the OpenVINO ecosystem, it offers the same level of scalability and performance, just delivered remotely instead of deployment on edge devices.
vLLM#
vLLM is a highly performant and efficient inference and serving engine for Large Language Models.
It supports Intel GPU for data center, AI, as well as workstation purposes. Based on Python, it offers tensor-parallel inference/serving, with pipeline-parallel serving as an early adoption feature.
SGLang#
SGLang is a high-performance serving framework for large language and multimodal models. It is optimized for Intel® Arc™ Pro B-Series Graphics and Intel® Arc™ B-Series Graphics.
OneAPI#
An API devised by Intel to offer direct-to-metal interfaces to offload accelerator devices.
Llama#
Llama is an engine for LLM inference in C/C++. It is integrated with Open Edge Platform via the Vulkan backend, connected to OpenVINO.