Why Should You Use Cloud Inference (Inference as a Service) in 2024?
Figure 1. Popularity of the keyword “cloud inference” on Google search engine worldwide in the past 5 years.
Deep learning models have emerged as powerful tools in areas like speech recognition and image classification, even surpassing human performance in tasks such as object classification and real-time strategy games.1 However, their effectiveness hinges on the availability of substantial training datasets and inference, necessitating considerable computational resources. Cloud inference presents itself as a viable solution to meet these demanding requirements.
This article delves into the concept of cloud inference, contrasts it with on-device (at source) inference, and explores the various benefits and challenges associated with cloud-based inference systems.
What is inference?
Figure 2. Visualization of how inference works.
Inference is the phase where a model trained is used to make predictions or decisions based on new data. The new data is passed through the trained model, which starts to use the learned weights to make predictions (run inference) without further adjustment.
For example, if your trained model consists of cat images and you feed a new image to the model, it should be able to tell you if there’s a cat in the image or not. However, there comes a challenge; this process involves billions of parameters, requiring high computational resources.2 This is where cloud inference comes in.
What is cloud inference/inference as a service?
Meta has boosted its infrastructure capacity by 250% to accommodate the increasing demand for machine learning (ML) inference.3 Requirements of the inference process led organizations to look for alternatives.
Cloud inference refers to the deployment of machine learning models on cloud platforms, allowing users to access powerful computational resources remotely. This service is particularly beneficial for applications requiring significant computational power or specialized hardware, like GPUs or TPUs. These are often expensive and difficult to maintain locally.
Cloud inference vs. inference at source
Source: SoftmaxAI4
Figure 3. Visualization of how and where edge vs cloud inference differs.
The key difference between inference as a service and at source lies in their deployment models. Inference at source, or edge inference, happens directly on the device where data is generated (like an IoT device). In contrast, inference as a service relies on cloud-based resources, where data is sent to the cloud for processing and then returned to the source.
Advantages of cloud inference
1- It is time sensitive
Cloud-based solutions often offer lower latency due to the high-performance compute capabilities available in the cloud. This is an advantage for applications where real-time results are needed (e.g., fraud detection).
2- It works anywhere
Cloud-based solutions are inherently global, offering the advantage of location independence and the ability to operate across different geographical regions.
3- It has a strong battery life
By offloading heavy computational tasks to cloud servers, there is a significant reduction in the power consumption of local devices.
Key challenges of cloud inference
1- Data leaks
Consider the operation of recommendation systems; they are developed through training on substantial volumes of data, much of which is personal in nature. When inference is conducted utilizing this trained dataset and the process is executed in a cloud environment, there arises a potential risk that the data could be accessed by unauthorized parties.
2- Attacks
Models used in cloud inference are susceptible to various forms of cyberattacks. For instance, most current cloud inference attacks are targeted at images.5 These attacks can compromise the integrity and confidentiality of sensitive data.
Strategies to prevent attacks
To safeguard against attacks in cloud-based inference systems, the following strategies can be implemented:
- Cryptographic methods: They are used to prevent unauthorized replication of training data and access to the model. An example would be encrypting the cloud-based models. Encryption acts as a barrier, ensuring that even if an attacker gains access to the model, they cannot easily understand or replicate the training data.
- Controlled noise: You can minimize the risk of information leakage from the model’s output by adding designed noise to the output vector of the model. This will make it more difficult for attackers to extract meaningful data.
- Prevent overfitting: Overfitting makes a model too tailored to the training data, potentially revealing sensitive information. By randomly removing some connections, or edges, during training, the model becomes less prone to overfitting.
3- Costs
Another major challenge in cloud inference systems is efficiently deploying machine learning models over large areas while meeting the demands for latency and accuracy in a cost-effective way. These systems are resource-heavy, both in terms of computation (for instance, up to 192.2 TFLOPs for BERT-Large) and bandwidth (as high-resolution images may be needed). Consequently, the costs can escalate significantly with a large number of inference queries.
4- Network delays
Despite aiming for low latency, cloud inference can be hindered by bandwidth limitations and network delays. These challenges are especially pronounced for those handling high-resolution data, where quick and efficient data processing is crucial.
If you need assistance in choosing between edge vs cloud inference, don’t hesitate to contact us:
External Links
- 1. Joshi, Praveen; Hasanuzzaman, Mohammed; Thapa, Chandra; Afli, Haithem; Scully, Ted (2023). “Enabling All In-Edge Deep Learning: A Literature Review“. IEEE. Retrieved January 31,2024.
- 2. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., … & Fiedel, N. (2023). “Palm: Scaling language modeling with pathways.” Journal of Machine Learning Research, 24(240), 1-113. Retrieved January 31,2024.
- 3. Li, B., Samsi, S., Gadepally, V., & Tiwari, D. (2023). “Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service.” In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1-15). Retrieved January 31,2024.
- 4. “Edge AI vs Cloud AI” SoftmaxAI. Retrieved January 31,2024.
- 5. Gong, X., Chen, Y., Wang, Q., Wang, M., & Li, S. (2022). “Private data inference attacks against cloud: Model, technologies, and research directions.” IEEE Communications Magazine, 60(9), 46-52. Retrieved January 31,2024.
Cem is the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per Similarweb) including 60% of Fortune 500 every month.
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Sources:
AIMultiple.com Traffic Analytics, Ranking & Audience, Similarweb.
Why Microsoft, IBM, and Google Are Ramping up Efforts on AI Ethics, Business Insider.
Microsoft invests $1 billion in OpenAI to pursue artificial intelligence that’s smarter than we are, Washington Post.
Data management barriers to AI success, Deloitte.
Empowering AI Leadership: AI C-Suite Toolkit, World Economic Forum.
Science, Research and Innovation Performance of the EU, European Commission.
Public-sector digitization: The trillion-dollar challenge, McKinsey & Company.
Hypatos gets $11.8M for a deep learning approach to document processing, TechCrunch.
We got an exclusive look at the pitch deck AI startup Hypatos used to raise $11 million, Business Insider.
To stay up-to-date on B2B tech & accelerate your enterprise:
Follow on
Comments
Your email address will not be published. All fields are required.