AI Center of Excellence (AI CoE): Meaning & Setup in 2025

The adoption of artificial intelligence (AI) is increasing as companies try to capture value from enterprise AI applications. However, according to an IBM survey, challenges such as limited AI expertise, increasing data complexity, and lack of tools for AI development hinder AI adoption for enterprises.¹

To reduce AI project failures, organizations need a dedicated unit to oversee AI initiatives. This is where an AI Center of Excellence (CoE) plays a crucial role, already established in 37% of large U.S. companies. An AI CoE enhances coordination, drives innovation, and ensures best practices.² Here, we explore its purpose, benefits, and key strategies for effective implementation.

What is an AI Center of Excellence (CoE)?

An AI Center of Excellence (CoE) is an organizational structure, or a dedicated team of technical experts that advises, guides, and oversees AI projects in an organization. An AI CoE bridges the gap between executive decision-making and AI implementation by:

This structure, or team, functions as a central repository of expertise, best practices, and resources, facilitating the alignment of AI initiatives with the organization’s strategic objectives. By ensuring that implementations deliver tangible business value, it contributes to the organization’s overall success.

Why the need for AI CoE?

As technologies like deep learning, generative AI, and computer vision advance, companies face challenges in coordinating AI initiatives, leading to inefficiencies, duplicated efforts, and difficulties in scaling. A CoE streamlines AI implementation, aligning projects with business objectives and ensuring measurable impact.

Beyond operational efficiency, an AI CoE is needed to establish standardized best practices, fosters cross-functional collaboration, and provides a framework for governance and ethical AI use.

What are the benefits of building an AI CoE?

The key benefits of building an AI CoE are:

• Constituting a dedicated center that coordinates AI efforts across the organization,
• Creating a unified vision for AI within an organization which helps consistent and efficient communication between stakeholders,
• Creating a set of standardized practices and processes for AI. This makes scaling AI efforts easier,
• Managing relationships with external parties such as start-ups and universities. This enables companies to benefit from external expertise but also to identify investment opportunities,
• Acquiring and developing AI talent within the organization for long-term success.

What are the best practices for building an AI CoE?

Assess the AI maturity level of your organization

There is no single best formula for establishing an AI CoE, as organizations differ from one another. One key difference between organizations is their AI maturity level, which is the readiness of an organization to take advantage of artificial intelligence technologies. For instance, Gartner’s AI maturity model is as follows (Figure 1):

Figure 1. Gartner’s framework for AI maturity.

Your organization’s maturity level can determine the structure and composition of your AI CoE team as well as the next steps required.

Define the AI CoE Function

Defining the AI CoE’s role and objectives can be a good starting point when building the CoE. Key focus areas include:

• Business Strategy: Align AI with business goals, prioritize use cases, set KPIs, and develop an adoption roadmap.
• Technology Strategy: Design AI-ready infrastructure, establish a build-vs-buy framework, and ensure scalable compute and storage.
• AI Development: Create customer-centric AI solutions, streamline model deployment, and ensure alignment with business needs.
• Cultural Integration: Secure executive buy-in, formalize AI governance, and upskill employees for ethical AI adoption.
• Governance: Implement accountability structures for AI ethics, data security, and responsible AI use.

Assemble a multi-disciplinary team

An AI CoE should contain technical experts, such as data scientists and engineers, as well as business executives and leaders from departments who will adopt AI use cases. Moreover, IT and cybersecurity experts are also crucial to help integrate new technologies into existing structures and to ensure the security of new systems.

There are also other key personnel that can help coordinate AI efforts across an organization. For instance, project managers or procurement specialists are vital depending on whether you will build in-house solutions or work with third-party AI vendors.

Here is an example of a multi-disciplinary team:

Evaluate the impact of the center periodically

Establishing KPIs and other metrics for AI initiatives would enable organizations to measure the impact of the CoE. It is important to link AI initiatives and their business impact in terms of organizational efficiency, revenue, time, and cost savings.

Such assessments would help organizations evaluate their progress with AI against the above metrics and specifically pinpoint the areas that need improvement.

Provide education to stakeholders

AI is transforming industries and business functions, but the expectations of what AI can achieve can become unrealistically high. It is important that team members in an AI CoE are educated about AI technologies and their potential business benefits. This will inform stakeholders about what AI can and cannot do for their departments.

3 Real life examples for AI CoE

JP Morgan Chase: Machine Learning Center Of Excellence

JPMorgan Chase’s Machine Learning Center of Excellence (MLCOE) is a team of ML experts collaborating with internal business and analytics teams. Their mission is to develop and implement advanced ML solutions using AI innovations. By leveraging data-driven insights and technology, MLCOE enhances business outcomes through a partnership-driven approach, driving efficiency and innovation across the organization.³

Siemens: AI Lab

Siemens’ AI Center of Excellence (CoE) operates through the Siemens AI Lab, advancing Industrial AI with a focus on innovation, training, and scalable AI solutions. It supports AI-driven applications across manufacturing, automation, and digitalization. Siemens also integrates generative AI into its operations via Siemens Xcelerator, combining AI with industrial automation for enhanced efficiency and innovation. ⁴

IBM and HCLTech: Generative AI Center of Excellence

HCLTech has partnered with IBM to create a new Generative AI Center of Excellence (CoE) leveraging IBM’s watsonx™ AI and data platform. ⁵ This CoE will be located in HCLTech’s AI and Cloud Native Labs in London and Austin, Texas, and will assist enterprises globally in building custom AI applications, expanding data capabilities, and advancing responsible AI workflows.

By leveraging HCLTech’s Cloud Native Labs and generative AI with IBM’s watsonx™ platform, the CoE helps enterprises modernize legacy systems, simplify coding, and enhance ITSM use cases. It also fosters skill development on watsonx while driving innovation. Additionally, HCLTech clients gain access to IBM’s expertise, enabling them to scale and customize AI solutions efficiently.

Frequently Asked Questions (FAQ)

We have an article dedicated to the myths and misconceptions about AI that businesses should be aware of. Feel free to check it out.

Consultants can help you establish centers of excellence in your business. Feel free to check our data-driven lists of AI consultants and data science consultants.

External Links

• 1. IBM Newsroom - The future of hybrid cloud and AI is here.
• 2. How to Set Up an AI Center of Excellence. Harvard Business Review
• 3. Machine Learning Center Of Excellence.
• 4. AI Lab - Siemens Global.
• 5. HCLTech and IBM Announce Generative AI Center of Excellence to Support Clients with Customized AI Solutions.