AI Center of Excellence (AI CoE): Meaning & Setup
The adoption of artificial intelligence (AI) is increasing as companies try to capture value from enterprise AI applications. However, according to an IBM survey, challenges such as limited AI expertise, increasing data complexity, and lack of tools for AI development hinder AI adoption for enterprises.1
To reduce AI project failures, organizations need a dedicated unit to oversee AI initiatives. This is where an AI Center of Excellence (CoE) plays a crucial role, already established in 37% of large U.S. companies. An AI CoE enhances coordination, drives innovation, and ensures best practices.2 Here, we explore its purpose, benefits, and key strategies for effective implementation.
What is an AI Center of Excellence (CoE)?
An AI Center of Excellence (CoE) is an organizational structure, or a dedicated team of technical experts that advises, guides, and oversees AI projects in an organization. An AI CoE bridges the gap between executive decision-making and AI implementation by:
- Identifying AI use cases to solve different business problems,
- Determining and enabling the development of the necessary infrastructure for these AI use cases,
- Building implementation roadmap for AI projects, from which tools and technologies to be used, responsibilities, and setting targets and KPIs.
This structure, or team, functions as a central repository of expertise, best practices, and resources, facilitating the alignment of AI initiatives with the organization’s strategic objectives. By ensuring that implementations deliver tangible business value, it contributes to the organization’s overall success.
Why the need for AI CoE?
As technologies like deep learning, generative AI, and computer vision advance, companies face challenges in coordinating AI initiatives, leading to inefficiencies, duplicated efforts, and difficulties in scaling. A CoE streamlines AI implementation, aligning projects with business objectives and ensuring measurable impact.
Beyond operational efficiency, an AI CoE is needed to establish standardized best practices, fosters cross-functional collaboration, and provides a framework for governance and ethical AI use.
What are the benefits of building an AI CoE?
The key benefits of building an AI CoE are:
- Constituting a dedicated center that coordinates AI efforts across the organization,
- Creating a unified vision for AI within an organization which helps consistent and efficient communication between stakeholders,
- Creating a set of standardized practices and processes for AI. This makes scaling AI efforts easier,
- Managing relationships with external parties such as start-ups and universities. This enables companies to benefit from external expertise but also to identify investment opportunities,
- Acquiring and developing AI talent within the organization for long-term success.
What are the best practices for building an AI CoE?
Assess the AI maturity level of your organization
There is no single best formula for establishing an AI CoE, as organizations differ from one another. One key difference between organizations is their AI maturity level, which is the readiness of an organization to take advantage of artificial intelligence technologies. For instance, Gartner’s AI maturity model is as follows (Figure 1):
Figure 1. Gartner’s framework for AI maturity.
Your organization’s maturity level can determine the structure and composition of your AI CoE team as well as the next steps required.
Define the AI CoE Function
Defining the AI CoE’s role and objectives can be a good starting point when building the CoE. Key focus areas include:
- Business Strategy: Align AI with business goals, prioritize use cases, set KPIs, and develop an adoption roadmap.
- Technology Strategy: Design AI-ready infrastructure, establish a build-vs-buy framework, and ensure scalable compute and storage.
- AI Development: Create customer-centric AI solutions, streamline model deployment, and ensure alignment with business needs.
- Cultural Integration: Secure executive buy-in, formalize AI governance, and upskill employees for ethical AI adoption.
- Governance: Implement accountability structures for AI ethics, data security, and responsible AI use.
Assemble a multi-disciplinary team
An AI CoE should contain technical experts, such as data scientists and engineers, as well as business executives and leaders from departments who will adopt AI use cases. Moreover, IT and cybersecurity experts are also crucial to help integrate new technologies into existing structures and to ensure the security of new systems.
There are also other key personnel that can help coordinate AI efforts across an organization. For instance, project managers or procurement specialists are vital depending on whether you will build in-house solutions or work with third-party AI vendors.
Here is an example of a multi-disciplinary team:
Role | Function | Key Responsibility |
|---|---|---|
AI CoE Lead / Director | Oversees AI strategy | Secure executive buy-in, set priorities |
Data Scientist | Develops AI models and data insights | Model training, data analysis |
Machine Learning Engineer | Deploys and optimizes AI models | Ensure model scalability and performance |
AI Architect | Designs AI infrastructure and integrations | System architecture, interoperability |
Business Analyst | Advocates AI adoption, secures funding | Align AI with business goals |
IT / Cloud Engineer | Manages AI platform and cloud infrastructure | Ensure scalability and system availability |
Cybersecurity Specialist | Ensures AI security and compliance | Data protection, risk mitigation |
Project Manager | Coordinates AI projects and stakeholders | Manage timelines, resources, execution |
AI Governance & Compliance Officer | Oversees AI ethics and regulatory compliance | Develop governance policies, monitor risks |
UX Designer / Human Factors Specialist | Optimizes AI interfaces and adoption | Improve user experience, AI usability |
Evaluate the impact of the center periodically
Establishing KPIs and other metrics for AI initiatives would enable organizations to measure the impact of the CoE. It is important to link AI initiatives and their business impact in terms of organizational efficiency, revenue, time, and cost savings.
Such assessments would help organizations evaluate their progress with AI against the above metrics and specifically pinpoint the areas that need improvement.
Provide education to stakeholders
AI is transforming industries and business functions, but the expectations of what AI can achieve can become unrealistically high. It is important that team members in an AI CoE are educated about AI technologies and their potential business benefits. This will inform stakeholders about what AI can and cannot do for their departments.
3 Real life examples for AI CoE
JP Morgan Chase: Machine Learning Center Of Excellence
JPMorgan Chase’s Machine Learning Center of Excellence (MLCOE) is a team of ML experts collaborating with internal business and analytics teams. Their mission is to develop and implement advanced ML solutions using AI innovations. By leveraging data-driven insights and technology, MLCOE enhances business outcomes through a partnership-driven approach, driving efficiency and innovation across the organization.3
Siemens: AI Lab
Siemens’ AI Center of Excellence (CoE) operates through the Siemens AI Lab, advancing Industrial AI with a focus on innovation, training, and scalable AI solutions. It supports AI-driven applications across manufacturing, automation, and digitalization. Siemens also integrates generative AI into its operations via Siemens Xcelerator, combining AI with industrial automation for enhanced efficiency and innovation. 4
IBM and HCLTech: Generative AI Center of Excellence
HCLTech has partnered with IBM to create a new Generative AI Center of Excellence (CoE) leveraging IBM’s watsonx™ AI and data platform. 5 This CoE will be located in HCLTech’s AI and Cloud Native Labs in London and Austin, Texas, and will assist enterprises globally in building custom AI applications, expanding data capabilities, and advancing responsible AI workflows.
By leveraging HCLTech’s Cloud Native Labs and generative AI with IBM’s watsonx™ platform, the CoE helps enterprises modernize legacy systems, simplify coding, and enhance ITSM use cases. It also fosters skill development on watsonx while driving innovation. Additionally, HCLTech clients gain access to IBM’s expertise, enabling them to scale and customize AI solutions efficiently.
For more on the AI Center of Excellence, you can refer to AI consultants who can help you establish centers of excellence in your business.
Frequently Asked Questions (FAQ)
FAQ
Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Comments 0
Share Your Thoughts
Your email address will not be published. All fields are required.