AI Scouting Initiative – Functional Evaluation of AI Tools (Focus on Business Analysis)
Project duration: 1 month
Brief description
The project supports the client in the strategic planning, selection, and implementation of standardized, AI-enabled software tools for internal processes and in-house use. The focus is a functional analysis for the business unit Business Analysis. To this end, the role-specific tasks of business analysts, coaches, and process managers are prioritized as business-critical use cases and then systematically evaluated and documented in selected AI tools on the basis of synthetic test data.
Supplement
To derive and prioritize the use cases within the scope of the business unit Business Analysis, existing work instructions were analyzed and internal role descriptions were examined with regard to their suitability for support by AI tools. In total, over 35 relevant use cases were identified, 18 of which were prioritized. For each use case, detailed prompts were developed and synthetic test data with strong practical relevance were prepared. The responses of the AI tools were systematically examined and compared to document individual strengths and weaknesses. In addition to purely chat-based user interaction, extended features were evaluated, including custom agent creation and connectors to third-party services (e.g., Microsoft SharePoint). Another focus of the functional evaluation was to assess the user experience (usability, GUI, integration into Office applications) when interacting with AI tools.
Subject description
Three deployment formats are being considered for an internal, general-purpose AI tool: Microsoft Copilot as a product integrated into the Microsoft-based IT landscape of the client, OpenAI ChatGPT as a purely browser-based standalone option and an internally developed GUI/frontend based on AI models from an external Germany-based provider. With regard to the functional focus of the project, data-protection and the financial aspects of long-term licensing are not the immediate priority. However, the results of the functional evaluation are intended to provide a solid, well-founded basis for their future analysis. For this reason, the project offers a substantive foundation for strategic selection and for reducing potential adverse sunk costs for the client.