Copilot Studio Kit is a free, open-source toolkit developed by Microsoft’s Power CAT (Power Customer Advisory Team). Built on top of Microsoft Copilot Studio, it brings together a set of tools for testing, prompt development, governance, and analytics—areas where the core product has limited built-in capabilities.

So, what exactly does this kit offer? Let’s take a closer look at each feature.

Webchat Playground

This feature lets you visually customize your agent’s web chat interface. You can adjust colors, fonts, avatars, thumbnails, and more, all with a live preview. The interface consists of three panels: saved themes on the left, a live preview in the center, and JSON or HTML code output on the right. You can directly upload the HTML output to your web server. Accessibility checks are also built in.

Prompt Advisor

A tool that helps you analyze and improve your prompts. Simply enter your prompt and click “Analyze” to receive a confidence score between 0–100, along with a detailed evaluation. It also suggests optimized alternative prompts using techniques such as zero-shot, few-shot, and chain-of-thought. You can select one and continue refining it using the “Apply and Improve” option.

The confidence score can be interpreted as follows: scores between 80–100 indicate a well-structured prompt that is ready to produce accurate results; scores between 50–79 represent a reasonable prompt with room for improvement; scores between 0–49 signal an insufficient prompt that requires significant changes.

Note: Prompt Advisor uses AI Builder credits.

Adaptive Cards Gallery

Provides ready-to-use Adaptive Card templates for various scenarios. Each template includes sample data along with backend implementation details. With the live preview feature, you can instantly see how the card will look. Using the “Open in Copilot Studio” option, you can directly review and copy the backend implementation for the relevant scenario.

SharePoint Synchronization

Synchronizes selected SharePoint content into your agent’s knowledge base as files at regular intervals. It runs automatically, but you can also trigger it manually using the “Sync Files” option in the agent configuration. You can verify your setup using the “Validate Connection” feature.

Compared to adding SharePoint as a direct data source, this method supports files up to 512 MB, provides lower response latency, indexes non-text elements in PDFs such as tables and charts, and offers broader file type support.

Conversation Analyzer

Allows you to generate insights by analyzing your agent’s conversation transcripts using custom prompts. It includes two built-in prompts: Sentiment Analysis and Personal Data Analysis. You can also create, save, and reuse your own prompts for future analyses. Results include the conversation ID, summary, duration, the prompt used, and a side-by-side transcript view.

Agent Inventory

Provides visibility into all custom Copilot Studio agents across the tenant. For each agent, you can view details such as creation date, publication status, authentication type, knowledge sources, prompt usage, orchestration type, and generative AI feature usage. The dashboard includes an agent growth timeline, a detailed agent view, a filtered list view, and usage metrics for the last 30 days. All data can be exported.

Agent Review Tool

Scans your agents for issues and anti-patterns that may impact performance or security. Findings are presented in a clear format with severity levels and guidance on how to resolve each issue.

Test Capabilities

Allows you to create tests and test sets for your agents and run them in bulk. Test results include latency metrics, observed responses, and execution-level aggregated data. Supported test types cover response matching, attachment matching, topic matching, and generative responses.

Automated Testing with Pipelines

Automates the testing and deployment of custom agents using Power Platform Pipelines. Agents must successfully pass test runs before being promoted to the production environment. By integrating Power Automate flows with Dataverse and Copilot Studio Kit, you can introduce a quality gate into your deployment process and support continuous delivery practices.

Conversation KPIs

Collects conversation outcome data per agent in Dataverse. Tracked metrics include session count, turn count, and conversation outcomes such as resolved, partially resolved, escalated, or abandoned. The process runs automatically twice a day and can also be triggered on demand. It includes sample Power BI reports, and you can define custom variables such as NPS and store transcripts alongside KPI records.

Agent Value

A modular tool for classifying Copilot Studio agents based on type, behavior, and value. It helps organizations better understand each agent’s role, align agents with strategic goals, and measure the value they deliver. Results are visualized in the Agent Value dashboard.

Conclusion

Copilot Studio Kit is a free, open-source toolkit that effectively fills key gaps in the core product. Whether you’re an agent developer or a tenant administrator, it delivers meaningful value. The kit continues to be actively developed by the Power CAT team.