#53 - Evaluating AI Skills / Trust Rubrics / Intelligence Crafting

Link to Folder with Everything

2026.09.09.officehours.53

Recording of the Video

Watch on Google Drive

Summary (by Gemini)

Office hours covered evaluating AI skills and managing risks via structured frameworks.

Evaluating AI Skills

Research identified 140 top skills across 5 business areas using provenance and trust rubrics. Trusted sources reduce risks during implementation.

Managing Skill Sprawl

Project-specific thread configurations contain the scope of active skills and prevent operational confusion. Customization is common for ecosystem adaptation.

Future AI Management

Intelligence crafting shifts software development toward configurations mapping necessary tools and connectors. Companies require structured approaches for alignment.

Next steps

• [Rahul Singh] Share Research Findings: Upload the catalog and research findings to Google Drive for all participants.
• [Rahul Singh] Share Research Prompts: Share the prompts used for the research process with participants.
• [Rahul Singh] Analyze Skill Trends: Investigate why certain skills are trending up or down in the data.
• [Rahul Singh] Share Evaluation Rubric: Share the evaluation rubric used for analyzing business skills.
• [Rahul Singh] Share Research Data: Provide the compiled database of skills and tool rubrics to all meeting participants.
• [The group] Enhance Skill List: Update and expand the shared database of skills and rubric criteria.
• [Rahul Singh] Research Software Tools: Evaluate Appflowy and Notion functionalities to determine their suitability for professional projects.

Details

• Welcome and Purpose of Office Hours: Rahul Singh opened "Office Hours 53" to facilitate a review of current AI tools and technologies, specifically focusing on the emerging "skills" layer in business applications (00:01:34). While new technology does not necessarily make established roles obsolete, it offers new paradigms for how work is conducted (00:05:09). The goal: curating skills to make AI usage more accessible while protecting against risks such as identity theft or data loss (00:09:51).
• Skills as Knowledge Layers: While AI models are powerful, human knowledge of "skills" acts as a crucial layer that provides context the AI currently lacks (00:07:44). Humans identify real-world connections and instruct the AI to execute specific tasks — the primary challenge for users is not access to the AI itself, but access to the right knowledge to utilize it effectively (00:08:51).
• Methodology for Skill Evaluation: To manage the 4,000+ available skills, Rahul employed an AI research process to identify 140 top skills across five key business areas: value discovery, creation, growth, delivery, and operation (00:09:51). A rubric assesses skills on provenance, trust, and maintenance, distinguishing reputable vendors from anonymous or deceptive repositories — filtering out low-scoring, ineffective skills like "soul trace" and prioritizing verified, well-maintained tools (00:12:05).
• Market Gaps and Skill Development Opportunities: The research identified gaps in the current skill market — finance, accounting, security, communications, PR, vendor management, and non-profit fundraising are underserved. Participants were encouraged to consider developing skills in these areas (00:14:22).
• Recommended Skill Sources: Focus on skills from established publishers rather than random, unvetted sources — Anthropic, Corey Haynes, and Matt PCO were highlighted as reliable providers with significant user validation and high download counts (00:16:45).
• Product Development Skills: Responding to E Z's question about supporting product development as a non-engineer, Rahul suggested "product marketing" and "brainstorming" skills to conceptualize and critique ideas before development. Different skills follow different "schools of thought" — some engineering-focused, others marketing-oriented — and should be selected per business goal (00:19:42).
• Risk and Data Management: The skill research, while data-driven, remains "experimental" — caution is needed when adopting new tools, and skills should not overreach by requesting excessive access to tools or data (00:22:29). The ultimate goal is establishing best practices for "scale management" internally and for clients (00:23:33).
• Managing Skill Sprawl: Raphael Szmir asked whether adding numerous unused skills hurts performance. AI agents use indexes of skill names and descriptions, so well-defined skills do not necessarily conflict (00:26:53) — but for company-level usage, create project-specific thread configurations to contain the scope of active skills and prevent operational confusion (00:28:53).
• Evaluating Skill Quality: Look for "router skills" that act as agents to manage tasks efficiently (00:33:59); a good skill uses YAML front matter to define its name and description clearly, and avoids overly long, non-portable files (00:32:30). Users can also prompt the AI to analyze installed skills for poorly constructed or problematic ones (00:34:59).
• Customization and User Intent: Answering Merziyah Poonawala, Rahul noted customization is common because engineers build skills for the ecosystem while non-engineers build for immediate personal intent (00:35:55) — understanding the builder's original intent is crucial for deciding how deeply to integrate a skill (00:39:45).
• 3D and Image Generation Tools: "Blender" skills that convert images into 3D structures show the ecosystem's versatility (00:41:37); image-generation skills are underrated for business use — consistent marketing or cover art — and integrate across platforms like Claude (00:42:58).
• Knowledge Management Strategies: P Murray raised platform-reliance and cost concerns and mentioned switching to AppFlowy — validated as an open-source, Notion-like alternative that runs locally or in the cloud (00:50:07). The advice: store data in portable formats (Markdown, JSON, YAML) to enable synchronization across devices via Git or Google Drive regardless of interface (00:52:21).
• Notion and Operational Views: While Notion can be expensive per user, its tables, views, and extensive API/MCP support make it viable as both a knowledge base and an operational hub for tracking tasks and projects (00:56:34).
• MCP Server and Skill Database: Rahul walked through the Model Context Protocol server and a comprehensive database including the skill-evaluation rubric, to be shared with the group for collective enhancement (01:00:17). Reviewing entries like "Soul Trace" demonstrated how to identify tools not worth the investment — inaccessible API keys, failure to perform as described, or no practical business utility (01:01:47).
• Relevance of SaaS Companies: SaaS companies remain viable so long as they provide tools and skills that integrate directly into the AI environments where users already work (01:03:25).
• Individual vs. Company Harness Structures: Individuals will develop personal harnesses for choosing skills, but companies require a structured approach — a company harness must rely on agreed-upon processes and built-in agents to ensure alignment and prevent organizational failure (01:04:30).
• Intelligence Crafting: Defined as a new professional skill set — understanding a domain and assembling existing tools, process skills, and authentication data to solve problems. Software development shifts away from writing layers of raw code toward creating configurations that map necessary tools and connectors; professionals who master this method of assembly can differentiate themselves (01:05:35).