Building Organizational Intelligence – O’Reilly

Introduction

Not way back, one in every of my engineering administrators got here to me with a request: His workforce appeared overloaded, and he needed to rent one other engineer. I made a decision to check a analysis assistant I had been constructing—an AI agent related to our inner techniques through MCP—by asking it to investigate the workforce’s workload and write a hiring case.

What got here again was thorough. Headcount, service possession, dash velocity, ticket backlog, and capability allocation, all of it neatly summarized. However studying via the doc, I felt the identical frustration I’d felt with each AI-generated organizational report that’s come throughout my desk. It informed me what was taking place with out serving to me perceive why, or what I ought to really do. It was organized across the knowledge quite than across the resolution. In brief, it was the form of response that’s simple to agree with and troublesome to behave on.

Then I added another factor to the configuration: the O’Reilly Knowledgeable MCP server. I reran the identical evaluation and requested a barely totally different query: “How would the specialists on O’Reilly assessment this request?”

As an alternative of main with headcount and ticket counts, the output now opened with a discovering: “The operational overhead drawback is structural, not a staffing deficiency.” Citing the Google SRE framework’s idea of operational toil, it famous that the workforce was working at roughly 67% toil, effectively above the edge at which the SRE literature recommends structural intervention, and made particular, concrete suggestions: run a toil audit, set specific discount targets, and assign operational runbook possession. This wasn’t a advice for whether or not to rent or not. It was a grounded, traceable argument for doing one thing else as a substitute.

That distinction—between a knowledge abstract and an expert-grounded advice—is what this paper is about.

What follows is a case research of how we constructed an organizational intelligence system at O’Reilly, utilizing our personal platform as a core part. The strategy I describe is grounded in engineering as a result of that’s the place I work, however it generalizes to any operate the place necessary data is scattered throughout a number of techniques and necessary selections require synthesizing all of it. The recipe has 4 steps: map your info hierarchy; join these techniques to an LLM through MCP and write a ability file that defines the way it ought to purpose; add the O’Reilly Knowledgeable MCP as an professional assessment layer that grounds the evaluation in established frameworks; and construct a light-weight system for human-in-the-loop assessment. I’ll clarify every step intimately and make the case for why the third step is the one which adjustments every part.

Why organizational intelligence is getting more durable

To know the issue this strategy solves, it helps to look briefly at how engineering has modified over the previous three many years. These forces have performed out first and quickest in engineering, however as AI instruments proliferate past the engineering workforce, the underlying dynamic of extra output, extra selections, and extra scattered info is spreading to each a part of the group.

Within the waterfall period of the Nineties, software program organizations ran on central plans. Every thing was specified up entrance, and leaders maintained visibility exactly as a result of all info flowed via a single coordinating doc. The plans have been brittle and sometimes fictional by the point they have been executed, however not less than everybody knew what was presupposed to be taking place.

Agile changed central plans with small, autonomous groups working briefly sprints, and this solved the reliability drawback whereas making a visibility drawback. Vital selections started taking place domestically and shortly—the fitting groups making the fitting calls—however the info wanted to see throughout all of these selections splintered into dozens of separate instruments. Product technique lived in a single system, challenge execution in one other, code in a 3rd, and repair possession in a fourth. Extra issues obtained shipped, however the big-picture view obtained more durable to take care of.

The agentic period has intensified this dynamic dramatically. Particular person engineers at this time can ship in a day what used to take a full dash workforce. The output is extraordinary, however the visibility is almost gone.

Any effort that spans a number of groups, resembling a platform migration, a shared infrastructure change, or a reorganization, now requires monumental coordination overhead just because the knowledge decision-makers want to grasp the total image is distributed throughout too many locations. And this isn’t an issue distinctive to engineering. It exists in any operate that runs on knowledge unfold throughout a number of techniques.

Confronted with this visibility drawback, I needed to construct one thing I might ask big-picture questions and get synthesized solutions again shortly. Issues like:

• What’s the standing of this cross-team migration effort, and which groups are behind?
• A workforce appears overloaded. Do they really want one other engineer, or is one thing else happening?
• What are the trade-offs of adopting this new infrastructure know-how?
• Assist me produce a scope assertion from this product temporary.

Constructing one thing that might reply these effectively took two foundational steps, and getting it to supply suggestions based mostly on my particular enterprise context took two extra. Whereas my particular instruments are from engineering, the construction applies equally to a gross sales workforce synthesizing CRM knowledge and market analysis, or a finance workforce working throughout an ERP, a planning software, and exterior benchmarks.

Step 1: Map your info hierarchy

Each group has a set of techniques the place necessary data lives, and people techniques type a pure hierarchy that spans from strategic intent on the prime to operational element on the backside. Earlier than you may construct a helpful analysis assistant, you could make that hierarchy specific, as a result of it’s the map of how selections get made, which sources carry essentially the most authority, and the way totally different sorts of questions ought to be approached.

At O’Reilly, our engineering hierarchy appears to be like like this:

Your group can have a unique set of instruments. A gross sales group may place Salesforce on the prime, adopted by a income intelligence platform, advertising and marketing automation, and market analysis. A authorized workforce may begin with a contract administration system, adopted by a regulatory tracker, inner coverage documentation, and a analysis database. The particular techniques matter lower than the act of mapping them: understanding which layer solutions which form of query, and which sources take priority after they battle.

Step 2: Join your techniques through MCP and write a ability that describes purpose

This step has two components that should work collectively. First, you could join your techniques to your AI instruments through MCP. Then you must write a ability file that tells the mannequin what to do with that entry. At O’Reilly, we name this whole grounding layer Expert Intelligence.

Configuring MCP is simple. Most main instruments now provide MCP connectors, and connecting them is usually a matter of routine JSON configuration. For techniques with out MCP connectors, a bash-capable agent with curl and jq can usually attain a REST API immediately. MCP simply makes it cleaner and extra dependable.

However MCP connections alone aren’t sufficient, and that is the half most implementations get incorrect. MCP offers the agent entry to your knowledge, however it doesn’t inform the agent use it successfully. With out specific steering, the agent retrieves info and organizes it the best way the underlying techniques set up it, which produces a knowledge dump, not an evaluation.

The ability file—a CLAUDE.md or SKILLS.md doc that gives particular reasoning directions—transforms retrieval into evaluation. Mine defines the reasoning hierarchy (which techniques to seek the advice of for which sorts of questions, and weigh them), the output format (this isn’t a coding agent—it produces reviews and proposals, not code), epistemic requirements (present your work, title gaps, floor assumptions for human verification), and tone. On that final level, I borrowed probably the most helpful directions from Ted Lasso: “be curious, not judgmental.” Including it meaningfully improved the standard of the output.

The ability is a codified model of how a talented analyst would strategy these questions. It encodes your group’s reasoning course of and makes it repeatable.

Step 3: Add the professional layer

With the analysis assistant related to our inner techniques, I had one thing genuinely helpful: quick, synthesized solutions to questions that beforehand would have taken days to analysis. However I saved operating into the identical drawback: The reviews felt generic, and folks didn’t belief them. This problem factors to a basic limitation of AI-generated organizational evaluation that goes past any explicit implementation.

The generic evaluation drawback

Common-purpose AI assistants have a tendency to provide a recognizable form of organizational evaluation: technically affordable, balanced, cautious, and finally not very helpful. This isn’t primarily a failure of data—each main LLM has absorbed an infinite quantity of administration and organizational pondering. It’s a failure of grounding. When an AI assistant has no particular framework anchoring its response, it tends to provide suggestions broad sufficient to use to virtually any state of affairs: contemplate the trade-offs, weigh your choices, and guarantee alignment throughout stakeholders. These responses are laborious to disagree with and simply as laborious to behave on.

When a report says, “The workforce seems overloaded. Think about including headcount,” it’s not incorrect. However that advice might apply to virtually any workforce in virtually any firm! It gained’t make a director change their thoughts, and it’s not one a management workforce can debate, refine, and act on.

What occurred after I added the professional layer

Calling on the O’Reilly Knowledgeable MCP didn’t present the mannequin with new info—a lot of the info was technically out there already. Nevertheless, with out the Knowledgeable MCP and related abilities, the mannequin couldn’t use that info for something however the broadest analyses. Incorporating the Knowledgeable MCP and related abilities modified the character of the analyses by grounding them in particular frameworks, citing named authors and thresholds, and organizing their conclusions round established our bodies of practitioner data quite than basic rules.

To make this concrete, right here’s the form of output the analysis assistant produced earlier than including the Knowledgeable MCP:

The workforce seems overloaded. The backlog is giant and the migration challenge is consuming vital dash capability. Think about including headcount or lowering scope.

And right here’s what it produced after:

In accordance with Google’s SRE steering, sustained operational toil above roughly 50% signifies structural inefficiency quite than a staffing scarcity. This workforce’s telemetry suggests roughly 67% operational toil. Hiring one other engineer would probably enhance whole toil until operational possession is first decreased. Really helpful actions: run a structured toil audit, set an specific toil-reduction goal under 50%, and assign runbook possession for recurring operational duties.

The second report cites a framework by title, references the precise threshold that framework establishes, applies it to the workforce’s precise knowledge, reaches a unique conclusion than the plain one, and makes actionable suggestions. It’s the form of evaluation that adjustments a dialog as a result of the director can see the place the conclusions got here from, interact with the reasoning, push again on the framework in the event that they disagree, or settle for it with confidence that it was reasoned quite than pattern-matched.

Once I shared this model with my engineering director, their response was instant: That is defensible.

Frameworks aren’t info

Essentially the most underappreciated side of O’Reilly’s content material library is that the worth isn’t primarily informational. Many of the info in an O’Reilly ebook can be found on the web, and LLMs have already learn a lot of the web.

The deeper worth of O’Reilly’s catalog is that it’s organized round coherent frameworks—full psychological fashions constructed by practitioners who spent years or many years growing them. Google SRE. Crew topologies. Speed up. Area-driven design. The Supervisor’s Path. Wardley mapping. Designing Knowledge-Intensive Functions. These are structured methods of interested by particular lessons of issues, developed with sufficient rigor that they will really information selections.

Frameworks are distinct from info in a crucial manner: They inform you not simply what’s true however what’s related, what to measure, what threshold issues, and what to do whenever you exceed it. A mannequin with entry to the SRE framework as an organized physique of practitioner data is extra more likely to floor it explicitly, apply it to the precise query at hand, and use it to anchor its suggestions, producing output that human reviewers can really interrogate.

This factors to the organizing precept behind the strategy described on this paper:

Organizational knowledge supplies native proof about what is occurring in your particular context. Knowledgeable frameworks present gathered practitioner data about how to consider issues of that sort. Good organizational judgment requires each.

The Knowledgeable MCP is the bridge between your particular enterprise context and practitioner insights. It connects the AI’s entry to your inner techniques with a curated physique of experience related to the choices your group must make.

Why use MCP quite than importing your individual paperwork

The pure objection at this level is “Couldn’t I get the identical impact by dumping related PDFs into Claude, or utilizing Claude Initiatives, or NotebookLM?”

The quick reply is just not fairly, and the explanations are sensible as a lot as they’re technical.

Importing paperwork offers you retrieval from these particular paperwork. The O’Reilly Knowledgeable MCP differs in a number of operationally vital methods. First, the corpus is editorially curated round coherent practitioner frameworks. Not like a set of PDFs, which tends to replicate no matter you occurred to search out, the Knowledgeable MCP gives a sustained curatorial perspective: The authors are vetted, the content material has been via editorial assessment, and it’s organized round established our bodies of data quite than assembled advert hoc. It is a way more expansive form of proof base. Second, the corpus is maintained and up to date by O’Reilly. New titles are added, new editions change outdated ones, and the content material stays present with none administration in your half. Third, the Knowledgeable MCP is configured as soon as and works persistently throughout your complete group and toolchain quite than being tied to a single consumer’s Claude Mission or a doc add that expires. Lastly, accessing content material via a correct API respects the suitable utilization phrases in a manner that importing copyrighted texts doesn’t.

And when paired with a well-written ability, the agent could be directed to look explicitly for competing frameworks, floor instances the place the literature disagrees, and title gaps within the out there proof, offering a significant verify towards the widespread tendency of AI instruments to quietly favor no matter framework first appears to suit. That’s one thing you may encourage with any retrieval setup, however it works extra reliably when the underlying corpus is organized round coherent our bodies of thought quite than a heterogeneous assortment of paperwork.

What we’re not claiming

I need to be clear concerning the limits of what Knowledgeable MCP does at this time. O’Reilly doesn’t declare that Knowledgeable MCP robotically selects the only appropriate framework for each state of affairs, or that including it to your configuration produces consultant-quality evaluation with out considerate prompting and human assessment.

The outcomes described on this paper have been the end result of all 4 components—the inner organizational knowledge, the rigorously designed ability structure, the Knowledgeable MCP, and human assessment—together working collectively.

The Knowledgeable MCP is a vital differentiator, however it’s not a magic layer you may add to an in any other case generic setup and anticipate to breed these outcomes. The system works as a result of every ingredient does one thing the others can’t. The ability defines the reasoning course of, the inner MCP connections present the organizational proof, the Knowledgeable MCP supplies the professional frameworks, and human assessment provides the judgment and context that no AI system can generate by itself.

What the Knowledgeable MCP reliably contributes to that system is entry to a curated physique of practitioner data: technical and managerial frameworks which are editorially organized round coherent our bodies of thought and troublesome to reconstruct from scattered internet content material or assembled doc collections. Your organizational knowledge nonetheless tells you what’s taking place, whereas the O’Reilly Knowledgeable MCP helps interpret what it means. That’s a significant and concrete enchancment over an ungrounded AI assistant, and it’s one thing you may put in manufacturing and construct on at this time.

A be aware on hallucinations

No AI system eliminates the chance of hallucination. The Knowledgeable MCP doesn’t make the mannequin infallible.

What it does is change the burden of proof. When each advice is grounded in a named framework, a named creator, and a traceable quotation, a human reviewer can verify the reasoning quite than merely accepting or rejecting a conclusion. The query shifts from “Is that this proper?” (unanswerable in isolation) to “Does this framework really say this, does it apply right here, and do I agree with the conclusion?” That’s a query people can interact with productively, which is strictly what you need from a decision-support software.

Step 4: Human assessment is nonnegotiable

Organizational techniques not often include the total context behind a choice. The assembly that modified every part occurred final Tuesday and hasn’t been written up but. A key particular person is quietly planning to depart. A strategic course shifted in a dialog that was by no means documented. AI can synthesize every part in your techniques with exceptional constancy, however it could’t know what isn’t there, and organizational actuality adjustments sooner than documentation does.

Extra basically: AI can determine trade-offs, however it could’t determine which trade-offs matter. That judgment requires human data of context, priorities, and threat tolerance that may’t be totally encoded in any system. The aim isn’t to take away people from the loop however to offer them better-structured enter to purpose from.

Lengthen the professional layer by fixing collaboration

As I began sharing analyses extra broadly, I bumped into a brand new set of limitations within the collaboration layer. The analysis assistant produced paperwork. I shared them in Google Docs, and folks added feedback, however when the AI up to date a doc based mostly on reviewer suggestions, I needed to paste in a brand new model, which worn out the present feedback. Paperwork proliferated with out clear relationships between them, and the AI had no visibility into the discussions within the feedback, which was the place a very powerful context and pushback lived.

To unravel the collaboration drawback, I labored with one in every of our engineering administrators to construct what we name Superanswers, a system that makes use of GitHub because the supply of reality for AI-generated analysis paperwork and their related discussions.

The structure is simple: Paperwork are saved as Markdown recordsdata in a GitHub repository, a GitHub Pages website renders them with a clear interface that helps inline commenting, and all dialogue occurs in GitHub Discussions, which means each remark, query, and revision is versioned and traceable. As a result of the paperwork and their discussions reside in GitHub, Claude Code has full entry to each. It may well learn the doc content material plus the whole dialog that’s developed round it.

This permits a qualitatively totally different form of AI participation. As an alternative of producing a doc and stepping again, we will now ask:

What’s the consensus round this challenge based mostly on the dialogue thus far? What questions stay unresolved? Incorporate the reviewer feedback and produce an up to date model.

The AI turns into a participant in an ongoing dialog quite than a one-shot report generator, which meaningfully shifts how organizational data will get constructed and refined.

What groups are utilizing Superanswers for

As Superanswers has unfold throughout our engineering group, the vary of questions folks carry to it has been broader than I anticipated:

None of those questions is about writing code. They’re about understanding a company, making selections, and coordinating work, and most of them would map naturally onto the issues of leaders in different features. The identical questions come up in any group navigating speedy change with info scattered throughout too many locations.

use the recipe

The AI dialog up to now has been dominated by a specific set of questions. However there are extra attention-grabbing questions we ought to be asking.

The challenges outlined on this chart aren’t distinctive to engineering. They exist wherever necessary info is scattered throughout a number of techniques and necessary selections require synthesizing all of it.

Particular person productiveness issues, however organizations don’t succeed by having contributors go sooner in arbitrary instructions. They accomplish that by making good selections about the place to speculate, allocating sources effectively, surfacing issues earlier than they compound, and constructing institutional data that persists over time.

The recipe I’ve described will help organizations make these selections and construct that data.

The recipe for constructing an organizational intelligence system:

1. Map your info hierarchy. Determine the techniques the place necessary data lives in your group, from strategic intent all the way down to operational element. That is an organizational process, not a technical one, and doing it effectively requires understanding how selections really get made.

2. Join these techniques through MCP and write a ability that describes purpose. The MCP connections give the AI entry to your knowledge; the ability file tells it assume with that knowledge. With out the ability, you get retrieval. With it, you get evaluation.

3. Add the O’Reilly Knowledgeable MCP as an professional assessment layer. Organizational knowledge supplies native proof about what is occurring in your particular context; professional frameworks present gathered practitioner data about purpose about issues of that sort. This step bridges the 2. The O’Reilly library spans engineering, administration, knowledge science, safety, finance, product, and extra, organized not as a set of info however as coherent frameworks developed by practitioners who spent careers constructing them. The result’s evaluation grounded in named frameworks with traceable citations, one thing human reviewers can interact with and query, quite than generic recommendation they will solely settle for or reject.
4. Construct a light-weight system for human-in-the-loop consensus. AI-generated evaluation is a place to begin, not an finish level. You want a mechanism for folks to assessment, problem, and refine what the AI surfaces, one the place these discussions develop into a part of the context the AI can be taught from in subsequent iterations.

The largest sensible lesson I took from this work is reframing what AI is definitely for in an organizational context. The distinction between a helpful AI analysis assistant and a generic one isn’t primarily about which mannequin you utilize or how a lot knowledge you feed it. It’s about whether or not the reasoning combines native organizational proof with established professional frameworks. Your knowledge tells you what occurred. Knowledgeable frameworks assist interpret what it means. That mixture, with human judgment utilized on the finish, is what makes the distinction between a report that will get learn (perhaps) and filed away and a advice that adjustments a choice.