How to Use AI to Pull Documentation for Coding

Introduction

a big part of coding is actually getting the documentation needed in order to accomplish the task. Especially if you’re coming from the javascript ecosystem…

Many forget that what we do as software engineers is research and development.

R&D.

We need to take advantage of the libraries that are out there.. but also not go overboard.

Inevitably you will reach for 3rd party tools.

The System

I’ve used Cursor for documentation but find that it is hit or miss.

I created a popular website called GPTsforDevs.com a number of months ago, that pulled all the documentation for popular swift libraries so that I could learn the language.

The problem I was solving was that I wanted to learn the language, but didn’t want to spend hours searching for the documentation.

In our case, we’re going to cover the 3 ways you can grok documentation for your coding needs.

1. Method for Pulling Docs with Cursor

This one is the easiest but also the least effective from my experience.

There’s a couple tactics you can use to get better results.

The recommendedway to add docs to your cursor is by:

Adding it from your settings by clicking the gear icon on the top right of the cursor window.
Click on features
Scroll down to docs
Click add doc

I prefer to use a syntax when naming the docs. The team over at Supabase had a great video on how to do this.

Best Practices for Pulling Docs with Cursor:

A best practice for using cursor docs is to individually add the docs for each section. eg. supabase-js, supabase-realtime, supabase-auth etc etc.
Avoid adding docs as URL’s by simply pasting the URL into the prompt.
When starting a new coding session, ensure you have all the docs you need.
Periodically check to make sure your docs are indexed.

2. Method for Pulling Docs with Aider

Aider lets you pair program with LLMs, to edit code in your local git repository. Start a new project or work with an existing code base. Aider works best with Claude 3.5 Sonnet, DeepSeek V3, o1 & GPT-4o and can connect to almost any LLM.

Aider is a great tool for pulling docs for your coding needs. It’s a bit more complex to set up but once you get the hang of it, it’s a powerful tool.

Aider has a ton of functionality out of the box that I won’t cover here but that you can find out more about here.

What we’re focused on is it’s ability to pull documentation on the fly.

The easiest way is to use the /web command.

From there docs, the web command will “Scrape a webpage, convert to markdown and send in a message”

This is wildly convenient for pulling documentation for libraries.

As a quick aside, there are a ton of other amazing in-line commands you can run…

My favorites are using architect and chat to swap between planning and executing.

This is what I’ll use if it’s something small, for instance how to use the navigation menu component from Shadcn/ui.

3. Method for Pulling Docs with Prompt Chain

This is the most effective way to pull documentation for your coding needs.

It’s a bit more complex to set up but once you get the hang of it, it’s a powerful tool.

First off we’ll use the following repo to be our scraper.

gpt-crawler is by a company called Builder.io

This is a great tool for scraping documentation from websites.

I run it locally, then update the config.ts to point at whichever docs I want.

import { Config } from "./src/config";
 
export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
  maxTokens: 2000000,
};

After it runs, I’ll get a JSON file with all the docs..

But wait! We have a problem. This might be too much information. They’re might be too much output for the AI to handle.

So we’ll need to summarize it and pull out the juicy bits.

That’s where a metaprompt comes in.

Metaprompts are a way to get your AI to “plus up” your existing prompt.

At the time of writing I have not published an article on how to write metaprompts, but be sure to bookmark the site as I add at least one article a day.

So we have our json, now we need to place it inside the following metaprompt.

The Prompt

<purpose> You are an expert technical writer specializing in API documentation analysis and summarization. Your goal is to create comprehensive, well-structured markdown summaries of scraped API documentation. </purpose>
<instructions>
    <instruction>Analyze the provided API documentation thoroughly</instruction>
    <instruction>Organize information into the specified markdown sections</instruction>
    <instruction>Use clear formatting with proper code blocks for examples</instruction>
    <instruction>Verify technical accuracy of endpoint descriptions and parameters</instruction>
    <instruction>Include all required sections with detailed, relevant information</instruction>
    <instruction>Follow the examples to maintain consistent structure and depth</instruction>
</instructions>
<sections>
    <section>
        <name>API Summary</name>
        <description>Brief overview of the API's purpose and main capabilities</description>
    </section>
    <section>
        <name>Endpoints</name>
        <description>
            <endpoint>
                <method>HTTP method</method>
                <path>Endpoint path</path>
                <parameters>List of parameters</parameters>
                <description>Endpoint functionality</description>
            </endpoint>
        </description>
    </section>
    <section>
        <name>Examples</name>
        <description>Code samples for each major endpoint</description>
    </section>
    <section>
        <name>Error Handling</name>
        <description>Common error codes and their meanings</description>
    </section>
    <section>
        <name>Rate Limits</name>
        <description>API usage restrictions and thresholds</description>
    </section>
    <section>
        <name>Pricing</name>
        <description>Cost structure and tier details</description>
    </section>
</sections>
<examples>
    <example>
        <api-type>Weather API</api-type>
        <endpoints>
            <endpoint>
                <method>GET</method>
                <path>/current/{location}</path>
                <parameters>location, units, lang</parameters>
                <example-request>curl https://api.weather.com/current/London?units=metric</example-request>
            </endpoint>
        </endpoints>
        <error-codes>
            <error>
                <code>401</code>
                <message>Invalid API key</message>
            </error>
        </error-codes>
    </example>
    <example>
        <api-type>E-commerce API</api-type>
        <endpoints>
            <endpoint>
                <method>POST</method>
                <path>/orders</path>
                <parameters>items, shipping_address, payment_token</parameters>
                <example-request>curl -X POST https://api.store.com/orders -d @order.json</example-request>
            </endpoint>
        </endpoints>
        <rate-limits>
            <limit>
                <tier>Free</tier>
                <requests>100/day</requests>
            </limit>
        </rate-limits>
    </example>
</examples>
<api-docs> {{api-docs}} </api-docs>
Your markdown documentation:

Copy and paste that into a chain of thought model.

Pick one of the following models based on what you have access to:

o1-pro
o1
o1-preview
DeepSeek R1
Gemini Flash 2.0 Thinking
Claude 3.5 Sonnet

My favorite is o1, but you can use claude 3.5, you just need to make sure you tell it to use chain-of-thought ahead of pasting in the meta prompt.

Now take your output.json and paste it in between the <api-docs> tags.

Now you can ask the AI to summarize it for you.

What you end up with is a markdown file that you can use to pull documentation for your coding needs.

Conclusion

I hope you found this article helpful. If you have any questions, please feel free to reach out to me on X.