Building a Medical AI Application with Grok 4

Grok 4 is one of the most advanced AI models available today, featuring native tool use, advanced agentic reasoning, and reliable API integration. Unlike some AI models that have tool capabilities added as an afterthought, Grok 4 was trained with tools from the start. This means you can build workflows that are more accurate and complex than some simpler AI solutions.

In this tutorial, we’ll build a medical prescription analyzer to explore these capabilities. Users can upload a prescription image, and the app will automatically extract medical data, provide dosage information, display prices, and offer direct purchase links. We’ll use Grok 4’s image analysis to read prescriptions, its function calling to trigger web searches, and Firecrawl’s API to scrape medicine information from pharmacy websites.

In this section, we will set up an xAI account, purchase some credits, generate an API key, and use the xAI SDK to run text-based and image-based medical prescription examples.

Go to console.x.ai and sign up for an account. Once logged in, navigate to the “Billing” tab and add least $5 worth of credits using a debit or credit card.

Next, click on the “API Keys” tab and generate your API key.

Save the API key as an environment variable in your local system:

export XAI_API_KEY="your_api_key"

Install the xAI Python SDK. You’ll need Python 3.10 or higher.

pip install xai-sdk

We will first test the simple Grok 4 workflows to understand how the API works. We will create the xAI client using the API key:

import os
from xai_sdk import Client
from xai_sdk.chat import system, user

# Initialize the client with extended timeout
client = Client(
    api_key=os.getenv("XAI_API_KEY"),
    timeout=3600,  # Extended timeout for reasoning-intensive tasks
)

Initialize the chat with the system prompt:

chat = client.chat.create(
    model="grok-4",
    messages=[
        system("""
        You are MedGuide AI, a helpful and intelligent assistant that helps users understand their medical prescriptions.
        You explain each medicine's price, availability, and prescribed duration in a clear and concise manner.
        """),
    ],
)

Provide the user prompt and ask about Paracetamol and Azithromycin:

# Append a user prompt simulating a text-only prescription input
chat.append(
    user("""
My prescription says:
- Paracetamol 500mg, take twice a day for 5 days.
- Azithromycin 250mg, once a day for 3 days.
Can you explain the duration and check if these medicines are commonly available?
""")
)

Now, we will generate the response and display the reasoning content, the final response, the number of completion tokens, and the number of reasoning tokens. It will take a few seconds for it to complete.

# Get the response from Grok
response = chat.sample()

# Print detailed response information
print("Reasoning Content:")
print(response.reasoning_content)

print("\nFinal Response:")
print(response.content)

print("\nNumber of completion tokens:")
print(response.usage.completion_tokens)

print("Number of reasoning tokens:")
print(response.usage.reasoning_tokens)

Please note that Grok 4 does not currently support reasoning traces (reasoning_content). If you want to obtain the model’s step-by-step thinking (reasoning trace), you need to use Grok-3-mini or Grok-3-mini-fast. As a result, the reasoning content may return empty even if 289 reasoning tokens were used.

Reasoning Content:


Final Response:
Below, I'll explain your prescription based on the details you provided. As MedGuide AI, I'll cover the prescribed duration for each medicine, their common availability (based on general knowledge in most countries like the US, UK, or India—availability can vary by location and regulations), and approximate prices (these are estimates and can fluctuate based on brand, location, pharmacy, and whether it's generic or branded; I recommend checking with a local pharmacy for exact details). Remember, I'm not a substitute for professional medical advice—always follow your doctor's instructions and consult them if you have questions.

### 1. **Paracetamol 500mg**
   - **Prescribed Duration**: Take one tablet twice a day (e.g., morning and evening) for 5 days. This means a total course of 10 tablets (2 per day × 5 days). It's commonly used for pain relief, fever, or mild inflammation.
   - **Availability**: Very commonly available over-the-counter (OTC) in most pharmacies, supermarkets, and online stores (e.g., via Amazon or local apps like CVS or Boots). No prescription is typically needed for this strength, but it's always good to confirm with your pharmacist.
   - **Approximate Price**: Around $0.10–$0.50 per tablet (generic versions are cheaper). A pack of 10 tablets might cost $1–$5 (e.g., $2–$3 in the US for a generic brand like Tylenol equivalent).

### 2. **Azithromycin 250mg**
   - **Prescribed Duration**: Take one tablet once a day for 3 days. This means a total course of 3 tablets. It's an antibiotic often used for bacterial infections like respiratory issues—complete the full course even if you feel better to avoid resistance.
   - **Availability**: Commonly available but usually requires a prescription from a doctor, as it's an antibiotic. You can find it at most pharmacies (e.g., Walgreens, local chemists) or online with a valid prescription. It's widely stocked due to its common use.
   - **Approximate Price**: Around $1–$5 per tablet (generic is cheaper). A pack for a 3-day course might cost $3–$15 (e.g., $5–$10 in the US for a generic like Zithromax equivalent).

If you need help calculating the total quantity to buy or have questions about side effects, interactions, or alternatives, feel free to provide more details. Stay healthy!

Number of completion tokens:
507
Number of reasoning tokens:
289

In this section, we will test the image reasoning capabilities by providing the model with an image from the Handwritten Medical Prescriptions Collection, available in the Kaggle dataset. Save it to your computer for the next step.

We will initiate a chat instance, provide it with the system prompt, encode the image into base64 format, and then give the chat a user prompt that includes the image in base64 format (note the file location). Finally, we will generate the response.

import base64
from xai_sdk.chat import image

# Set the path to your prescription image
image_path = "data/image1.jpg"

# Start a new chat with Grok 4
chat = client.chat.create(model="grok-4")

# Define the AI's role and behavior
chat.append(
    system(
        "You are MedGuide AI, a helpful and intelligent assistant that reads medical prescriptions and explains each medicine's price, availability, and prescribed duration. Be concise, clear, and accurate."
    )
)

# Function to encode image in base64
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read()).decode("utf-8")
        return encoded_string

# Encode the image
base64_image = encode_image(image_path)

# Upload the image with a user query
chat.append(
    user(
        "Please analyze this prescription and tell me the details, price, availability, and duration for each medicine.",
        image(image_url=f"data:image/jpeg;base64,{base64_image}", detail="high"),
    )
)

# Get the AI response
response = chat.sample()
print(response.content)

When you run this code, you’ll see Grok 4 extract the prescription data and generate a detailed medical report. It should looking similar the example below.

Below is a clear and concise analysis of the prescription dated 13/10/2022 for Ms. Fiza Mozawala (19/F). I've interpreted the handwritten items based on standard medical terminology (e.g., "1-0-1" means 1 dose in the morning, 0 in the afternoon, and 1 in the evening). For each item, I've provided:

- **Details**: What it is, common use, and dosage.
- **Prescribed Duration**: As noted.
- **Price**: Approximate retail price in INR (based on common Indian pharmacy rates as of 2023; prices may vary by location, brand, and discounts—check current rates at pharmacies like Apollo or 1mg).
- **Availability**: General status in India (prescription or over-the-counter; widely available unless noted).

This prescription appears to be for dental/oral health issues (e.g., infection, pain, gum care). Consult your doctor or pharmacist for exact usage, side effects, or alternatives. The last item is a recommended product, not a medicine.

### 1. Cap. Phexin 500mg
- **Details**: Cephalexin (antibiotic) capsule, 500mg strength. Used for bacterial infections (e.g., dental infections). Dosage: 1 capsule morning and evening (total 2 per day).
- **Prescribed Duration**: 5 days (total ~10 capsules needed).
- **Price**: ₹220-₹280 for a strip of 10 capsules (generic alternatives ~₹150-₹200).
- **Availability**: Prescription required; widely available at pharmacies and online (e.g., 1mg, Netmeds).

### 2. Tab. Zerodol PT
- **Details**: Tablet containing Aceclofenac (pain reliever/anti-inflammatory) + Paracetamol (fever/pain reducer) + possibly other actives (variant of Zerodol-P). Used for pain and inflammation (e.g., dental pain). Dosage: 1 tablet morning and evening (total 2 per day).
- **Prescribed Duration**: 5 days (total ~10 tablets needed).
- **Price**: ₹100-₹150 for a strip of 10 tablets (generics ~₹80-₹120).
- **Availability**: Prescription required; widely available at pharma….

Let’s see how many tokens we used to generate the response.

print(response.usage)

The majority of tokens were used for the image prompt due to the large size of the image. The rest were fairly normal.

completion_tokens: 893
prompt_tokens: 1860
total_tokens: 3351
prompt_text_tokens: 68
prompt_image_tokens: 1792
reasoning_tokens: 598
cached_prompt_text_tokens: 4

Now that we have become familiar with the xAI SDK, let’s test the tool calling capabilities of Grok 4. In this section, we will build a medical prescription analyzer using the Grok 4 and Firecrawl APIs.

First, you need to create a Firecrawl account and generate an API key. Save the API key as an environment variable:

FIRECRAWL_API_KEY=your_firecrawl_api_key_here

After that, install the Firecrawl Python SDK:

pip install firecrawl-py

Next, initialize both Firecrawl and xAI clients using the API keys:

import os
import json
import base64
from rich.console import Console
from rich.markdown import Markdown as RichMarkdown
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Dict, List
from xai_sdk import Client
from xai_sdk.chat import system, user, image, tool, tool_result
from firecrawl import FirecrawlApp, ScrapeOptions

# Initialize clients with connection pooling
client = Client(
    api_key=os.getenv("XAI_API_KEY"),
    timeout=3600,
)
fc = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))

We will create two functions for medicine search and scraping web pages:

• get_medicine_info_fast - Takes a medicine name and retrieves its price, availability, and scrapes the page for more information.
• get_multiple_medicines_concurrent - Uses multithreading to fetch information for several medicines at once, optimizing speed by running searches concurrently and handling errors or timeouts gracefully.

def get_medicine_info_fast(name: str) -> Dict:
    """Optimized medicine info fetcher with error handling"""
    try:
        results = fc.search(
            query=f"{name} medicine price availability",  # Shorter query for faster search
            limit=1,
            scrape_options=ScrapeOptions(formats=["markdown"]),
        )
        snippet = results.data[0] if results.data else {}
        return {
            "name": name,
            "info_markdown": snippet.get("markdown", "N/A"),
            "url": snippet.get("url", "N/A"),
            "description": snippet.get("description", "N/A"),
            "status": "success",
        }
    except Exception as e:
        return {
            "name": name,
            "info_markdown": "Error fetching data",
            "url": "N/A",
            "description": f"Error: {str(e)}",
            "status": "error",
        }


def get_multiple_medicines_concurrent(
    medicine_names: List[str], max_workers: int = 5
) -> List[Dict]:
    """Fetch multiple medicine info concurrently"""
    results = []

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Submit all tasks
        future_to_medicine = {
            executor.submit(get_medicine_info_fast, name): name
            for name in medicine_names
        }

        # Collect results as they complete
        for future in as_completed(future_to_medicine):
            try:
                result = future.result(timeout=30)  # 30 second timeout per request
                results.append(result)
            except Exception as e:
                medicine_name = future_to_medicine[future]
                results.append(
                    {
                        "name": medicine_name,
                        "info_markdown": "Timeout or error",
                        "url": "N/A",
                        "description": f"Error: {str(e)}",
                        "status": "error",
                    }
                )


    return results

Let’s test the functions by providing them with the name of a medication:

print(get_medicine_info_fast("Aspirin"))

We receive the name, scraped webpage in markdown format, URL, description, and status in JSON format:

{'name': 'Aspirin', 'info_markdown': '[Skip to main content](https://www.goodrx.com/aspirin#skip-to-content)nnAre you a healthcare professional? [Join GoodRx for HCPs](https://www.goodrx.com/hcp/join?redire………, and other pharmacies. Prices start at $2.80.', 'status': 'success'}

Now, let’s try to fetch information for multiple medicines:

print(get_multiple_medicines_concurrent(["Aspirin", "Ibuprofen"]))

We get quick results for both medications in JSON format, along with additional metadata:

[{'name': 'Aspirin', 'info_markdown': '[Skip to main content](https://www.goodrx.com/aspirin#skip-to-content)nnAre you a healthcare professional? [Join GoodRx for H………mon version, by using a GoodRx coupon.', 'status': 'success'}]

To enable function calling with xAI models, we will define our Python functions as tools that the model can invoke dynamically, similar to how the OpenAI SDK handles function calling.

Each tool is described in a dictionary format, specifying its name, description, parameters, and required arguments.

In this project, we have defined two tools: one for fetching information about a single medicine (get_medicine_info_fast), and another for retrieving data on multiple medicines concurrently (get_multiple_medicines_concurrent).

The following tool definitions tell Grok 4 what functions are available to call. For each function, the AI knows the name, what it does, and what information it needs to run. Based on the prescription image content, Grok 4 automatically selects and runs the most appropriate function.

# Tool definitions
tool_definitions = [
    tool(
        name="get_medicine_info_fast",
        description="Fetch markdown info, URL, and description for a medicine via Firecrawl (optimized)",
        parameters={
            "type": "object",
            "properties": {
                "name": {"type": "string", "description": "Name of the medicine"},
            },
            "required": ["name"],
        },
    ),
    tool(
        name="get_multiple_medicines_concurrent",
        description="Fetch info for multiple medicines concurrently",
        parameters={
            "type": "object",
            "properties": {
                "medicine_names": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "List of medicine names",
                },
                "max_workers": {
                    "type": "integer",
                    "description": "Maximum concurrent workers (default: 5)",
                    "default": 5,
                },
            },
            "required": ["medicine_names"],
        },
    ),
]

# Map tools to functions
tools_map = {
    "get_medicine_info_fast": get_medicine_info_fast,
    "get_multiple_medicines_concurrent": get_multiple_medicines_concurrent,
}

The prescription analysis workflow uses function calling tools to automate the extraction and reporting of medicine information from a prescription image.

It starts by encoding the prescription image into a base64 string using a utility function so that AI can use it.

The main analyze_prescription_fast function initiates a chat session with the Grok 4 AI model, providing it with the tool definitions. The AI analyzes the image, identifies medicine names, and calls the relevant tools to fetch detailed information such as descriptions, prices, and purchase links.

Finally, the results are compiled into a markdown report, with clear sections for each medicine, enabling efficient and scalable prescription analysis within an AI assistant.

def encode_image(path: str) -> str:
    """Utility to encode image to base64"""
    try:
        with open(path, "rb") as f:
            return base64.b64encode(f.read()).decode("utf-8")
    except FileNotFoundError:
        raise FileNotFoundError(f"Image file not found: {path}")
    except Exception as e:
        raise Exception(f"Error encoding image: {str(e)}")

def analyze_prescription_fast(image_path: str) -> str:
    """Main function to analyze prescription with optimizations"""
    try:
        # Encode the prescription image
        encoded_img = encode_image(image_path)

        # Create chat session
        chat = client.chat.create(
            model="grok-4",
            tools=tool_definitions,
            tool_choice="auto",
        )

        # Enhanced system prompt for better extraction
        chat.append(
            system(
                "You are MedGuide AI. Extract ALL medicine names from the prescription image. "
                "If you find multiple medicines, use get_multiple_medicines_concurrent to fetch "
                "all information at once for faster processing. For single medicine, use get_medicine_info_fast. "
                "Create a comprehensive markdown report."
            )
        )

        # User provides the prescription image
        chat.append(
            user(
                "Extract all medicine names from this prescription and get their details efficiently.",
                image(image_url=f"data:image/jpeg;base64,{encoded_img}", detail="high"),
            )
        )

        # Initial model call
        response = chat.sample()
        chat.append(response)

        # Execute tool calls if any
        if response.tool_calls:
            for tc in response.tool_calls:
                func_name = tc.function.name
                func_args = json.loads(tc.function.arguments)

                print(f"Executing {func_name} with args: {func_args}")

                result = tools_map[func_name](**func_args)
                chat.append(tool_result(json.dumps(result)))

            # Request final formatted report
            chat.append(
                user(
                    "Create a comprehensive markdown report with H2 heading for each medicine that contains: Description, "
                    "Typical Duration, Price Information, and Purchase Link"
                )
            )

            # Generate final report
            final = chat.sample()
            return final.content
        else:
            return response.content

    except Exception as e:
        return f"Error analyzing prescription: {str(e)}"

We’ll now test our prescription analysis workflow by giving it with an image path and then displaying the result in markdown format. It may take up to a minute to generate the report.

image_path = "data/image1.jpg"
result = analyze_prescription_fast(image_path)
console = Console()
console.print(RichMarkdown(result))

In this example, we can see that Grok 4 has accurately identified the medications, ran the functions, and correcly formatted the report in markdown.

Executing get_multiple_medicines_concurrent with args: {'medicine_names': ['Phexin', 'Zerodol PT', 'Stolin gum paint', 'Colgate Plax']}

You can also build this app using the Agentic frameworks. Check out The Best Open Source Frameworks For Building AI Agents in 2025 to learn more.

Now that we have developed our backend, we can create a frontend so that anyone can run the app, upload an image, and generate a medical prescription report with all the key information.

The source code for the UI is quite extensive, so we have hosted it on GitHub for you to copy and paste into the app.py file. You can find it here: Medical-AI-with-Grok4/app.py at main().

The app.py file contains the code that creates a user interface for our application. Here is a brief explanation of how it works:

• Web Interface: The app first builds a web interface using Gradio where a user can upload a prescription image and view a detailed report. The interface includes collapsible sections for logs and a medical disclaimer .
• AI Image Analysis: When an image is uploaded, the script sends it to the Grok 4 AI model. The AI’s first job is to act like a smart OCR, reading the image to identify and extract the names of all the medicines listed .
• Concurrent Web Search: The AI then uses function calling to trigger a web search for each medicine using the Firecrawl API. To speed things up, it fetches information for multiple medicines at the same time (concurrently) using a ThreadPoolExecutor .
• Live Report Generation: The search results are sent back to Grok 4, which then generates a final, detailed report. The script uses streaming (chat.stream() and yield) to display the AI’s thought process and the final report in real-time.

Before we run the app.py code, you’ll need to install the Gradio and Pillow packages:

pip install gradio
pip install pillow

Now you can try running the app:

python app.py

* Running on local URL:  http://127.0.0.1:7860
* To create a public link, set `share=True` in `launch()`.

Visit the URL: http://127.0.0.1:7860 to access the MedGuide AI application.

Upload the image of the medical prescription from your local directory and press the “Analyze Prescription” button. The analysis will begin within a few seconds.

You can see what is happening in the background by clicking on the “Processing Logs” tab. The screenshot below shows an exmample where the concurrent function is invoked, triggering multiple Firecrawl API requests.

At the end, you’ll receive a properly formatted medical report detailing the medications, their descriptions, duration, price, and purchase link.

Please note that the app is not perfect. It is intended for educational purposes and may make mistakes. Please consult a doctor about medical questions and any medication you’ve been prescribed.

The purpose of this guide is to show how easy it has become to use vision reasoning models to invoke functions and generate highly accurate reports.

If you are facing issues running the app, please check out the GitHub repository: kingabzpro/Medical-AI-with-Grok4.

In this tutorial, we created an app that helps patients quickly identify all medications from a prescription by simply uploading an image and receiving a detailed report for each medicine. The goal was to show how we can combine these AI tools to pull information from complex images like handwritten prescriptions and automatically gather related data from the web. This same approach works well for any task where you’d normally have to manually enter data and do research.

Throughout this tutorial, we explored how the xAI SDK and Firecrawl API can be combined to create powerful web applications. I especially love the Firecrawl search with content scraping API, scrapes and structures the page content to provide richer information than just a simple search does. You can find more information by reading the documentation: Search | Firecrawl