Enhancing Unity Experiences with Azure OpenAI for XR — Part 2

2 min read1 day ago

In the previous part of our series, we explored how to run Azure OpenAI using FastAPI to interactively generate responses to user queries. Now, let’s take it a step further by integrating this AI capability directly into Unity, allowing us to create interactive and intelligent experiences within our Unity projects.

Create a New C# Script:

In your Unity project, right-click in the Assets panel, go to Create > C# Script. Name the script AIController.

Write the Script:

Implement the script to handle sending requests to your FastAPI server (which interacts with Azure OpenAI) and updating the UI with responses:

using UnityEngine;
using UnityEngine.UI;
using UnityEngine.Networking;
using System.Collections;

public class AIController : MonoBehaviour
{
    public InputField questionInput;
    public Text responseText;

    private string baseURL = "http://localhost:8000/ask_ai/";

    public void AskAI()
    {
        StartCoroutine(GetRequest(questionInput.text));
    }

    IEnumerator GetRequest(string question)
    {
        string url = baseURL + WWW.EscapeURL(question);
        UnityWebRequest request = UnityWebRequest.Get(url);
        yield return request.SendWebRequest();

        if (request.result == UnityWebRequest.Result.Success)
        {
            string response = request.downloadHandler.text;
            responseText.text = response;
        }
        else
        {
            responseText.text = "Error: " + request.error;
        }
    }
}

Adding the Script to a Component

Attach Script to GameObject:

In Unity Editor, create a UI canvas or use an existing UI element where users can input questions and receive responses.

Attach Script to UI Component:

Drag and drop the AIController script onto a UI component (e.g., a button) that triggers the AskAI() method when clicked.

Overview of the Script

InputField and Text Variables:

questionInput: Reference to an InputField UI element where users type their questions.
responseText: Reference to a Text UI element where AI responses will be displayed.

baseURL:

Stores the base URL of your FastAPI server (http://localhost:8000/ask_ai/). Adjust this URL according to your server setup.

AskAI() Method:

Called when the user interacts with the UI element (e.g., presses a button) to initiate the AI request.

GetRequest() Coroutine:

Sends a GET request to your FastAPI server with the user’s question.
Handles the response asynchronously and updates responseText with the AI's answer or displays an error if the request fails.

Conclusion

Integrating Azure OpenAI with Unity using a simple script allows developers to enhance their projects with interactive AI capabilities. By following these steps, you can quickly set up a basic AI chat feature within your Unity applications, enabling dynamic interactions based on user queries.

Stay tuned for more insights and happy coding!

To implement FastAPI, refer to my previous blog post for detailed instructions. https://elanthirayan.medium.com/enhancing-unity-experiences-with-azure-openai-for-xr-bridging-virtual-worlds-with-intelligent-177ce3ebb470