Implementing AI-enabled software with large language models (LLMs) other than ChatGPT can still be quite challenging. However, there are compelling reasons to consider open-source LLMs. For instance, you can fine-tune them yourself, they can be more cost-effective than proprietary models, and they may offer unique capabilities. All of these are valid reasons to perform due diligence when selecting a model for your use case.
That said, working with open-source models is often difficult because most will not run on standard hardware unless you have access to high-end GPUs. This is where services such as Microsoft AI Foundry come in, offering a platform that combines open-source LLMs with the convenience of serverless model hosting, ample computational power from the cloud, and pay-as-you-go billing. This approach makes it easier to experiment with different models for your use case while speeding up development, since you do not have to manage model hosting yourself.
Function calling on Azure AI Foundry
My go-to framework for AI-enabled development is Microsoft’s Semantic Kernel, which offers a connector for Azure AI Foundry. Many LLM development use cases involve function calling or structured outputs, features that allow the LLM to integrate deeply into your software. However, not every model supports these features. Even if a model supports them generally, it may not do so on AI Foundry, and it’s currently difficult to determine which model supports which feature.
You might expect to find this information easily in the Model Explorer, but it’s not yet available there. Instead, the details are somewhat hidden in the AI Foundry documentation.
For example, although the Llama 3 family generally supports function calling, it currently does not on Azure at the time of writing. Even more problematic is that Semantic Kernel does not warn you if you provide tools in a prompt for a model lacking this capability on Azure. Therefore, it’s helpful to consult the documentation to verify whether a model supports your required features. For instance, Mistral-Large can be deployed in scenarios where function calling is a requirement.
Structured Outputs
Developing with OpenAI’s GPT models is convenient because they are highly capable and easy to access through Azure OpenAI or the OpenAI API. Their strong capabilities, however, can be a downside if you want to leverage one of Semantic Kernel’s key features: being model-agnostic. For example, OpenAI’s models support the "json_schema" response type. Technically, this means you can provide Semantic Kernel with a C# class describing the structure you want in a JSON object, and OpenAI handles the rest to ensure the LLM responds in the correct structure. This is extremely useful for generating structured data.
However, "json_schema" is not widely supported by other LLMs. Most open-source models that support JSON responses use the "json_object" mode instead. In this mode, the API ensures the response is valid JSON, but it does not guarantee any particular schema. As a result, you—the consumer—are responsible for instructing the LLM to adhere to the structure you expect.
In Semantic Kernel and AI Foundry, you can achieve this by passing a configured AzureAIInferencePromptExecutionSettings instance in your KernelArguments when invoking Semantic Kernel. The listing below shows a sample of these PromptExecutionSettings. Note that some open-source models do not support "json_object" responses when used in combination with tool usage, hence the feature is disabled in this example configuration. The below sample shows how to create appropriate PromptExecutionSettings
new AzureAIInferencePromptExecutionSettings
{
ResponseFormat = "json_object",
FunctionChoiceBehavior = FunctionChoiceBehavior.None(
options: DefaultFunctionChoiceBehavior,
functions: []
),
Tools = [],
}Next, you need to ensure the LLM understands your expected response schema. To do this, you can use JSON Schema to describe your structure. The .NET framework offers functionality to generate schemas from C# types. If you want to include metadata from DescriptionAttributes, you can extend the standard approach as shown in the following listing.
using System.ComponentModel;
using System.Diagnostics;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.Json.Schema;
public static class JsonSchemaGenerator
{
public static JsonNode GetSchemaNode<T>()
{
var options = JsonSerializerOptions.Default;
JsonSchemaExporterOptions exporterOptions =
new()
{
TransformSchemaNode = (context, schema) =>
{
// Determine if a type or property and extract the relevant attribute provider.
ICustomAttributeProvider? attributeProvider = context.PropertyInfo is not null
? context.PropertyInfo.AttributeProvider
: context.TypeInfo.Type;
// Look up any description attributes.
DescriptionAttribute? descriptionAttr = attributeProvider
?.GetCustomAttributes(inherit: true)
.Select(attr => attr as DescriptionAttribute)
.FirstOrDefault(attr => attr is not null);
// Apply description attribute to the generated schema.
if (descriptionAttr != null)
{
if (schema is not JsonObject jObj)
{
// Handle the case where the schema is a Boolean.
JsonValueKind valueKind = schema.GetValueKind();
Debug.Assert(valueKind is JsonValueKind.True or JsonValueKind.False);
schema = jObj = new JsonObject();
if (valueKind is JsonValueKind.False)
{
jObj.Add("not", true);
}
}
jObj.Insert(0, "description", descriptionAttr.Description);
}
return schema;
},
};
return options.GetJsonSchemaAsNode(typeof(T), exporterOptions);
}
public static string GetSchema<T>()
where T : class => GetSchemaNode<T>().ToString();
}
Afterward, you can augment the type you wish to parse with more detailed instructions for the LLM. An example of this is provided below.
using System.ComponentModel;
public class TripAdvise
{
[Description(
"The city where a traveler can do sightseeings"
)]
public required string City{ get; set; }
[Description("A detailed description of what a tourist can do in this city")]
public required string TravelAdvisory{ get; set; }
}
Putting it all together allows you to obtain structured responses from any compatible LLM. The result is an LLM response materialized into your own data structure, which you can then process further as needed.
var promptSettings = new AzureAIInferencePromptExecutionSettings
{
ResponseFormat = "json_object",
FunctionChoiceBehavior = FunctionChoiceBehavior.None(
options: DefaultFunctionChoiceBehavior,
functions: []
),
Tools = [],
};
var kernelArguments = new KernelArguments(promptSettings);
var schema = JsonSchemaGenerator.GetSchema<TripAdvise>();
var prompt = $"Create a list of cities with activities a tourist could do in the following schema: {schema }";
var response = await kernel.InvokePromptAsync(prompt, kernelArguments);
var result = JsonSerializer.Deserialize<IEnumerable<TripAdvise>>();In result we receive the response of the LLM materialized into our own data structure and can proceed with working on it.
Developing with LLMs is becoming more convenient thanks to excellent tools like Semantic Kernel. However, if you plan to implement advanced scenarios—particularly when using open-source models on platforms like Azure AI Foundry—there are still limitations that demand a bit of creativity to overcome. By carefully selecting models, verifying their capabilities, and using structured output features effectively, you can build robust AI-enabled applications that truly meet your needs.