In-Depth

Accepted for Limited Beta: Hands On with Azure AI-Assisted GitHub Models

• By David Ramel
• 08/13/2024

GitHub Models debuted this month in a limited beta program designed to get feedback on the free AI model playground.

Like similar AI playgrounds, GitHub Models (requires GitHub account) lets users experiment with various models before going to production, which is done with Azure AI.

"We believe every developer can be an AI engineer with the right tools and training," said GitHub's Thomas Dohmke in announcing the playground on Aug. 1. "From playground to coding with the model in Codespaces to production deployment via Azure, GitHub Models shows you how simple it can be."

The playground currently sports 24 models, with many familiar names.

"From Llama 3.1, to GPT-4o and GPT-4o mini, to Phi 3 or Mistral Large 2, you can access each model via a built-in playground that lets you test different prompts and model parameters, for free, right in GitHub," Dohmke said. "And if you like what you're seeing on the playground, we've created a glide path to bring the models to your developer environment in Codespaces and VS Code. And once you are ready to go to production, Azure AI offers built-in responsible AI, enterprise-grade security & data privacy, and global availability, with provisioned throughput and availability in over 25 Azure regions for some models. It's never been easier to develop and run your AI application."

Let's go hands-on.

Waitlist
The first thing to do is apply for the limited public beta at the waitlist.

GitHub advises new users to review preview terms and reminds them feedback is welcomed in community discussions.

Marketplace
Once accepted, sign in to GitHub and go to the platform's Marketplace.

That's the starting point for other things, too, including: Copilot extensions; apps; actions for API management and more; and a host of others.

Models
Click on the Models item in the left-side menu to "Try, test, and deploy from a wide range of model types, sizes, and specializations."

To adjust parameters for the model, users select the Parameters tab in the sidebar. Then, to see code that corresponds to the parameters that you selected, switch from the Chat tab to the Code tab.

OpenAI GPT-4o mini
Opening up this model reveals it's "An affordable, efficient AI solution for diverse text and image tasks," described as a smaller, faster, and more efficient version of GPT-4o that sports a training date of October 2023.

Along with the README pictured above, it has tabs for Evaluation, Transparency and License.

Here's the evaluation.

The model comes with three ready-made prompts to get started:

• What are some popular tourist attractions in Paris?
• What is the history of the Great Wall of China?
• Can you explain the basics of machine learning?

I asked a different question: "What is Virtualization & Cloud Review?"

It provides different answers to the same question, as is typical with AI models. That makes it harder to compare system prompt parameters, such as Temperature (Controls randomness in the response, use lower to be more deterministic.) and Top P (Controls text diversity by selecting the most probable words until a set probability is reached.). Various models have different parameters that can be set, varying in type and number.

Taking all that into account, here is the reply with the default parameter settings:

Virtualization & Cloud Review is a publication focused on the topics of virtualization technology, cloud computing, and related innovations in IT infrastructure. Its content typically includes:

1. Product Reviews: Assessments and evaluations of virtualization and cloud products, tools, and platforms, helping readers understand their features and benefits.
2. Industry News: Updates on the latest trends, advancements, and market changes in the fields of virtualization and cloud services.
3. Educational Content: Articles, how-tos, and tutorials that help IT professionals understand and implement virtualization and cloud strategies effectively.
4. Case Studies: Real-world examples of organizations successfully using virtualization and cloud solutions, including best practices and lessons learned.
5. Expert Insights: Opinions and analysis from industry experts on emerging technologies and their implications for businesses.

The publication serves IT professionals, decision-makers, and organizations looking to leverage virtualization and cloud technologies to optimize their infrastructure and operations.

And here's the response with the Temperature set to 0 and Top P set to 0.01:

Virtualization & Cloud Review is a publication or resource that focuses on topics related to virtualization technologies and cloud computing. It typically covers a range of subjects, including:

1. Virtualization Technologies: This includes discussions on hypervisors, virtual machines, and containerization technologies like Docker and Kubernetes. It may also cover the benefits of virtualization, such as resource optimization, cost savings, and improved disaster recovery.
2. Cloud Computing: The publication often explores various cloud service models (IaaS, PaaS, SaaS), deployment models (public, private, hybrid), and trends in cloud adoption. It may also review cloud service providers and their offerings.
3. Industry Trends: Insights into the latest trends in the virtualization and cloud sectors, including emerging technologies, best practices, and case studies.
4. Product Reviews: Evaluations of software and hardware solutions related to virtualization and cloud services, helping IT professionals make informed decisions.
5. Expert Opinions and Analysis: Articles and commentary from industry experts, providing insights into the future of virtualization and cloud technologies.
6. Best Practices and Guides: Practical advice for implementing and managing virtualization and cloud solutions effectively.

Overall, Virtualization & Cloud Review serves as a valuable resource for IT professionals, decision-makers, and anyone interested in the evolving landscape of virtualization and cloud computing.

So, not much difference, and it would be hard to attribute any such difference to the model's propensity to provide different answers to the same question or to adjusting system parameters.

Anyway, in a real-world scenario with a data scientist, those things would be accounted for, presumably.

Users can also click on a green Get Started button to let users try out basic code samples that can be run in GitHub's Codespaces, a cloud-hosted developer environment. Here's what it looks like to get started with JavaScript:

For this, documentation is provided, along with samples.

"Use the knowledge you've gained from the playground and Codespaces to set up a prototype or proof-of-concept within your own applications," Dohmke said. "Run prompt evals in GitHub Actions with a series of JSON files that you just pipe in the GitHub Models command within the GitHub CLI. Or you can leverage GitHub Models to build a GitHub Copilot Extension, extending GitHub's platform ecosystem for every stage of software development. And finally, go to production with Azure AI by replacing your GitHub personal access token with an Azure subscription and credential."

Feedback
Again, the main idea of the limited beta is to let users put it through its paces and solicit feedback to improve the product, and current discussions range from "Response streaming is not working properly" to "Please add pwsh/PowerShell to the Codespace so that we can use Invoke-RestMethod."

In alignment with GitHub and Microsoft's "continued commitment to privacy and security," users are advised that no prompts or outputs in GitHub Models will be shared with model providers, nor used to train or improve the models.