Exploring Azure AI:

A Tour of Azure AI

Services for Modern

Applications

by Marian Veteanu

Azure AI services

Azure OpenAI Custom Vision

Azure AI Search Face

Content Safety Translator

Speech Language

Document Indexer Video Intelligence

Immersive

Vision Reader

Azure OpenAI

The Azure OpenAI Service is a platform that provides access to

advanced generative AI models developed by OpenAI, such as

GPT-4, Codex, and DALL-E, directly through the Azure ecosystem.

Key Models

▪ GPT-4 and GPT-3: These language models generate human-like

text, complete sentences, answer questions, summarize

content, and provide conversational AI capabilities.

▪ Codex: Codex is designed to understand and generate code,

making it a valuable tool for developers and software

automation.

▪ DALL-E: DALL-E is a model that generates images based on text

prompts, useful for creative and design-related tasks.

▪ Whisper (Speech Recognition): Whisper provides robust

speech-to-text capabilities, enabling transcription and real-

time voice applications.

Example using Azure OpenAI GPT-4

require('dotenv').config();

const endpoint = process.env.OPENAI_ENDPOINT; const apiKey = process.env.OPENAI_API_KEY; const deploymentName = process.env.DEPLOYMENT_NAME;

// Function to call GPT-4 model

async function generateResponse(prompt) {

try {

const response = await fetch(

`${endpoint}/openai/deployments/

${deploymentName}/completions? api-version=2023-05-15`,

{

method: 'POST', headers: {

'Content-Type': 'application/json', 'api-key': apiKey,

},

body: JSON.stringify({

prompt: prompt, max_tokens: 100, temperature: 0.7, top_p: 0.95

}),

}

);

if (!response.ok) {

throw new Error(`HTTP error! ${response.status}`);

}

const data = await response.json(); console.log('GPT-4 Response:', data.choices[0].text.trim());

}

catch (error) {

console.error('Error calling Azure OpenAI:', error.message);

}

}

// Example prompt

const prompt = "What are some interesting applications of AI in healthcare?";

generateResponse(prompt); GPT-4 Response: AI has several interesting

applications in healthcare, such as

predictive analytics, personalized

medicine, and medical image analysis...

Example using Azure OpenAI Whisper

dotenv.config();

const endpoint = process.env.OPENAI_ENDPOINT; const apiKey = process.env.OPENAI_API_KEY; const deploymentName = process.env.DEPLOYMENT_NAME;

// Function to call Whisper for transcription async function transcribeAudio(filePath) {

try {

// Read the audio file into a buffer const audioBuffer = fs.readFileSync(filePath);

const response = await fetch(

`${endpoint}/openai/deployments/

${deploymentName}/transcriptions?api-version=2023-05-15`,

{

method: 'POST', headers: {

'Content-Type': 'audio/wav', 'api-key': apiKey, 'model': 'whisper-1'

},

body: audioBuffer

}

);

if (!response.ok)

throw new Error(`HTTP error! Status:

${response.status}`);

const data = await response.json(); console.log('Transcription:', data.text);

} catch (error) {

console.error('Error transcribing audio:', error.message);

}

}

// Example usage

const audioFilePath = 'path/to/your/audio-file.wav'; transcribeAudio(audioFilePath);

You should see the transcribed text printed to the console, like:

Transcription: "This is a sample audio transcription text."

Azure AI Search

Azure AI Search is a powerful, AI-enhanced search service from

Microsoft Azure that allows developers to integrate search

functionality into web and mobile applications.

Azure AI Search indexes and searches through data sources

connected to it, allowing it to search various types of content:

▪ Structured and Unstructured Data in Databases. It can index

structured data from databases like Azure SQL Database,

Cosmos DB, or other SQL and NoSQL sources.

▪ Documents and Files. Azure AI Search can index content from

document storage like Azure Blob Storage, including files such

as PDFs, Word documents, Excel spreadsheets, images, and

text files.

▪ Text from Images and Scanned Documents. Through cognitive

skills, Azure AI Search can apply Optical Character Recognition

(OCR) to extract text from images and scanned documents,

making the text searchable.

▪ Metadata from Audio and Video Files. Azure AI Search can use

Azure Video Indexer to extract and index metadata, like

keywords, speakers, and key phrases, from audio and video

content.

▪ Web Content and JSON Data. Azure AI Search can index JSON

files and structured web content to enable searching through

structured, semi-structured, or hierarchical data.

▪ Application-Specific Data Models. Azure AI Search works well

with custom data models, where data is organized to fit the

needs of specific applications or industries.

Example Basic Full-Text Search in PDFs

Perform a Basic Full-Text Search in PDFs stored in Azure Blob

Storage using Azure AI Search.

1. Store PDFs in Azure Blob Storage

Upload your PDF files to a container in Azure Blob Storage.

2. Create an Azure AI Search Resource

Create an Azure AI Search resource if you don’t already have one. Get the endpoint URL and API key for your search service.

3. Create a Data Source in Azure AI Search Go to your Azure AI Search resource in the Azure portal. Create a new data source with

the following configuration:

Name: Give your data source a name (e.g., pdf-datasource). Type: Choose Azure Blob Storage.

Connection String: Use your blob storage connection string. Container: Enter the name of the blob container where your PDFs are stored.

Parsing Mode: Set to delimited if your PDFs contain structured data, or leave as default.

4. Create a Skillset (Optional)

This is optional if you only need basic text extraction, as Azure AI Search includes a built-in OCR capability.

5. Create an Index

In Azure AI Search, create a new index that defines the fields you want to make searchable. Define fields like content, metadata_title, and metadata_author to store the extracted content and metadata from PDFs.

6. Create an Indexer

Create an indexer to pull data from your blob storage and populate your search index: Name: Give your indexer a name (e.g., pdf-indexer). Data source: Select the data source you created.

Index: Select the index you created.

Schedule: Configure the indexer to run on a schedule or on-demand.

7. Perform a Basic Full-Text Search (see next page)

Example Basic Full-Text Search in PDFs

– part 2

Once indexing is complete, you can perform a full-text search on

your PDFs. Here’s an example for querying the indexed PDFs:

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const searchEndpoint = process.env.SEARCH_ENDPOINT;

const searchApiKey = process.env.SEARCH_API_KEY;

const indexName = 'pdf-index';

// Function to perform a basic full-text search

async function searchPDFs(query) {

const url = `${searchEndpoint}/indexes/

${indexName}/docs/search?api-version=2023-07-01`;

const response = await fetch(url, {

method: 'POST',

headers: {

'Content-Type': 'application/json', 'api-key': searchApiKey,

},

body: JSON.stringify({

search: query,

top: 10 // Retrieve the top 10 results

})

});

if (!response.ok) {

throw new Error(`Search request failed:

${response.statusText}`);

}

const data = await response.json(); console.log("Search Results:", data.value);

}

// Example query to search PDFs for the term "Azure AI"

searchPDFs("Azure AI");

Example: Searching documents in

Azure Cosmos DB

Step 1: Store Documents in Azure Cosmos DB. Example:

{

"id": "1",

"title": "Introduction to Cloud Computing", "content": "Cloud computing is a way to store and access

data over the internet...",

"category": "Technology"

}

Step 2: Create an Azure AI Search Resource

Step 3: Go to Data Sources and create a new data source:

Name: Give your data source a name (e.g., cosmosdb-datasource). Type: Select Azure Cosmos DB.

Connection String: Provide the Cosmos DB connection string.

Container: Specify the Cosmos DB container that holds your documents. Query: Optionally, provide a query to filter documents if needed.

Step 4: Go to Indexes and create a new index:

id: Set as key.

title: Set as searchable.

content: Set as searchable.

category: Set as filterable.

Step 5: Create an Indexer

Step 6: Query the Indexed Documents

Example: Searching documents in

Azure Cosmos DB – part 2

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const searchEndpoint = process.env.SEARCH_ENDPOINT;

const searchApiKey = process.env.SEARCH_API_KEY;

const indexName = 'cosmosdb-index';

// Perform a full-text search on Cosmos DB documents

async function searchDocuments(query) {

const url = `${searchEndpoint}/indexes/

${indexName}/docs/search?api-version=2023-07-01`;

const response = await fetch(url, {

method: 'POST',

headers: {

'Content-Type': 'application/json', 'api-key': searchApiKey,

},

body: JSON.stringify({

search: query,

top: 10 // Limit the results to top 10

})

});

if (!response.ok) {

throw new Error(`Search failed: ${response.statusText}`);

}

const data = await response.json(); console.log("Search Results:", data.value);

}

// Example query to search for "cloud computing"

searchDocuments("cloud computing");

The console output should show the top 10 documents containing

the term "cloud computing," including their titles, content, and

other fields defined in your index.

Content Safety

Azure AI Content Safety is a service within Microsoft's Azure

ecosystem designed to help organizations detect and filter

harmful, offensive, or inappropriate content. This service enables

developers to protect users from exposure to unsafe content by

flagging or blocking it, which is critical for applications in social

media, customer support, gaming, and educational platforms.

Key Features of Azure AI Content Safety

Text Moderation. Detects inappropriate or harmful text content,

including profanity, hate speech, bullying, and sexually explicit

content.

Image Moderation. Scans images to detect inappropriate visual

content, such as nudity, violent imagery, or other unsafe elements.

Customizable Moderation Policies. Allows for custom rules

based on the organization’s needs, supporting unique definitions

of "unsafe" content based on industry or community standards.

Real-Time and Batch Processing. Supports both real-time

moderation for quick responses and batch processing for large

volumes of content.

Integration with Other Azure Services. Can be integrated with

other Azure services to create automated moderation workflows.

Example of Text Moderation

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const contentSafetyEndpoint = process.env.CONTENT_SAFETY_ENDPOINT; const apiKey = process.env.CONTENT_SAFETY_API_KEY;

// Function to check text content for unsafe language async function moderateText(content) {

const url = `${contentSafetyEndpoint}/contentmoderation/

text/moderate?api-version=2023-07-01`;

try {

const response = await fetch(url, {

method: 'POST',

headers: {

'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': apiKey,

},

body: JSON.stringify({ content })

});

if (!response.ok) {

throw new Error(`Error: ${response.statusText}`);

}

const result = await response.json(); console.log("Moderation Results:", result);

}

catch (error) {

console.error("Error moderating text:", error.message);

}

}

// Example usage

const textContent = "This is a sample message with potentially offensive language.";

moderateText(textContent);

Speech

The Azure AI Speech Service is a cloud-based service that

provides powerful speech capabilities, including speech-to-

text (STT), text-to-speech (TTS), speech translation, and

speaker recognition. These capabilities are great for

applications in customer support, accessibility, virtual

assistants, and more.

Key Capabilities of Azure Speech Service

Speech-to-Text (STT). Converts spoken language into written

text in real-time or from pre-recorded audio files.

Text-to-Speech (TTS). Converts written text into natural-

sounding audio. Azure offers neural voices, which provide

highly realistic speech synthesis.

Speech Translation. Provides real-time translation of spoken

language, enabling multi-language conversations.

Speaker Recognition. Identifies and verifies speakers based

on their unique voice characteristics.

);

ion ce

);

"

v

hReg voi a ); );

w

. ) { );

eec ify o fig d s n." p c i n e l o s d i

u a

, a t y Spe - ioCo plet e rati

/ t D / u r

hKe p aud Com ); o nst

; t r " u , r

o e

peec al " ig udio wav" . demo s ( t ( t l n eur u onf ngA io. u ce o N p C i s i

t e u aud r

; O -' ipti enny e eech esiz t , Serv

- i s : tp J l p h u "

sdk scr F ( d ch - US o e

- i { Synt ou l

; d . i

ech N mSub en u izer n o a Spee

" A => t f

m

spe GIO fro = o hes t aso d s AI - E . r t l e e i

; f s

es . iz e

g h

ic KEY CH_R nfig Name i hSyn resu ultR es t zure v f c h n

n , y

er CH_ PEE hCo ice o Res nt s xt e A

S C .

es . ) { o Spee ext sy h

i . t c

to Speech iv SPEE nv xt peec isVo d ( sdk h e e te o th

t . e e S u c e

. t . A sdk p

; ni nv s ( hes . ync == ee S mpl e t

. sdk ( ();

xt ' g e s h t k w s = p " a m

co sd ne "S r

= ( o

o r p c T r l

Te tenv e/ ess roce peec hSyn = = extA son r ose on s elco

d g log e c

p f . e

e ' roc = ToS fig i er rea . . . ch ! W

@azu spee n peak l ' = . o s o

rom ion text hCon g C esiz . ult sole { s izer Spee ello f o n - H

(); i o "

v rom g Key Reg on eec nfi d nth zer res con se c hes to ( n f i h h i p o u y i ( l - h te k nf ynt

ampl s a s e t if } } s

do sd co peec peec unct st echC s st thes Text peec

. s s f n

Ex o rt rt nv t t c con spe c con syn }); un ToS impo impo dote cons cons asyn } // R text

Document Intelligence

Azure AI Document Intelligence (formerly known as Azure Form

Recognizer) is a service in Azure that enables applications to

extract structured information from various types of documents

such as forms, receipts, invoices, and other documents.

Key Capabilities of Azure AI Document Intelligence

Pre-built Models. Provides ready-to-use models for common

document types: Invoices, Receipts, IDs, US Health Insurance

Cards, US Personal Tax, US mortgage, US pay stubs, US bank

statements, US checks, Credit cards, US marriage certificates,

Contracts, Business cards

Custom Document Models. Allows you to train custom models

for specific document types that don’t fit the pre-built templates.

Layout Extraction. Extracts text, tables, and other layout

information from documents, making it easy to parse structured

and semi-structured documents.

Key-Value Pair Extraction. Detects and extracts key-value pairs

from documents, which is useful for forms and structured

documents.

Document

Intelligence Studio

Document to JSON!

Extract text, key-value pairs, and tables accurately from diverse

documents, forms, receipts, invoices, and cards—no need for

manual labeling, extensive coding, or ongoing maintenance by

document type.

Vision

The cloud-based Azure AI Vision service offers developers

powerful algorithms for processing images and extracting

valuable information.

By uploading an image or providing an image URL, Azure AI

Vision can analyze visual content in multiple ways, tailored

to your inputs and preferences.

Key services:

OCR Image analysis

- Extract text from images - Recognize products on shelves

- Search photos with image retrieval

Spatial analysis - Remove background from images

- Add captions to images

- Video Retrieval and Summary - Detect common objects in images

- Count people in an area - Extract common tags from images

- Detect when people cross a line - Detect sensitive content in images

- Detect when people enter/exit a zone - Create smart-cropped images

- Monitor social distance

Face

- Detect faces in an image

- Liveness detection

- Portrait processing

- Photo ID matching

Vision recipes in “Azure AI

Vision Studio”

The service identified

products on a shelf and

outputted this JSON

The OCR service

identified text from this

image and outputted

this JSON

Custom Vision

Azure AI Custom Vision enables developers to build, train, and

deploy custom image classification and object detection

models tailored to specific use cases. Unlike pre-built models,

Custom Vision allows users to create models that recognize

objects and categories unique to their applications by

uploading and labeling their own images.

The Custom Vision service leverages a machine learning algorithm

to analyze images for specific custom features. You provide sets of

images that either contain or lack the visual characteristics you

want to identify, labeling each image with custom tags during

submission.

Key Capabilities of Azure AI Custom Vision

Image Classification. Identifies and categorizes images based on

custom-defined labels.

Example: A manufacturer can use image classification to categorize

defective and non-defective parts in an assembly line, automating

quality control.

Object Detection. Detects and localizes multiple objects within an

image, providing bounding boxes around detected objects.

Custom Training with Labeled Images. Users upload their own

labeled datasets, allowing Custom Vision to learn to recognize

specific items.

Example: Train the model to recognize different grades or ripeness

levels of produce, such as apples or tomatoes, based on visual

characteristics like color, size, and surface quality. This helps

automate quality grading in food processing plants.

Iteration and Model Improvement. Allows for iterative training,

where users can refine their model by uploading new data and

retraining to improve accuracy.

Face

The Azure AI Face service is part of Azure’s AI Services suite,

specifically designed to detect, analyze, and identify human faces

in images. This service provides a range of face recognition features

that are useful for applications requiring facial analysis, such as

verifying identity, organizing photos, analyzing demographics, or

enhancing security.

Key Capabilities of Azure AI Face Service

Face Detection. Detects human faces in images and returns

information such as face coordinates, bounding boxes, and

rotation.

Face Attributes Analysis. Analyzes facial attributes, including age,

gender, emotions, facial hair, and head pose.

Face Verification. Verifies whether two faces belong to the same

person, commonly used in identity verification.

Face Identification. Identifies a face by comparing it with faces

stored in a known group of people. This requires a training set of

labeled faces.

Face Grouping. Groups similar faces together without knowing the

identities, useful for organizing unlabeled faces.

Emotion Detection. Detects emotions like happiness, sadness,

surprise, anger, and more.

You submit the image,

and the service returns

the JSON with detected

faces.

Face Detection and Attributes Analysis

This example demonstrates how to detect faces in an image

and analyze basic attributes such as age, gender, and

emotion.

import dotenv from 'dotenv';

import { FaceClient } from '@azure/cognitiveservices-face'; import { CognitiveServicesCredentials } from '@azure/ms-rest-js';

dotenv.config();

const endpoint = process.env.FACE_API_ENDPOINT; const apiKey = process.env.FACE_API_KEY;

const credentials = new CognitiveServicesCredentials(apiKey); const client = new FaceClient(credentials, endpoint);

async function detectFacesWithAttributes(imageUrl) {

const attributes = ["age", "gender", "emotion",

"facialHair", "glasses", "smile"];

const detectedFaces = await client.face.detectWithUrl(imageUrl, {

returnFaceId: true,

returnFaceAttributes: attributes

});

console.log("Detected Faces with Attributes:"); detectedFaces.forEach(face => {

console.log(`Face ID: ${face.faceId}`); console.log(` - Age: ${face.faceAttributes.age}`); console.log(` - Gender: ${face.faceAttributes.gender}`); console.log(` - Emotion:

${JSON.stringify(face.faceAttributes.emotion)}`);

console.log(` - Smile: ${face.faceAttributes.smile}`);

});

}

// Detect faces and attributes in an image detectFacesWithAttributes("https://example.com/sample-image.jpg");

Translator

Azure AI Translator is another Azure AI service that enables

developers to integrate language translation capabilities into

applications. It supports real-time or batch translation across

over 100 languages and dialects, making it useful for a wide range

of scenarios, such as multilingual customer support, global

communication, content localization, and website translation.

Key Features of Azure AI Translator

Text Translation. Translates text from one language to another in

real-time or in batches.

Document Translation. Translates entire documents while

preserving their original layout and format.

Speech Translation. Translates spoken language in real-time,

allowing people to communicate verbally across language

barriers.

Transliteration. Converts text from one script to another within

the same language (e.g., Hindi written in Latin script).

Language Detection. Detects the language of input text

automatically, which is useful when the language isn’t specified.

Example of using Text Translation

This example demonstrates how to translate text from one

language to another.

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const endpoint = process.env.TRANSLATOR_API_ENDPOINT;

const apiKey = process.env.TRANSLATOR_API_KEY;

async function translateText(text, fromLanguage, toLanguage) {

const url = `${endpoint}/translate?

api-version=3.0&from=${fromLanguage}&to=${toLanguage}`;

const response = await fetch(url, {

method: 'POST',

headers: {

'Ocp-Apim-Subscription-Key': apiKey, 'Content-Type': 'application/json'

},

body: JSON.stringify([{ text }])

});

const result = await response.json(); console.log("Translation:", result[0].translations[0].text);

}

// Translate "Hello, world!" from English to Romanian

translateText("Hello, world!", "en", "ro");

Language

Azure AI Language is a cloud-based service in the Azure AI Services

suite that provides natural language processing (NLP) capabilities.

Key Capabilities of Azure AI Language Service

Sentiment Analysis. Determines the sentiment of text (positive,

neutral, or negative), along with the sentiment score.

Named Entity Recognition (NER). Recognizes entities like names,

organizations, dates, and locations within text, categorizing them

into predefined types.

Language Detection. Identifies the language of a given text, which

is useful for multilingual applications.

Key Phrase Extraction. Identifies the main points or key phrases in

a text.

Text Summarization. Provides concise summaries of long

documents, helping readers get the main points quickly.

Question Answering. Finds answers to questions based on a set

of documents or a knowledge base.

Conversational Language Understanding. Understands intent

and extracts relevant information from conversations, which is

useful for building intelligent chatbots.

Example of using Text Summarization from Azure AI Language service

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const endpoint = process.env.LANGUAGE_API_ENDPOINT; const apiKey = process.env.LANGUAGE_API_KEY;

async function summarizeText(text) {

const url = `${endpoint}/language/analyze-text/jobs?

api-version=2022-05-15`;

const response = await fetch(url, {

method: 'POST',

headers: { 'Ocp-Apim-Subscription-Key': apiKey,

'Content-Type': 'application/json' },

body: JSON.stringify({

analysisInput: { documents: [{ id: '1’,

language: 'en', text }] },

tasks: [{ kind: "ExtractiveSummarization",

parameters: { sentenceCount: 3 } }]

})

});

const { jobId } = await response.json();

let summary;

do {

await new Promise(r => setTimeout(r, 2000)); const result = await fetch(`${endpoint}/language/

analyze-text/jobs/${jobId}?api-version=2022-05-15`,

{

headers: { 'Ocp-Apim-Subscription-Key': apiKey }

});

summary = (await result.json()).tasks.items[0].

results.documents[0]?.sentences;

} while (!summary);

console.log("Summary:", summary.map(s => s.text).join(" ")); }

const longText = `

Artificial intelligence (AI) has become essential in tech. Innovations in machine learning and deep learning are transforming industries like healthcare and finance. However, AI also brings challenges, such as ethics and data privacy. As AI evolves, addressing these issues will be crucial.`;

summarizeText(longText);

Video Indexer

Azure AI Video Indexer is a service in the Azure AI Services suite that

provides advanced video analysis capabilities. This service is

especially useful for media companies, broadcasters, and content

creators.

Key Capabilities of Azure AI Video Indexer

Speech-to-Text and Transcription. Transcribes spoken words into

text, making it easy to search through video content by dialogue.

Face and Emotion Recognition. Detects and identifies faces, along

with emotions such as happiness, sadness, or anger.

Scene and Shot Detection. Recognizes changes in scenes and

shots, allowing the video to be segmented into logical parts.

Object and Activity Detection. Identifies objects and actions

within the video, like "car," "running," or "applause."

Topic and Sentiment Analysis. Extracts topics discussed in the

video and analyzes sentiment based on the text and audio content.

Optical Character Recognition (OCR). Detects and extracts text

displayed in videos, such as subtitles, captions, or on-screen

information.

Keyword Extraction and Named Entity Recognition (NER).

Extracts keywords and named entities such as people, locations,

and organizations mentioned in the video.

Example – Part I: Upload a video

The first step in using Video Indexer is to upload a video for

analysis.

import dotenv from 'dotenv';

import fetch from 'node-fetch';

import fs from 'fs';

dotenv.config();

const endpoint = process.env.VIDEO_INDEXER_ENDPOINT; const apiKey = process.env.VIDEO_INDEXER_API_KEY; const location = process.env.LOCATION; const accountId = process.env.ACCOUNT_ID;

async function uploadVideo(filePath) {

const url = `${endpoint}/${location}/Accounts/${accountId}/Videos?

accessToken=${apiKey}&name=my-video&privacy=Private`;

const response = await fetch(url, {

method: 'POST',

headers: {

'Content-Type': 'multipart/form-data', 'Ocp-Apim-Subscription-Key': apiKey,

},

body: fs.createReadStream(filePath)

});

if (!response.ok) {

throw new Error(`Video upload failed with status:

${response.status}`);

}

const data = await response.json();

console.log("Video uploaded:", data);

return data.id; // Video ID for further processing }

// Upload a sample video

uploadVideo("path/to/video.mp4");

Example – Part II: Retrieve Insights from a Video

After uploading the video, you can retrieve various insights, such as

transcription, face recognition, and scene segmentation.

async function getVideoInsights(videoId) {

const url = `${endpoint}/${location}/Accounts/${accountId}/Videos/

${videoId}/Index?accessToken=${apiKey}`;

const response = await fetch(url, {

method: 'GET',

headers: {

'Ocp-Apim-Subscription-Key': apiKey,

}

});

if (!response.ok) {

throw new Error(`Failed to retrieve insights with status:

${response.status}`);

}

const insights = await response.json(); console.log("Video Insights:"); console.log("Transcript:", insights.videos[0].insights.transcript); console.log("Faces:", insights.videos[0].insights.faces); console.log("Keywords:", insights.videos[0].insights.keywords);

}

// Retrieve insights for the uploaded video getVideoInsights("your-video-id");

Output: The transcript, recognized faces, and extracted keywords

from the video.

Use Case: Useful for building searchable metadata for video

content, making it easy to find specific moments or people.

Immersive Reader

Azure AI Immersive Reader is a service that provides reading and

comprehension support, making it easier for users to engage with

and understand text. It’s designed to help improve reading

comprehension, especially for individuals with dyslexia, visual

impairments, or language processing difficulties. Immersive Reader

is valuable for educational applications, e-learning platforms, and

any application aiming to improve accessibility.

Key Capabilities of Azure AI Immersive Reader

Text-to-Speech. Converts text into spoken audio, allowing users to

listen to content.

Word and Line Highlighting. Highlights text as it’s read, making it

easier for users to follow along.

Translation. Translates text into different languages to support

multilingual users.

Grammar Support. Breaks down words by syllables and identifies

parts of speech like nouns, verbs, adjectives, and adverbs.

Line Focus. Focuses on one or a few lines at a time, reducing visual

clutter and improving readability.

Picture Dictionary. Displays images for certain words, which helps

users better understand the meaning.

Example - Part I: Basic server endpoint

In this example, we’ll integrate Immersive Reader into a web

application, where users can highlight and read aloud a piece of text.

Here’s a basic server endpoint to get a token:

import dotenv from 'dotenv';

import fetch from 'node-fetch';

dotenv.config();

const apiKey = process.env.IMMERSIVE_READER_API_KEY; const endpoint = process.env.IMMERSIVE_READER_API_ENDPOINT;

async function getImmersiveReaderToken() {

const url = `${endpoint}/authorize`;

const response = await fetch(url, {

method: 'POST',

headers: {

'Ocp-Apim-Subscription-Key': apiKey, 'Content-Type': 'application/x-www-form-urlencoded'

},

body: 'scope=https://cognitiveservices.azure.com/'

});

if (!response.ok) {

throw new Error(`Token err: ${response.statusText}`);

}

const tokenResponse = await response.json(); return tokenResponse.access_token;

}

// Endpoint to serve the token to client-side code export default async function (req, res) {

try {

const token = await getImmersiveReaderToken(); res.json({ token });

} catch (error) {

res.status(500).json({ error: error.message });

}

}

Example - Part II: Client-side

Use the token in the client-side to initialize Immersive Reader.

The token and text content are passed to Immersive Reader for text-

to-speech, highlighting, and more.

Text for Reading

Azure AI provides a variety of cognitive

services, enabling developers to add intelligent features to apps.

Open in Reader

From cloud to on-prem

Azure AI services offer Docker containers (for select Azure AI

services) that allow you to run the same APIs available in

Azure within your own on-premises environment. These

containers provide the flexibility to bring Azure AI capabilities

closer to your data, supporting needs around compliance,

security, and operational requirements.

Metering information

The Azure AI containers are required to submit metering

information for billing purposes.

The host should allowlist port 443 and the following domains:

*.cognitive.microsoft.com

*.cognitiveservices.azure.com

Further reading on containers

Container support is currently available for a subset of Azure

AI services.

See page: https://learn.microsoft.com/en-us/azure/ai-

services/cognitive-services-container-support

Marian Veteanu

Technology Architect and Product Leader

Excited to join an organization

where I can make an impact!

Let’s connect and explore opportunities—

message me!

https://www.linkedin.com/in/mveteanu/

https://x.com/mveteanu