Skip to content

AI-900

Pre-Trained Models(APIs)

Cognitive Services

Bring AI within reach of every developer

  • AI without building custom models
  • Does NOT need machine-learning expertise
  • Exposed as APIs

Services:

  • Images/videos: Computer vision, Face API, Form Recognizer
  • Text: Text Analytics, Translator Text, Text-to-Speech API
  • Audio: Speech-to-Text API, Language Understanding Intelligent Service - LUIS
  • Others:
  • Conversations (QnA Maker, Azure Bot Service)
  • Anomaly Detector service, Content moderator
  • Computer Vision - Analyze content in images and videos
  • Face API - Detect and identify people and emotions in images
  • Custom Vision - Customize image recognition to fit your business

Vision Terminology

  • Image analysis: Extract tags from image
  • Create text description about an image
  • Image classification: Classify image into different groups
  • Object detection: Identify objects in image
  • For each object:
    • class of object, confidence level, coordinates of a bounding box
  • Goes deeper than image classification
  • Face detection: Detect human faces
  • Face detection and analysis: Security, tag friends on facebook, Identity validation
  • Optical character recognition (OCR): Detect text in images (license plates, invoices etc)

FaceAPI

  • Improve accuracy of face identification:
  • Quality of images (Higher face image quality)
    • Recommendations: frontal, clear, and face size - 200x200 pixels or bigger
  • Variety of images: From diff. angles in variety of lighting setups
  • Important concepts:
  • FaceList (up to 1K faces) & LargeFaceList (up to 1M faces)
  • PersonGroup (up to 1K persons) & LargePersonGroup (up to 1M persons)
    • Each person can have multiple face images
  • Operations:
  • Detect: Detect human faces (box co-ordinates)
  • Find Similar: Find similar faces (Find images of this specific person)
  • Group: Divide candidate faces (3-1000) into groups based on face similarity
  • Identify: 1-to-many identification
  • Verify
    • Do two faces belong to same person?
    • Does a face belong to a specific person?

Form Recognizer API

Operations (over pdf or image):

  • Analyze Business Card (Get Analyze Business Card Result)
  • Analyze ID Document
  • Analyze Invoice
  • Analyze Receipt
  • Custom Form: Design & extract key-value pairs, tables, and semantic values from custom documents - pdf or image

Search APIs(Bing)

  • Web Search API: Execute web search using API
  • Image Search API: search for relevant images
  • News Search API: search for relevant news articles
  • Visual Search API: search based on an image

Text Analytics API

Text Analytics: Natural language processing (NLP)

Operations:

  • Detect Language
  • Language Name,ISO 6391 Code, Score (NaN-ambiguous)
  • Entities containing personal information
  • Returns a list of entities with personal information ("SSN", "Bank Account" etc) in the document
  • Key Phrases: returns a list of strings denoting the key phrases
  • Example: Summarize a document
  • Named Entity Recognition: list of general named entities in a given document
  • Person, Location, Organization, Quantity, DateTime, URL, Phone Number, IP Address etc
  • Sentiment: detailed sentiment analysis
  • positive/negative review - example: 0.1(negative), 0.9(positive)

Translator & Speech API

  • Speech-to-Text API
  • Text-to-Speech API
  • Translation

Conversational AI

  • QnA Maker: Convert your FAQ into a Q&A Bot
  • You need a knowledge base (cannot talk to db)
  • Azure Bot Service: Build your own bots
  • Enable multiple conversation channels(Alexa, Office 365 email, Facebook, Microsoft Teams, Skype, Slack, Telegram ....)
  • Recommended Architecture: QnA Maker service + Azure Bot Service

LUIS: Language Understanding Intelligence Service

  • Understands spoken (and text) commands
  • Detects Intents (FoodOrder, BookFlight) and Entities (Pizzas, Cairo)

Understanding Decision Services

  • Anamaly Detector: Find anomalies
  • Content Moderator: Detect unwanted content

Building ML Models

Steps

1: Obtain Data 2: Clean Data 3: Feature Engineering: Identify Features and Label 4: Create a Model using the Dataset and the ML algorithm 5: Evaluate the accuracy of the model 6: Deploy the model for use

Terminology

  • Process
  • Training: The process of creating a model
  • Evaluation: Is the model working?
  • Inference: Using model to do predictions in production
  • Dataset: Data used to create, validate & test the model
  • Features: Inputs
  • Label: Output/Prediction
  • Dataset Types
    • Training Dataset: Dataset used to create a model
    • Validation Dataset: Dataset used to validate the model (and choose the right algorithm) - Model Evaluation
    • Testing Dataset: Dataset used to do final testing before deployment

Azure Machine Learning

  • Azure Machine Learning: Simplifies creation of your models
  • Manage data, code, compute, models etc
  • Prepare data
  • Train models
  • Publish models
  • Monitor models
  • Multiple options to create models
  • Automated machine learning: Build custom models with minimum ML expertise
  • Azure Machine Learning designer: Enables no-code development of models
    • Build Your Own Models: Data Scientists
    • Data and compute management, pipelines

Terminology:

  • Studio: Website for Azure Machine Learning
  • Workspace: Top-level resource for Azure Machine Learning
  • Azure Machine Learning designer: drag-and-drop interface to create your ML workflows(Canvas)
    • Pipelines: Reusable workflows (training and re-training)
    • Datasets: Manage your data
    • Module: An algorithm to run on your data
    • Data preparation: Data Transformation, Feature Selection
    • Machine learning algorithms: Regression, Classification, Clustering
    • Building and evaluating models: Model Training, Model Scoring and Evaluation
    • Compute:
    • Compute Instances: Development machines (CPU or GPU instances) for data engineers and data scientists
    • Compute Clusters: Training machines
      • Single or multi node compute cluster for your training
    • Inference Clusters: Deployment machines
      • Deploy your model to Azure Kubernetes Service or Azure Container Instances
    • Attached Compute: Use HDInsight cluster, a Virtual Machine, or a Databricks cluster as target for Azure Machine Learning workspace

Classification

  • Accuracy: The ratio of correct predictions (true positives + true negatives) to the total number of predictions.
  • Precision: The fraction of positive cases correctly identified (the number of true positives divided by the number of true positives plus false positives).
  • Recall: The fraction of the cases classified as positive that are actually positive (the number of true positives divided by the number of true positives plus false negatives).
  • F1 Score: An overall metric that essentially combines precision and recall.

AI considerations

Responsible AI Principles

  • Fairness
  • Reliability and safety
  • Privacy and security
  • Inclusiveness
  • Transparency
  • Accountability

Remember

  • A cognitive services resource can be used for both training and prediction.
  • Creating a Custom Vision resource for both training and prediction results in two separate resources being provisioned, each with their own key and endpoint.
  • Extreme angles may impair facial detection
  • The Read API is better suited for larger images and it runs asynchronously so it will not block your application while it is running
  • The maximum file size for the pre-built receipt model is 50 MB
  • Computer Vision returns objects with a bounding box to indicate their location in the image.
  • The Translator service can translate from a text source, but not from an audio source.
  • To author a Conversational Language Understanding model, you need a Language resource.
  • Question Answering service is used to create a knowledge base for bots.
  • Language Detection will return NaN when it cannot determine the language in the provided text.
  • Seasonal times series is considered to be a pattern in your data, that occurs at regular intervals.
  • Cognitive Search can index JSON documents. JSON is also used to define index schemas, indexers, and data source objects.
  • An indexer serializes a source document into JSON before passing it to a search engine for indexing. An indexer automates several steps of data ingestion, reducing the amount of code you need to write.
  • Cognitive Search is used for full text search over indexes containing alphanumeric content.
  • Anomaly detection requires time series data.
  • A learning model learns the relationship between the features and the label, a hyperparameter is used to tune machine learning model - the columns in the dataset are not hyperparameters.
  • Tabular and File datasets are the data source types supported in Azure Auto ML.
  • Azure Machine Learning Studio does not support the use of C# or .NET framework. You can use VS or VSCode to create a model in C# using the ML.NET SDK.
  • Sample datasets are included with the Azure Machine Learning designer by default.New Azure Machine learning datesets are merely references to the original data source locations, with a copy of metadata.
  • Azure Machine Learning compute can be used to run your pipeline for training and experimentation. To host your deployed models, you can use Azure Kubernetes Services.
  • The Analyze Image API of Computer Vision can detect adult content, either sexual or gory.
  • A channel is added to make a bot available on a different platform.
  • In the Azure Bot Service a skill is a bot that can extend the functionality of another bot by performing a set of tasks for it. A skill manifest is a JSON file. A root bot is a user-facing bot that is also a skill consumer.