Convert Speech to Text in Power Apps

This post is part 1 of a 3 part series on Converting Speech to Text in Power Apps.

In this article, we will convert Speech recorded with the Microphone Control of Power Apps to Text using Azure Cognitive Services.

This is an advanced topic related to a business scenario since it effectively allows a Power User to consume the Speech API in Azure Cognitive services for converting Speech to Text.

To reduce the complexity, we will divide this article in four parts:

1. Design a Canvas App with The Microphone Control to capture Audio.
2. Create an Azure Function to convert audio captured in Power Apps from WEBM to WAV format using FFmpeg.
3. Create a Power Automate (Flow) to create an HTML file, using the text obtained from the output of the Speech to Text action.

Prerequisites-

Before you begin, please make sure the following prerequisites are in place:

• An Office 365 subscription with access to PowerApps and Power Automate (Flow).
• An Azure Subscription to create a Function App.
• Muhimbi PDF Converter Services Online full, free or trial subscription (Start trial)
• Power Automate Plan for consuming the HTTP Custom actions.
• Appropriate privileges to create Power Apps and Power Automate(Flow).
• Working knowledge of both Power Apps and Power Automate(Flow).

Designing a Canvas App with The Microphone Control to Capture Audio

Step 1- Creating the basic structure of The Canvas App

• Go to powerapps.com, sign in with your work or school account, click on the Apps menu in the left navigation bar, and then click on ‘+Create’ and Select Canvas app from blank.
• Specify the Name and the Format of the APP and Click on ‘Create’.

• Add a Microphone control as below.

• Now add a Button to the Canvas as shown below.

• Next, rename the Button to ‘Submit’ as shown.

• Add two Lables to the Canvas that we will use as containers to hold values.

• Rename Label 1 to ‘AudioJSON’ and Label 2 to ‘TextOutput’, as shown.

• The final structure of Power Apps is shown below.

• Now that we have the outer body ready, let’s go ahead and configure our components with formulas.

Step 2-  Configuring the components with Formula’s

• We will first configure a collection called “AudioCollection” to add a recorded Audio sample .
• Then,we’ll create variable ‘JSONValue’ and set it with the JSON of the audio.
• Select the ‘OnStop’ property of Microphone and add the following formula to it:

ClearCollect(AudioCollection,Microphone1.Audio);Set(JSONValue,JSON(AudioCollection,JSONFormat.IncludeBinaryData));

• Select the ‘Text’property of ‘AudioJSON’ label as below to display it’s value set earlier.

• Now it’s time to add a Power Automate (Flow) to our Power Apps.
• Inside the Action menu, there is an option to add ‘Power Automate’ to your existing Power Apps. To do this, click on the ‘Power Automate’ option as highlighted.

• Then, click on ‘Create a new Flow’ as shown below.

• Rename the Power Automate (Flow) to “Convert Speech To Text” and add ‘PowerApps’ as a trigger .
• Once that has been done, add a ‘Compose Action’and select ‘Ask in PowerApps’ from the Dynamic content in the pane on the right side of the image below.
• Make sure you click on Save.

Note – We completed these steps in the Power Automate (Flow) just so that we can get the Power Automate (Flow) added to our Power Apps. Later in this article,  we will add more actions to the Power Automate (Flow), so as to carry out a Speech to Text conversion.

• Finally select the ‘OnSelect’ property of the ‘Submit’ button and add the following formula.

Set(DesiredOutput,’ConvertSpeechToText’.Run(JSON(First(AudioCollection),JSONFormat.IncludeBinaryData)));

• The above formula tells the ‘Submit’ button to trigger a Power Automate (Flow) with name ‘ConvertSpeechToText‘ created earlier using the .Run() method in which we are passing the JSON of the first audio sample sitting in the Audio Collection.
• Now that we have our Power App ready, let’s head towards an Azure function.

Now that we have our Power App ready let’s head towards creating and configuring an Azure function to convert Audio captured in Power Apps from Webm to Wav format using FFMpeg in the next post.

---