7/4/2017 Webmaster

Microsoft Cognitive Custom Vision Service .Net Core Angular 4 Application (Part Two)


You can easily create a .Net Core Angular 4 application that programmatically trains your Custom Vision Service predictive model.

Microsoft’s Custom Vision Service allows you to upload sample images, classify them, train a model, and then use that model to evaluate and automatically classify your images.

It also allows you to add additional images, correct, and re-train the model. All features can be performed and invoked using the REST based API so you can incorporate all functionality into your custom programs.

This article is the third installment in a series of tutorials:

  • Custom Vision Application – Create a Custom Vision project that classifies images and allows you to upload images to be classified.
  • Angular 4 Application (Part One) – Creating an application that allows you to upload an image and have it classified.
  • (this article) Angular 4 Application (Part Two) – Upload new training images, tag them, and re-train the model.

Set-up The Application

image

To run the sample code (available on the Downloads page of this site), we first need to obtain the Prediction URL and keys for the application (created in the first tutorial of the series: Microsoft Cognitive Custom Vision Service (Tutorial Series)).

image

Next, we open the .Net Core Angular 4 application (available on the Downloads page of this site), in Visual Studio 2017 (or higher), and update and save the values in the appsettings.json file.

Run The Application

image

We run the application.

image

Click the Training tab.

image

Select one of your Projects from the Projects dropdown, then one of the Tags for the Project in the Tags dropdown.

Any images uploaded and associated with that Tag will display in the pageable list.

(Note: You create Projects and Tags for those Projects in the Custom Vision Portal )

image

To add new images to be associated with the selected Tag, click the Choose button, select the files to upload, then click the Upload New Files button.

image

The files will be added to the list of images.

image

If you log into the Custom Vision Portal and select the Project, the Training Images tab, then the Tag, you will see the images.

image

Returning to the Angular project, we can now click the Train Project button to create the predictive model.

image

We can now log into the Custom Vision Portal and select the Project, and the Performance tab, to see the Iteration that is created and the performance.

We can follow the directions in the article Angular 4 Application (Part One) to update the PredictionKey and PredictionURL in the Angular application to point to this project and allow us to upload images and make predictions (on the Home tab of the Angular application).

The API

image

The REST based API called by the Angular application is documented at the following link: Custom Vision Service API Reference.

Get Projects

image

The first code to run is the code to determine what projects you have and to populate the dropdown.

The following code is used for the markup for the dropdown:

								
									<
									label
									for="Projects"
									>Projects</
									label
									>
								
									<
									br
									/>
								
									<
									p-dropdown
									id="Projects"
								
									appendTo="body"
								
									[options]="projectsDropdown"
								
									[(ngModel)]="selectedProject"
								
									[style]="{'width':'250px'}"
								
									(onChange)="onProjectChange()">
									</
									p-dropdown>
								

This dropdown is filled by the following code in the associated code file:

								
									public
									getProjects() {
								
									this.errorMessage = "";
								

								
									// Clear Projects Dropdown
								
									this.projectsDropdown = [];
								

								
									// Call the service -- to get Projects
								
									this._CustomVisionTrainingService.getProjects()
								
									.subscribe((projects) => {
								

								
									// Temp value to hold Projects because they cannot be
									
								
									// assigned to the public property so deep inside the
									
								
									// nested iteration
								
									var tempProjectsDropdown: SelectItem[] = [];
								

								
									// Loop through the returned Projects
								
									for
									(let project of projects) {
								

								
									// Create a new SelectedItem
								
									let newSelectedItem: SelectItem = {
								
									label: project.name + ' - ' + project.description,
								
									value: project
								
									}
								

								
									// Add Selected Item to the DropDown
								
									tempProjectsDropdown.push(newSelectedItem);
								
									}
								

								
									// Add items to the DropDown
								
									this.projectsDropdown = tempProjectsDropdown;
								

								
									if
									(this.projectsDropdown[0].value
									!== 'undefined') {
								

								
									// Set the selected option to the first option
								
									this.selectedProject =
									this.projectsDropdown[0].value;
								

								
									// Get the Tags for the Project
								
									this.getTags();
								
									}
								
									},
								
									error =>
									this.errorMessage = <any>error);
								
									}

(note: this method then calls the getTags method, to fill the Tags dropdown. This will be covered in the next section)

The getProjects method calls the following getProjects service method:

								
									// ** Get Projects **
								
									getProjects(): Observable<ICustomVisionProjectResponse[]> {
								
									var _Url = 'api/training/GetProjects';
								

								
									// Call the client side code
								
									return
									this._http.get(_Url)
								
									.map((response: Response) =>
								
									<ICustomVisionProjectResponse[]>response.json())
								
									.catch(this.handleError);
								
									}

This calls the following server side .Net Core method that calls the Microsoft Custom Vision Service API:

								
									// api/training/GetProjects
								
									[HttpGet("[action]")]
								
									#region
									public
									IActionResult GetProjects()
								
									public
									IActionResult GetProjects()
								
									{
								
									List<CustomVisionProjectResponse> colCustomVisionTrainingResponse =
								
									new
									List<CustomVisionProjectResponse>();
								

								
									// Create a HttpClient to make the request
								
									using
									(HttpClient client =
									new
									HttpClient())
								
									{
								
									// Set Training Key in the request headers
									
								
									client.DefaultRequestHeaders.Add("Training-key", _TrainingKey);
								

								
									// Build the request to the Custom Vision API
								
									StringBuilder uri =
									new
									StringBuilder();
								
									uri.Append("https://southcentralus.api.cognitive.microsoft.com");
								
									uri.Append("/customvision/v2.0/Training");
								
									uri.Append("/projects?");
								

								
									// Make the request to the Custom Vision API
								
									HttpResponseMessage response = client.GetAsync(uri.ToString()).Result;
								

								
									// Get the response
								
									string
									ResponseContent = response.Content.ReadAsStringAsync().Result;
								

								
									// Convert the response to the CustomVisionTrainingResponse object
								
									colCustomVisionTrainingResponse =
								
									JsonConvert.DeserializeObject<List<CustomVisionProjectResponse>>(ResponseContent);
								
									}
								

								
									// Return the Response to the Angular application
								
									return
									Ok(colCustomVisionTrainingResponse);
								
									}
								
									#endregion

Get Tags

The code that gets the Tags is similar to the code that gets the Projects. The difference is that the currently selected Project is used as a parameter because a set of Tags is always associated with a specific Project.

Markup:

								
									<
									label
									for="Tags"
									>Tags</
									label
									>
								
									<
									br
									/>
								
									<
									p-dropdown
									id="Tags"
								
									appendTo="body"
								
									[options]="tagsDropdown"
								
									[(ngModel)]="selectedTag"
								
									[style]="{'width':'250px'}"
								
									(onChange)="onTagChange()">
									</
									p-dropdown>
								

Code file:

								
									public
									getTags() {
								
									this.errorMessage = "";
								

								
									// Clear TotalRecords
								
									this.totalRecords = 0;
								

								
									// Clear Tags Dropdown
								
									this.tagsDropdown = [];
								

								
									// Call the service -- to get Tags
								
									this._CustomVisionTrainingService.getTags(this.selectedProject)
								
									.subscribe((tags) => {
								

								
									// Temp value to hold Tags because they cannot be
									
								
									// assigned to the public property so deep inside the
									
								
									// nested iteration
								
									var tempTagsDropdown: SelectItem[] = [];
								

								
									// Loop through the returned Tags
								
									for
									(let tag of tags.tags) {
								

								
									// Create a new SelectedItem
								
									let newSelectedItem: SelectItem = {
								
									label: tag.name,
								
									value: tag
								
									}
								

								
									// Add Selected Item to the DropDown
								
									tempTagsDropdown.push(newSelectedItem);
								
									}
								

								
									// Add items to the DropDown
								
									this.tagsDropdown = tempTagsDropdown;
								

								
									if
									(this.tagsDropdown[0].value
									!== 'undefined') {
								

								
									// Set the selected option to the first option
								
									this.selectedTag =
									this.tagsDropdown[0].value;
								

								
									// Set defaults
								
									this.currentPage = 1;
								
								
									// Get the Images for the Tag
								
									this.getImages();
								
									}
								
									},
								
									error =>
									this.errorMessage = <any>error);
								
									}

Service:

								
									// ** Get Tags **
								
									getTags(paramProject: ICustomVisionProjectResponse):
								
									Observable<ICustomVisionTagResponse> {
								
									var _Url = 'api/training/GetTags?projectId=' + paramProject.id;
								

								
									// Call the client side code
								
									return
									this._http.get(_Url)
								
									.map((response: Response) =>
								
									<ICustomVisionTagResponse>response.json())
								
									.catch(this.handleError);
								
									}

Server side code:

								
									// api/training/GetTags
								
									[HttpGet("[action]")]
								
									#region
									public
									IActionResult GetTags(string
									projectId)
								
									public
									IActionResult GetTags(string
									projectId)
								
									{
								
									List<CustomVisionTagResponse> objCustomVisionTagResponse =
								
									new
									List<CustomVisionTagResponse>();
								

								
									// Create a HttpClient to make the request
								
									using
									(HttpClient client =
									new
									HttpClient())
								
									{
								
									// Set Training Key in the request headers
									
								
									client.DefaultRequestHeaders.Add("Training-key", _TrainingKey);
								

								
									// Build the request to the Custom Vision API
								
									StringBuilder uri =
									new
									StringBuilder();
								
									uri.Append("https://southcentralus.api.cognitive.microsoft.com");
								
									uri.Append("/customvision/v2.0/Training");
								
									uri.Append("/projects/" + projectId);
								
									uri.Append("/tags?");
								

								
									// Make the request to the Custom Vision API
								
									HttpResponseMessage response = client.GetAsync(uri.ToString()).Result;
								

								
									// Get the response
								
									string
									ResponseContent = response.Content.ReadAsStringAsync().Result;
								

								
									// Convert the response to the CustomVisionTagResponse object
								
									objCustomVisionTagResponse =
								
									JsonConvert.DeserializeObject<List<CustomVisionTagResponse>>(ResponseContent);
								
									}
								

								
									// Return the Response to the Angular application
								
									return
									Ok(objCustomVisionTagResponse);
								
									}
								
									#endregion

Images

image

When a Project and a Tag are selected, the code to display the images associated with the selected Tag runs.

The following is the markup:

								
									<
									p-dataGrid
									[value]="images"
								
									[paginator]="true"
								
									[rows]="8"
								
									[lazy]="true"
								
									[totalRecords]="totalRecords"
								
									(onLazyLoad)="loadData($event)">
								
									<
									p-header
									>
								
									List of Images
								
									</
									p-header>
								
									<
									ng-template
									let-image
									pTemplate="item"
									>
								
									<
									div
									style="padding:3px"
									class="ui-g-12 ui-md-3"
									>
								
									<
									p-panel
									[header]="image.id" [style]="{'text-align':'center'}">
								
									<
									a
									href="{{image.imageUri}}"
								
									target="_blank"
									>
								
									<
									img
									src="{{image.thumbnailUri}}"
								
									width="100"
									height="100"
								
									style="border:2px solid blue"
									>
									</
									a
									>
								
									</
									p-panel>
								
									</
									div
									>
								
									</
									ng-template>
								
									</
									p-dataGrid>
								

Note, that the PrimeNG dataGrid control uses Lazy Loading , meaning data is requested one page at a time.

The following shows the method, in the associated code file, that is called when the user requests each page of data:

								
									loadData(event) {
								
									// Set Current Page
								
									this.currentPage = ((event.first + 8) / 8);
								

								
									// Get the Images for the Tag
								
									this.getImages();
								
									}

The code above calls the getImages method, that gets the image data:

								
									getImages() {
								
									this.errorMessage = "";
								

								
									// Clear Images
								
									this.images = [];
								

								
									// Call the service -- to get Images
								
									this._CustomVisionTrainingService.getImages(
								
									this.selectedProject,
								
									this.selectedTag.id,
								
									this.currentPage.toString()).subscribe((images) => {
								

								
									// Temp value to hold Images because they cannot be
									
								
									// assigned to the public property so deep inside the
									
								
									// nested iteration
								
									var tempImags: ICustomVisionImageResponse[] = [];
								

								
									// Loop through the returned Images
								
									for
									(let image of images) {
								

								
									// Add Selected Item to the list of images
								
									tempImags.push(image);
								
									}
								

								
									// Add items to the Image list
								
									this.images = tempImags;
								

								
									// Set total records
								
									this.totalRecords =
									this.selectedTag.imageCount;
								
									},
								
									error =>
									this.errorMessage = <any>error);
								
									}

This calls the following service method:

								
									// ** Get Images **
								
									getImages(
								
									paramProject: ICustomVisionProjectResponse,
								
									paramTag:
									string,
								
									paramPageNumber:
									string): Observable<ICustomVisionImageResponse[]> {
								
									var _Url = 'api/training/GetImages?'
								
									+ 'projectId=' + paramProject.id
								
									+ '&tagId=' + paramTag
								
									+ '&pageNumber=' + paramPageNumber;
								

								
									// Call the client side code
								
									return
									this._http.get(_Url)
								
									.map((response: Response) =>
								
									<ICustomVisionImageResponse[]>response.json())
								
									.catch(this.handleError);
								
									}

This calls the following server side method:

								
									
										// api/training/GetImages
								
									[HttpGet("[action]")]
								
									#region
									public
									IActionResult GetImages(string
									projectId,
									string
									tagId,
									string
									pageNumber)
								
									public
									IActionResult GetImages(string
									projectId,
									string
									tagId,
									string
									pageNumber)
								
									{
								
									List<CustomVisionImageResponse> colCustomVisionImageResponse =
								
									new
									List<CustomVisionImageResponse>();
								

								
									if
									((tagId !=
									null) && (projectId !=
									null))
								
									{
								
									// Create a HttpClient to make the request
								
									using
									(HttpClient client =
									new
									HttpClient())
								
									{
								
									// Set Training Key in the request headers
									
								
									client.DefaultRequestHeaders.Add("Training-key", _TrainingKey);
								

								
									// Build the request
								
									string
									SkipNumber = ((Convert.ToInt32(pageNumber) - 1) * 8).ToString();
								

								
									StringBuilder uri =
									new
									StringBuilder();
								
									uri.Append("https://southcentralus.api.cognitive.microsoft.com");
								
									uri.Append("/customvision/v2.0/Training");
								
									uri.Append("/projects/" + projectId);
								
									uri.Append("/images/tagged?");
								
									uri.Append("tagIds=" + tagId);
								
									uri.Append("&take=" + "8");
								
									uri.Append("&skip=" + SkipNumber);
								

								
									// Make the request to the Custom Vision API
								
									HttpResponseMessage response = client.GetAsync(uri.ToString()).Result;
								

								
									// Get the response
								
									string
									ResponseContent = response.Content.ReadAsStringAsync().Result;
								

								
									// Convert the response to the CustomVisionImageResponse object
								
									colCustomVisionImageResponse =
								
									JsonConvert.DeserializeObject<List<CustomVisionImageResponse>>(ResponseContent);
								
									}
								
									}
								

								
									// Return the Response to the Angular application
								
									return
									Ok(colCustomVisionImageResponse);
								
									}
								
									#endregion

Uploading

image

When a Project and a Tag are selected, you can upload images that will then be associated with the Tag.

The following is the markup for the PrimeNG upload control:

								
									<
									p-fileUpload
									name="myfile[]"
								
									url="api/training"
								
									multiple="true"
								
									accept="image/*"
								
									maxFileSize="10000000"
								
									uploadLabel="Upload New Files"
								
									(onBeforeUpload)="onBeforeUploadHandler($event)"
								
									(onUpload)="onUpload($event)">
								
									</
									p-fileUpload>
								

This has two associated methods in the code file, onBeforeUploadHandler to pass the selected Project and Tag to the server side code, and onUpload, that is called after the upload, to refresh the images in the dataGrid:

								
									// Upload
								

								
									public
									onBeforeUploadHandler(event) {
								
									// called before the file(s) are uploaded
								
									// Send the currently selected Project and Tag in the Header
								
									event.formData.append("selectedProject",
									this.selectedProject.id);
								
									event.formData.append("selectedTag",
									this.selectedTag.id);
								
									}
								

								
									public
									onUpload(event) {
								
									// This is called after the upload
								
									this.getImages();
								
									}

The following server side method handles the upload:

								
									// api/training
								
									[HttpPost]
								
									#region
									public
									IActionResult Index()
								
									public
									IActionResult Index()
								
									{
								
									if
									(!Request.HasFormContentType)
								
									{
								
									return
									BadRequest();
								
									}
								

								
									// Retrieve data from Form
								
									var form = Request.Form;
								

								
									// Retrieve Project and Tag
								
									string
									selectedProject = form["selectedProject"].First();
								
									string
									selectedTag = form["selectedTag"].First();
								

								
									if
									((selectedProject !=
									null) && (selectedTag !=
									null))
								
									{
								
									// Process all Files
								
									foreach
									(var file
									in
									form.Files)
								
									{
								
									// Process file
								
									using
									(var readStream = file.OpenReadStream())
								
									{
								
									// Create a HttpClient to make the request
								
									using
									(HttpClient client =
									new
									HttpClient())
								
									{
								
									// Set Training Key in the request headers
									
								
									client.DefaultRequestHeaders.Add("Training-key", _TrainingKey);
								

								
									// Serialize Request
								
									MultipartFormDataContent _multiPartContent =
								
									new
									MultipartFormDataContent();
								

								
									StreamContent _imageData =
								
									new
									StreamContent(readStream);
								

								
									_imageData.Headers.ContentType =
								
									new
									MediaTypeHeaderValue("application/octet-stream");
								

								
									ContentDispositionHeaderValue _contentDispositionHeaderValue =
								
									new
									ContentDispositionHeaderValue("form-data");
								

								
									_contentDispositionHeaderValue.Name = "imageData";
								
									_contentDispositionHeaderValue.FileName = file.Name;
								
									_imageData.Headers.ContentDisposition = _contentDispositionHeaderValue;
								

								
									_multiPartContent.Add(_imageData, "imageData");
								

								
									// Build the request to the Custom Vision API
								
									StringBuilder uri =
									new
									StringBuilder();
								
									uri.Append("https://southcentralus.api.cognitive.microsoft.com");
								
									uri.Append("/customvision/v2.0/Training");
								
									uri.Append("/projects/" + selectedProject);
								
									uri.Append("/images?");
								
									uri.Append("tagIds=" + selectedTag);
								

								
									// Make the request to the Custom Vision Service API
								
									HttpResponseMessage response =
								
									client.PostAsync(uri.ToString(), _multiPartContent).Result;
								

								
									// Get the response -- (Do nothing with it for now)
								
									string
									ResponseContent = response.Content.ReadAsStringAsync().Result;
								
									}
								
									}
								
									}
								
									}
								

								
									return
									Ok();
								
									}
								
									#endregion

Train Project

image

After uploading new images, you need to Train the project to create a new predictive model.

The following is the markup for the button:

								
									<
									button
									pButton
									type="button"
								
									(click)="TrainProject()"
								
									label="Train Project"
								
									icon="fa-plus-square"
								
									class="ui-button-success"
									>
									</
									button
									>
								

This calls the following method:

								
									TrainProject() {
								
									// Call the service -- to train the project
								
									this._CustomVisionTrainingService.trainProject(this.selectedProject)
								
									.subscribe((response: ICustomVisionTrainResponse) => {
								
									alert(response.status);
								
									});
								
									}

This calls the following service method:

								
									// ** Train Project **
								
									trainProject(paramProject: ICustomVisionProjectResponse):
								
									Observable<ICustomVisionTrainResponse> {
								
									var _Url = 'api/training/TrainProject?projectId=' + paramProject.id;
								

								
									// Call the client side code
								
									return
									this._http.get(_Url)
								
									.map((response: Response) =>
								
									<ICustomVisionTrainResponse>response.json())
								
									.catch(this.handleError);
								
									}

This calls the following server side code:

								
									// api/training/TrainProject
								
									[HttpGet("[action]")]
								
									#region
									public
									IActionResult TrainProject(string
									projectId)
								
									public
									IActionResult TrainProject(string
									projectId)
								
									{
								
									CustomVisionTrainResponse objCustomVisionTrainResponse =
									new
									CustomVisionTrainResponse();
								

								
									// Create a HttpClient to make the request
								
									using
									(HttpClient client =
									new
									HttpClient())
								
									{
								
									HttpResponseMessage response;
								

								
									// Set Training Key in the request headers
									
								
									client.DefaultRequestHeaders.Add("Training-key", _TrainingKey);
								

								
									// Build the request to the Custom Vision API
								
									StringBuilder uri =
									new
									StringBuilder();
								
									uri.Append("https://southcentralus.api.cognitive.microsoft.com");
								
									uri.Append("/customvision/v2.0/Training");
								
									uri.Append("/projects/" + projectId);
								
									uri.Append("/train?");
								

								
									// Request body
								
									byte[] byteData = Encoding.UTF8.GetBytes("{body}");
								

								
									// Make the request to the Custom Vision Service API
								
									using
									(var content =
									new
									ByteArrayContent(byteData))
								
									{
								
									content.Headers.ContentType =
									new
									MediaTypeHeaderValue("application/json");
								
									response = client.PostAsync(uri.ToString(), content).Result;
								
									}
								

								
									if
									(response.StatusCode == System.Net.HttpStatusCode.BadRequest)
								
									{
								
									objCustomVisionTrainResponse.Status = "No Training needed";
								
									}
								
									else
								
									{
								
									// Get the response
								
									string
									ResponseContent = response.Content.ReadAsStringAsync().Result;
								

								
									// Convert the response to the CustomVisionTrainResponse object
								
									objCustomVisionTrainResponse =
								
									JsonConvert.DeserializeObject<CustomVisionTrainResponse>(ResponseContent);
								
									}
								
									}
								

								
									// Return a Response to the Angular application
								
									return
									Ok(objCustomVisionTrainResponse);
								
									}
								
									#endregion

Links

https://www.customvision.ai/

Custom Vision Documentation

Custom Vision Service API Reference

PrimeNG dataGrid

PrimeNG upload

Download

You can download the code from the Download page

An error has occurred. This application may no longer respond until reloaded. Reload 🗙