7/4/2017 Webmaster
Microsoft Cognitive Custom Vision Service .Net Core Angular 4 Application (Part Two)
You can easily create a .Net Core Angular 4 application that programmatically trains your Custom Vision Service predictive model.
Microsoft’s Custom Vision Service allows you to upload sample images, classify them, train a model, and then use that model to evaluate and automatically classify your images.
It also allows you to add additional images, correct, and re-train the model. All features can be performed and invoked using the REST based API so you can incorporate all functionality into your custom programs.
This article is the third installment in a series of tutorials:
- Custom Vision Application – Create a Custom Vision project that classifies images and allows you to upload images to be classified.
- Angular 4 Application (Part One) – Creating an application that allows you to upload an image and have it classified.
- (this article) Angular 4 Application (Part Two) – Upload new training images, tag them, and re-train the model.
Set-up The Application
To run the sample code (available on the Downloads page of this site), we first need to obtain the Prediction URL and keys for the application (created in the first tutorial of the series: Microsoft Cognitive Custom Vision Service (Tutorial Series)).
Next, we open the .Net Core Angular 4 application (available on the Downloads page of this site), in Visual Studio 2017 (or higher), and update and save the values in the appsettings.json file.
Run The Application
We run the application.
Click the Training tab.
Select one of your Projects from the Projects dropdown, then one of the Tags for the Project in the Tags dropdown.
Any images uploaded and associated with that Tag will display in the pageable list.
(Note: You create Projects and Tags for those Projects in the Custom Vision Portal )
To add new images to be associated with the selected Tag, click the Choose button, select the files to upload, then click the Upload New Files button.
The files will be added to the list of images.
If you log into the Custom Vision Portal and select the Project, the Training Images tab, then the Tag, you will see the images.
Returning to the Angular project, we can now click the Train Project button to create the predictive model.
We can now log into the Custom Vision Portal and select the Project, and the Performance tab, to see the Iteration that is created and the performance.
We can follow the directions in the article Angular 4 Application (Part One) to update the PredictionKey and PredictionURL in the Angular application to point to this project and allow us to upload images and make predictions (on the Home tab of the Angular application).
The API
The REST based API called by the Angular application is documented at the following link: Custom Vision Service API Reference.
Get Projects
The first code to run is the code to determine what projects you have and to populate the dropdown.
The following code is used for the markup for the dropdown:
< label for="Projects" >Projects</ label >< br />< p-dropdown id="Projects"appendTo="body"[options]="projectsDropdown"[(ngModel)]="selectedProject"[style]="{'width':'250px'}"(onChange)="onProjectChange()"> </ p-dropdown>
This dropdown is filled by the following code in the associated code file:
public getProjects() {this.errorMessage = "";// Clear Projects Dropdownthis.projectsDropdown = [];// Call the service -- to get Projectsthis._CustomVisionTrainingService.getProjects().subscribe((projects) => {// Temp value to hold Projects because they cannot be// assigned to the public property so deep inside the// nested iterationvar tempProjectsDropdown: SelectItem[] = [];// Loop through the returned Projectsfor (let project of projects) {// Create a new SelectedItemlet newSelectedItem: SelectItem = {label: project.name + ' - ' + project.description,value: project}// Add Selected Item to the DropDowntempProjectsDropdown.push(newSelectedItem);}// Add items to the DropDownthis.projectsDropdown = tempProjectsDropdown;if (this.projectsDropdown[0].value !== 'undefined') {// Set the selected option to the first optionthis.selectedProject = this.projectsDropdown[0].value;// Get the Tags for the Projectthis.getTags();}},error => this.errorMessage = <any>error);}
(note: this method then calls the getTags method, to fill the Tags dropdown. This will be covered in the next section)
The getProjects method calls the following getProjects service method:
// ** Get Projects **getProjects(): Observable<ICustomVisionProjectResponse[]> {var _Url = 'api/training/GetProjects';// Call the client side codereturn this._http.get(_Url).map((response: Response) =><ICustomVisionProjectResponse[]>response.json()).catch(this.handleError);}
This calls the following server side .Net Core method that calls the Microsoft Custom Vision Service API:
// api/training/GetProjects[HttpGet("[action]")]#region public IActionResult GetProjects()public IActionResult GetProjects(){List<CustomVisionProjectResponse> colCustomVisionTrainingResponse =new List<CustomVisionProjectResponse>();// Create a HttpClient to make the requestusing (HttpClient client = new HttpClient()){// Set Training Key in the request headersclient.DefaultRequestHeaders.Add("Training-key", _TrainingKey);// Build the request to the Custom Vision APIStringBuilder uri = new StringBuilder();uri.Append("https://southcentralus.api.cognitive.microsoft.com");uri.Append("/customvision/v2.0/Training");uri.Append("/projects?");// Make the request to the Custom Vision APIHttpResponseMessage response = client.GetAsync(uri.ToString()).Result;// Get the responsestring ResponseContent = response.Content.ReadAsStringAsync().Result;// Convert the response to the CustomVisionTrainingResponse objectcolCustomVisionTrainingResponse =JsonConvert.DeserializeObject<List<CustomVisionProjectResponse>>(ResponseContent);}// Return the Response to the Angular applicationreturn Ok(colCustomVisionTrainingResponse);}#endregion
Get Tags
The code that gets the Tags is similar to the code that gets the Projects. The difference is that the currently selected Project is used as a parameter because a set of Tags is always associated with a specific Project.
Markup:
< label for="Tags" >Tags</ label >< br />< p-dropdown id="Tags"appendTo="body"[options]="tagsDropdown"[(ngModel)]="selectedTag"[style]="{'width':'250px'}"(onChange)="onTagChange()"> </ p-dropdown>
Code file:
public getTags() {this.errorMessage = "";// Clear TotalRecordsthis.totalRecords = 0;// Clear Tags Dropdownthis.tagsDropdown = [];// Call the service -- to get Tagsthis._CustomVisionTrainingService.getTags(this.selectedProject).subscribe((tags) => {// Temp value to hold Tags because they cannot be// assigned to the public property so deep inside the// nested iterationvar tempTagsDropdown: SelectItem[] = [];// Loop through the returned Tagsfor (let tag of tags.tags) {// Create a new SelectedItemlet newSelectedItem: SelectItem = {label: tag.name,value: tag}// Add Selected Item to the DropDowntempTagsDropdown.push(newSelectedItem);}// Add items to the DropDownthis.tagsDropdown = tempTagsDropdown;if (this.tagsDropdown[0].value !== 'undefined') {// Set the selected option to the first optionthis.selectedTag = this.tagsDropdown[0].value;// Set defaultsthis.currentPage = 1;// Get the Images for the Tagthis.getImages();}},error => this.errorMessage = <any>error);}
Service:
// ** Get Tags **getTags(paramProject: ICustomVisionProjectResponse):Observable<ICustomVisionTagResponse> {var _Url = 'api/training/GetTags?projectId=' + paramProject.id;// Call the client side codereturn this._http.get(_Url).map((response: Response) =><ICustomVisionTagResponse>response.json()).catch(this.handleError);}
Server side code:
// api/training/GetTags[HttpGet("[action]")]#region public IActionResult GetTags(string projectId)public IActionResult GetTags(string projectId){List<CustomVisionTagResponse> objCustomVisionTagResponse =new List<CustomVisionTagResponse>();// Create a HttpClient to make the requestusing (HttpClient client = new HttpClient()){// Set Training Key in the request headersclient.DefaultRequestHeaders.Add("Training-key", _TrainingKey);// Build the request to the Custom Vision APIStringBuilder uri = new StringBuilder();uri.Append("https://southcentralus.api.cognitive.microsoft.com");uri.Append("/customvision/v2.0/Training");uri.Append("/projects/" + projectId);uri.Append("/tags?");// Make the request to the Custom Vision APIHttpResponseMessage response = client.GetAsync(uri.ToString()).Result;// Get the responsestring ResponseContent = response.Content.ReadAsStringAsync().Result;// Convert the response to the CustomVisionTagResponse objectobjCustomVisionTagResponse =JsonConvert.DeserializeObject<List<CustomVisionTagResponse>>(ResponseContent);}// Return the Response to the Angular applicationreturn Ok(objCustomVisionTagResponse);}#endregion
Images
When a Project and a Tag are selected, the code to display the images associated with the selected Tag runs.
The following is the markup:
< p-dataGrid [value]="images"[paginator]="true"[rows]="8"[lazy]="true"[totalRecords]="totalRecords"(onLazyLoad)="loadData($event)">< p-header >List of Images</ p-header>< ng-template let-image pTemplate="item" >< div style="padding:3px" class="ui-g-12 ui-md-3" >< p-panel [header]="image.id" [style]="{'text-align':'center'}">< a href="{{image.imageUri}}"target="_blank" >< img src="{{image.thumbnailUri}}"width="100" height="100"style="border:2px solid blue" > </ a ></ p-panel></ div ></ ng-template></ p-dataGrid>
Note, that the PrimeNG dataGrid control uses Lazy Loading , meaning data is requested one page at a time.
The following shows the method, in the associated code file, that is called when the user requests each page of data:
loadData(event) {// Set Current Pagethis.currentPage = ((event.first + 8) / 8);// Get the Images for the Tagthis.getImages();}
The code above calls the getImages method, that gets the image data:
getImages() {this.errorMessage = "";// Clear Imagesthis.images = [];// Call the service -- to get Imagesthis._CustomVisionTrainingService.getImages(this.selectedProject,this.selectedTag.id,this.currentPage.toString()).subscribe((images) => {// Temp value to hold Images because they cannot be// assigned to the public property so deep inside the// nested iterationvar tempImags: ICustomVisionImageResponse[] = [];// Loop through the returned Imagesfor (let image of images) {// Add Selected Item to the list of imagestempImags.push(image);}// Add items to the Image listthis.images = tempImags;// Set total recordsthis.totalRecords = this.selectedTag.imageCount;},error => this.errorMessage = <any>error);}
This calls the following service method:
// ** Get Images **getImages(paramProject: ICustomVisionProjectResponse,paramTag: string,paramPageNumber: string): Observable<ICustomVisionImageResponse[]> {var _Url = 'api/training/GetImages?'+ 'projectId=' + paramProject.id+ '&tagId=' + paramTag+ '&pageNumber=' + paramPageNumber;// Call the client side codereturn this._http.get(_Url).map((response: Response) =><ICustomVisionImageResponse[]>response.json()).catch(this.handleError);}
This calls the following server side method:
// api/training/GetImages[HttpGet("[action]")]#region public IActionResult GetImages(string projectId, string tagId, string pageNumber)public IActionResult GetImages(string projectId, string tagId, string pageNumber){List<CustomVisionImageResponse> colCustomVisionImageResponse =new List<CustomVisionImageResponse>();if ((tagId != null) && (projectId != null)){// Create a HttpClient to make the requestusing (HttpClient client = new HttpClient()){// Set Training Key in the request headersclient.DefaultRequestHeaders.Add("Training-key", _TrainingKey);// Build the requeststring SkipNumber = ((Convert.ToInt32(pageNumber) - 1) * 8).ToString();StringBuilder uri = new StringBuilder();uri.Append("https://southcentralus.api.cognitive.microsoft.com");uri.Append("/customvision/v2.0/Training");uri.Append("/projects/" + projectId);uri.Append("/images/tagged?");uri.Append("tagIds=" + tagId);uri.Append("&take=" + "8");uri.Append("&skip=" + SkipNumber);// Make the request to the Custom Vision APIHttpResponseMessage response = client.GetAsync(uri.ToString()).Result;// Get the responsestring ResponseContent = response.Content.ReadAsStringAsync().Result;// Convert the response to the CustomVisionImageResponse objectcolCustomVisionImageResponse =JsonConvert.DeserializeObject<List<CustomVisionImageResponse>>(ResponseContent);}}// Return the Response to the Angular applicationreturn Ok(colCustomVisionImageResponse);}#endregion
Uploading
When a Project and a Tag are selected, you can upload images that will then be associated with the Tag.
The following is the markup for the PrimeNG upload control:
< p-fileUpload name="myfile[]"url="api/training"multiple="true"accept="image/*"maxFileSize="10000000"uploadLabel="Upload New Files"(onBeforeUpload)="onBeforeUploadHandler($event)"(onUpload)="onUpload($event)"></ p-fileUpload>
This has two associated methods in the code file, onBeforeUploadHandler to pass the selected Project and Tag to the server side code, and onUpload, that is called after the upload, to refresh the images in the dataGrid:
// Uploadpublic onBeforeUploadHandler(event) {// called before the file(s) are uploaded// Send the currently selected Project and Tag in the Headerevent.formData.append("selectedProject", this.selectedProject.id);event.formData.append("selectedTag", this.selectedTag.id);}public onUpload(event) {// This is called after the uploadthis.getImages();}
The following server side method handles the upload:
// api/training[HttpPost]#region public IActionResult Index()public IActionResult Index(){if (!Request.HasFormContentType){return BadRequest();}// Retrieve data from Formvar form = Request.Form;// Retrieve Project and Tagstring selectedProject = form["selectedProject"].First();string selectedTag = form["selectedTag"].First();if ((selectedProject != null) && (selectedTag != null)){// Process all Filesforeach (var file in form.Files){// Process fileusing (var readStream = file.OpenReadStream()){// Create a HttpClient to make the requestusing (HttpClient client = new HttpClient()){// Set Training Key in the request headersclient.DefaultRequestHeaders.Add("Training-key", _TrainingKey);// Serialize RequestMultipartFormDataContent _multiPartContent =new MultipartFormDataContent();StreamContent _imageData =new StreamContent(readStream);_imageData.Headers.ContentType =new MediaTypeHeaderValue("application/octet-stream");ContentDispositionHeaderValue _contentDispositionHeaderValue =new ContentDispositionHeaderValue("form-data");_contentDispositionHeaderValue.Name = "imageData";_contentDispositionHeaderValue.FileName = file.Name;_imageData.Headers.ContentDisposition = _contentDispositionHeaderValue;_multiPartContent.Add(_imageData, "imageData");// Build the request to the Custom Vision APIStringBuilder uri = new StringBuilder();uri.Append("https://southcentralus.api.cognitive.microsoft.com");uri.Append("/customvision/v2.0/Training");uri.Append("/projects/" + selectedProject);uri.Append("/images?");uri.Append("tagIds=" + selectedTag);// Make the request to the Custom Vision Service APIHttpResponseMessage response =client.PostAsync(uri.ToString(), _multiPartContent).Result;// Get the response -- (Do nothing with it for now)string ResponseContent = response.Content.ReadAsStringAsync().Result;}}}}return Ok();}#endregion
Train Project
After uploading new images, you need to Train the project to create a new predictive model.
The following is the markup for the button:
< button pButton type="button"(click)="TrainProject()"label="Train Project"icon="fa-plus-square"class="ui-button-success" > </ button >
This calls the following method:
TrainProject() {// Call the service -- to train the projectthis._CustomVisionTrainingService.trainProject(this.selectedProject).subscribe((response: ICustomVisionTrainResponse) => {alert(response.status);});}
This calls the following service method:
// ** Train Project **trainProject(paramProject: ICustomVisionProjectResponse):Observable<ICustomVisionTrainResponse> {var _Url = 'api/training/TrainProject?projectId=' + paramProject.id;// Call the client side codereturn this._http.get(_Url).map((response: Response) =><ICustomVisionTrainResponse>response.json()).catch(this.handleError);}
This calls the following server side code:
// api/training/TrainProject[HttpGet("[action]")]#region public IActionResult TrainProject(string projectId)public IActionResult TrainProject(string projectId){CustomVisionTrainResponse objCustomVisionTrainResponse = new CustomVisionTrainResponse();// Create a HttpClient to make the requestusing (HttpClient client = new HttpClient()){HttpResponseMessage response;// Set Training Key in the request headersclient.DefaultRequestHeaders.Add("Training-key", _TrainingKey);// Build the request to the Custom Vision APIStringBuilder uri = new StringBuilder();uri.Append("https://southcentralus.api.cognitive.microsoft.com");uri.Append("/customvision/v2.0/Training");uri.Append("/projects/" + projectId);uri.Append("/train?");// Request bodybyte[] byteData = Encoding.UTF8.GetBytes("{body}");// Make the request to the Custom Vision Service APIusing (var content = new ByteArrayContent(byteData)){content.Headers.ContentType = new MediaTypeHeaderValue("application/json");response = client.PostAsync(uri.ToString(), content).Result;}if (response.StatusCode == System.Net.HttpStatusCode.BadRequest){objCustomVisionTrainResponse.Status = "No Training needed";}else{// Get the responsestring ResponseContent = response.Content.ReadAsStringAsync().Result;// Convert the response to the CustomVisionTrainResponse objectobjCustomVisionTrainResponse =JsonConvert.DeserializeObject<CustomVisionTrainResponse>(ResponseContent);}}// Return a Response to the Angular applicationreturn Ok(objCustomVisionTrainResponse);}#endregion
Links
Custom Vision Service API Reference
Download
You can download the code from the Download page