November 3, 2020 – ACE IT SKILLS

Google Cloud Platform Tutorial

Posted on November 3, 2020November 18, 2020 by thanhnguyen118

What is Cloud Computing?

Cloud computing is defined as the services offered through remote servers on the internet. These services might include database storage, applications, compute power and other IT resources over the pay-as-you-go pricing approach. The remote server allows users to save, modify, or process data on the internet or cloud-based platform instead of storing it on a local server or their devices.

Cloud computing is evolving due to fast performance, better manageability, and less maintenance. It helps organizations to minimize the number of resources and overall infrastructure costs. Additionally, it helps IT teams better focus on the important applications, services, and processes and achieve the company’s goals.

Typically, the cloud-computing providers offer their services according to the following three standard models:

Điện toán đám mây được định nghĩa là các dịch vụ được cung cấp thông qua các máy chủ từ xa trên internet. Các dịch vụ này có thể bao gồm lưu trữ cơ sở dữ liệu, ứng dụng, sức mạnh máy tính và các tài nguyên CNTT khác theo cách tiếp cận định giá theo phương thức trả tiền. Máy chủ từ xa cho phép người dùng lưu, sửa đổi hoặc xử lý dữ liệu trên internet hoặc nền tảng dựa trên đám mây thay vì lưu trữ trên máy chủ cục bộ hoặc thiết bị của họ. Điện toán đám mây đang phát triển nhờ hiệu suất nhanh, khả năng quản lý tốt hơn và ít bảo trì hơn. Nó giúp các tổ chức giảm thiểu số lượng tài nguyên và chi phí cơ sở hạ tầng tổng thể. Ngoài ra, nó giúp các nhóm CNTT tập trung tốt hơn vào các ứng dụng, dịch vụ và quy trình quan trọng và đạt được các mục tiêu của công ty. Thông thường, các nhà cung cấp dịch vụ điện toán đám mây cung cấp dịch vụ của họ theo ba mô hình tiêu chuẩn sau:

What is Google Cloud Platform?

Google Cloud Platform (GCP) is a suite of cloud computing services provided by Google. It is a public cloud computing platform consisting of a variety of services like compute, storage, networking, application development, Big Data, and more, which run on the same cloud infrastructure that Google uses internally for its end-user products, such as Google Search, Photos, Gmail and YouTube, etc.

The services of GCP can be accessed by software developers, cloud administrators and IT professionals over the Internet or through a dedicated network connection.

Why Google Cloud Platform?

Google Cloud Platform is known as one of the leading cloud providers in the IT field. The services and features can be easily accessed and used by the software developers and users with little technical knowledge. Google has been on top amongst its competitors, offering the highly scalable and most reliable platform for building, testing and deploying the applications in the real-time environment.

Apart from this, GCP was announced as the leading cloud platform in the Gartner’s IaaS Magic Quadrant in 2018. Gartner is one of the leading research and advisory company. Gartner organized a campaign where Google Cloud Platform was compared with other cloud providers, and GCP was selected as one of the top three providers in the market.

Most companies use data centers because of the availability of cost forecasting, hardware certainty, and advanced control. However, they lack the necessary features to run and maintain resources in the data center. GCP, on the other side, is a fully-featured cloud platform that includes:

Capacity: Sufficient resources for easy scaling whenever required. Also, effective management of those resources for optimum performance.
Security: Multi-level security options to protect resources, such as assets, network and OS -components.
Network Infrastructure: Number of physical, logistical, and human-resource-related components, such as wiring, routers, switches, firewalls, load balancers, etc.
Support: Skilled professionals for installation, maintenance, and support.
Bandwidth: Suitable amount of bandwidth for peak load.
Facilities: Other infrastructure components, including physical equipment and power resources.

Therefore, Google Cloud Platform is a viable option for businesses, especially when the businesses require an extensive catalog of services with global recognition.

Benefits of Google Cloud Platform

Some of the main benefits of Google Cloud Platform are explained below:

Best Pricing: Google enables users to get Google Cloud hosting at the cheapest rates. The hosting plans are not only cheaper than other hosting platforms but also offer better features than others. GCP provides a pay-as-you-go option to the users where users can pay separately only for the services and resources they want to use.

Work from Anywhere: Once the account is configured on GCP, it can be accessed from anywhere. That means that the user can use GCP across different devices from different places. It is possible because Google provides web-based applications that allow users to have complete access to GCP.

Private Network: Google has its own network that enables users to have more control over GCP functions. Due to this, users achieve smooth performance and increased efficiency over the network.

Scalable: Users are getting a more scalable platform over the private network. Because Google uses fiber-optic cables to extend its network range, it is likely to have more scalability. Google is always working to scale its network because there can be any amount of traffic at any time.

Security: There is a high number of security professionals working at Google. They always keep trying to secure the network and protect the data stored on servers. Additionally, Google uses an algorithm that encrypts all the data on the Cloud platform. This gives assurance to the users that their data is completely safe and secure from unauthorized sources.

Redundant Backup: Google always keeps backup of user’s data with built-in redundant backup integration. In case a user has lost the stored data, it’s not a big problem. Google always has a copy of the users’ data unless the data is deleted forcefully. This adds data integrity, reliability and durability with GCP.

Key Features of Google Cloud Platform

The following are some key features of Google Cloud Platform:

On-demand services: Automated environment with web-based tools. Therefore, no human intervention is required to access the resources.
Broad network access: The resources and the information can be accessed from anywhere.
Resource pooling: On-demand availability of a shared pool of computing resources to the users.
Rapid elasticity: The availability of more resources whenever required.
Measured service: Easy-to-pay feature enables users to pay only for consumed services.

Working of Google Cloud Platform

When a file is uploaded on the Google cloud, the unique metadata is inserted into a file. It helps identify the different files and track the changes made across all the copies of any particular file. All the changes made by individuals get synchronized automatically to the main file, also called a master file. GCP further updates all the downloaded files using metadata to maintain the correct records.

Let’s understand the working of GCP with a general example:

Suppose that MS Office is implemented on Cloud to enable several people to work together. The primary aim of using cloud technology is to work on the same project at the same time. We can create and save a file on the cloud once we install a plugin for the MS Office suite. This will allow several people to edit a document at the same time. The owner can assign access to specific people to allow them to download and start editing the document in MS Office.

Once users are assigned as an editor, they can use and edit the document’s cloud copy as desired. The combined, edited copy is generated that is known as the master document. GCP helps to assign a unique URL to each specific copy of the existing document given to different users. However, any of the authorized users’ changes will be visible on all the copies of documents shared over the cloud. In case multiple changes are made to the same document, then GCP allows the owner to select the appropriate changes to keep.

Google Cloud Platform Services

Google provides a considerable number of services with several unique features. That is the reason why Google Cloud Platform is continually expanding across the globe. Some of the significant services of GCP are:

Compute Services
Networking
Storage Services
Big Data
Security and Identity Management
Management Tools
Cloud AI
IoT (Internet of Things)

Let’s understand each of these services in details:

Compute Services

GCP offers a scalable range of computing services, such as:

Google App Engine: It is a cloud computing platform that follows the concept of Platform-as-a-Service to deploy PHP, Java and other software. It is also used to develop and deploy web-based software in Google-managed data centers. The most significant advantage of Google App Engine is its automatic scaling capability. This means that the App Engine automatically allocates more resources for the application when there is an increase in requests.
Compute Engine: It is a cloud computing platform that follows the concept of Infrastructure-as-a-Service to run Windows and Linux based virtual machines. It is an essential component of GCP. It is designed on the same infrastructure used by Google search engine, YouTube and other Google services.
Kubernetes Engines: This computing service is responsible for offering a platform for automatic deployment, scaling, and other operations of application containers across clusters of hosts. The engine supports several container tools like a docker, etc.

Networking

GCP includes the following network services:

VPC: VPC stands for Virtual Private Network. The primary function of VPC is to offer a private network with routing, IP allocation, and network firewall policies. This will help to create a secure environment for the application deployments.
Cloud Load Balancing: As its name states, Cloud balancing is used to distribute workload across different computing resources to balance the entire system performance. This also results in cost-reduction. The process also helps in minimizing the availability and maximizing the capability of the resources.
Content Delivery Network: CDN is a geographically distributed network of proxy servers and their data centers. The primary aim of using CDN is to provide maximum performance to the users. Additionally, it also helps deliver high availability of resources by equally distributing the related services to the end-users.

Storage Services

GCP has the following storage services:

Google Cloud Storage: It is an online data storage web service that Google provides to its users to store and access data from anywhere. The service also includes a wide range of features like maximum performance, scalability, security and sharing.
Cloud SQL: It is a web-service that enables users to create, manage, and use relational databases stored on Google Cloud servers. The service itself maintains and protects the databases, which helps users focus on their applications and other operations.
Cloud Bigtable: It is known for its fast performance and highly manageable feature. It is a highly scalable NoSQL database service that allows collecting and retaining data from as low as 1 TB to hundreds of PB.

Big Data

GCP provides a variety of services related to big data; they are:

BigQuery: It is a fully managed data analysis service by Google. The primary aim of Google BigQuery service is to helps businesses to analyze Big Data. It offers a highly scalable data management option. This means BigQuery allows users to perform ad-hoc queries and share data insights across the web.
Google Cloud Datastore: Google Cloud Datastore is a kind of datastore service that is fully managed, schema-less, and non-relational. This service enables businesses to perform automatic transactions and a rich set of queries. The main advantage of Google Cloud Datastore is the capability of automatic scaling. This means that the service can itself scale up and down, depending on the requirement of resources.
Google Cloud Dataproc: It is a very fast and easy to use big data service offered by Google. It mainly helps in managing Hadoop and Spark services for distributed data processing. The service allows users to create Hadoop or Spark clusters sized according to the overall workload and can be accessed whenever users want them.

Security and Identity Management

GCP includes the following services related to Security and Identity management:

Cloud Data Loss Prevention API: It is mainly designed to manage sensitive data. It helps users manage sensitive data elements like credit card details, debit card details, passport numbers, etc. It offers fast and scalable classification for sensitive data.
Cloud IAM: It stands for Cloud Identity and Access Management. It is a framework that contains rules and policies and validates the authentication of the users for accessing the technology resources. That is why it is also known as Identity Management (IdM).

Management Tools

GCP includes the following services related to management tools:

Google Stackdriver: Google Stackdriver service is primarily responsible for displaying the overall performance and diagnostics information. This may include insights of data monitoring, tracing, logging, error reporting, etc. The service also prompts an alert notification to the public cloud users.
Google Cloud Console App: It is a native mobile application powered by Google. The primary aim of this service is to enable users to manage the core features of Google Cloud services directly from their mobile devices anytime, anywhere. The primary functions of this service are alerting, monitoring, and performing critical actions on resources.

Cloud AI

When it comes to Cloud AI, GCP offers these services:

Cloud Machine Learning Engine: It is another fully managed service that allows users to create Machine Learning models. The service is mainly used for those ML models, which are based on mainstream frameworks.
Cloud AutoML: It is the type of service that is based on Machine Learning. It helps users to enter their data sets and gain access to quality trained pre-designed ML models. The service works by following Google’s transfer learning and Neural Architecture Search method.

IoT (Internet of Things)

GCP contains the following IoT services:

Cloud IoT Core: It is one of the fully managed core services. It allows users to connect, control, and ingest data from various devices that are securely connected to the Internet. This allows other Google cloud services to analyze, process, collect and visualize IoT data in real-time.

Cloud IoT Edge: The Edge computing service brings memory and other computing-power resources near to the location where it is required.

Advantages of Google Cloud Platform

There are several advantages of using Google Cloud Platform, such as:

Google Cloud Offers Quick and Easy Collaboration: Multiple users can access the data and simultaneously contribute their information. This is possible because the data is stored on the cloud servers, not on the user’s personal computers.
Higher Productivity with Continuous Development: Google is always working on adding new features and functionalities to provide higher productivity to the customers. Therefore, Google delivers frequent updates to its products and services.
Less Disruption with Adopting New Features: Instead of pushing huge disruptive updates of changes, Google provides small updates weekly. This helps users to understand and adopt new features easily.
Least or Minimal Data is stored on Vulnerable Devices: Google does not store data on local devices unless a user explicitly tries to do it. This is because the data stored on local devices may get compromised compared to the cloud’s data.
Users can access Google Cloud from Anywhere: The best thing is that a user can easily access the information stored on Google cloud from anywhere because it is operated through web-based applications.
Google provides Maximum Security with its Robust Structure: Google hires leading security professionals to protect user’s data. Users get process-based and physical security features made by Google.
Users have Full Control over their Data: Users gain full control over services and the data stored in Google Cloud. If a user does not want to use Google services any longer and wants to delete the cloud data, it can be easily performed.
Google provides Higher Uptime and Reliability: Google uses several resources to provide higher and reliable up-time servers. If a data center is not working for technical issues, the system will automatically communicate with the secondary center without interruption visible to users.

Creating a Free Tier Account on GCP

To start using Google Cloud Platform, we are first required to create an account GCP. Here, we will create a free tier account for explaining the upcoming topic of this tutorial. The best thing about free account is that Google provides $300 worth credit to spend over the next 90 days after the date of account creation. Google offers all the core services of GCP with a free account for the next 90 days.

However, users must have a credit card to start a free tier account. Google asks for the credit card details to make sure that it is a genuine human request. Google does not charge automatically even after the 90 days or when we have exhausted the $300 free credit. The amount will only be charged when we will be upgrading our free account to a paid account manually.

Let’s start with the steps of creating a free tier account on Google Cloud Platform:

Step 1: First, we are required to navigate to the following link: https://cloud.google.com/gcp/

Step 2: On the next screen, we need to click on ‘Get started for free’, as shown below:

Step 3: Next, we are required to login to the Google Account. We can use the ‘create an account’ button if we don’t have an existing Google account.

Step 4: Once we have logged in, we will get to the following screen:

Here, we must select the Country, agree to the Terms of Service, and then click on the ‘CONTINUE’ button.

Step 5: On the next screen, we have to enter some necessary details such as name and address details. Also, we have to enter payment details like the method of payments and credit card details. After filling all the details, we need to click on the button ‘START MY FREE TRIAL’ from the bottom of the page:

Step 6: Google asks for the confirmation to use the credit card for the small deduction to ensure that the card information is correct. However, the amount is refunded back to the same account. Here, we need to click on the ‘CONTINUE’ button:

Step 7: On the next screen, we must click on the ‘GO TO CONSOLE’ button:

After clicking on the ‘GO TO CONSOLE’ button, we will be redirected to the Dashboard that includes a summary of GCP services along with projects and other insights. It looks like this:

To be specific, a Dashboards of GCP displays the summarized view of the followings:

Project Info: contains project details such as project name, ID, and number.
Resources: contains a list of resources being used in the related project.
APIs: contains various API requests running with the project (in request/sec form).
Google Cloud Platform Status: displays an overall summary of services that are part of GCP.
Monitoring: displays alerts, performance stats, Uptime, etc. to ensure that systems are running reliably.
Error Reporting: displays errors occurring in the projects, but it needs to be configured first.
Trace: displays latency data of existing applications across a distributed tracing system.
Compute Engine: displays the insights of CPU usage in percentage (%).
Tutorials: contains Getting Started guides (basic guides) to explain how the GCP features work.
News: displays news and other important updates regarding Google Cloud Platform.
Documentation: contains in-depth guides to teach more about Compute Engine, Cloud Storage, and App Engine.

Google Cloud Platform Pricing

When it comes to pricing, Google Cloud Platform is the cheapest solution in the market. GCP is not only low on price but also offers more features and services than other providers.

When comparing GCP with other leading competitors, it has more benefits over them. Google provides its users a massive 60% savings, including:

15% rightsizing recommendation
21% list price differences
24% of sustained usage discounts

Some of the main benefits of GCP pricing are:

No Hidden Charges: There are no hidden charges behind the GCP pricing. Google’s pricing structure is straightforward and can be easily understood.

Pay-as-you-go: Google offers its customer ‘use now, pay later’ option. So, users will have to pay only for those services which they want to use or already using.

No Termination Fee: Users are free to stop using Google services whenever they want, and there will not have to pay any termination fee. That means the moment users stop using Google services; they stop paying for it.

Difference between Google Cloud Platform, AWS and Azure

Like Google cloud platform, AWS and Azure are the other popular cloud-based platforms. However, there are differences amongst them. Some of the main differences between GCP, AWS and Azure are tabulated below:

Google Cloud	AWS	Azure
It uses GCE (Google Compute Engine) for computing purposes.	AWS EC2 offers core compute services.	It uses virtual machines for computation purposes.
It uses Google Cloud Storage for storage purposes.	It uses Amazon S3 for storing the data.	It uses a storage block bob that comprises blocks for storing the data.
It offers the lowest price to the customers to beat other cloud providers.	AWS pricing is generally keen to have inscrutable. The overall structure of granular pricing is a bit complex.	Like AWS, Azure pricing structure is also difficult to understand unless you have considerable experience.
It uses Cloud Test labs for App Testing purposes.	It uses a device farm for App Testing purposes.	It uses DevTest labs for App Testing purposes.
It uses Subnet as a virtual network.	It uses VPC as a virtual network.	It uses VNet as a virtual Network.
It follows the Cloud Load Balancing configuration.	It follows the Elastic Load Balancing configuration.	It follows the Load-Balancer Application Gateway configuration.

Job Opportunities with GCP

Having a piece of in-depth knowledge in the Google Cloud Platform is very useful as per job purposes. However, an experience or little more expertise in using GCP will help a person stand aside from the crowd. This will not only make a resume more effective but will also open a variety of job opportunities.

There are many job-opportunities with GCP. Some popular job-roles are listed below:

Technical Lead Manager
Sales Engineer
Technical Solutions Engineer
Account Executive
Technical Program Manager
Cloud Software Engineer
Data Center Software Engineer
Solutions Architect
Strategic Customer Engineer

Prerequisite

There is no special prerequisite for this GCP Tutorial. All you need is continuous learning and practicing with the tools. However, if you want to extend functionalities to match your requirements, then a basic knowledge of working with cloud-based software and tools will be beneficial and put you at an advantage. We have designed this tutorial to help you learn all the concepts of Google Cloud Platform from scratch.

Audience

Our Google Cloud Platform Tutorial is designed to help beginners and professionals.

Problem

We assure you that you will not find any difficulty while learning through our Google Cloud Platform Tutorial. But if you find any mistake in this tutorial, we request you to kindly post the problem in the contact form so that we can improve it.

Machine Learning

Single Layer neural network-perceptron model on the IRIS dataset using Heaviside step activation Function

Posted on November 3, 2020November 18, 2020 by thanhnguyen118

In this tutorial, we won’t use scikit. Instead we’ll approach classification via historical Perceptron learning algorithm based on “Python Machine Learning by Sebastian Raschka, 2015”.

We’ll extract two features of two flowers form Iris data sets. Then, we’ll updates weights using the difference between predicted and target values.

Although the Perceptron classified the two Iris flower classes perfectly, convergence is one of the biggest problems of the perceptron.

Code : Perceptron learning algorithm

The following code defines perceptron interface as a Python Class:

# perceptron.py
import numpy as np

class Perceptron(object):
   def __init__(self, rate = 0.01, niter = 10):
      self.rate = rate
      self.niter = niter

   def fit(self, X, y):
      """Fit training data
      X : Training vectors, X.shape : [#samples, #features]
      y : Target values, y.shape : [#samples]
      """

      # weights
      self.weight = np.zeros(1 + X.shape[1])

      # Number of misclassifications
      self.errors = []  # Number of misclassifications

      for i in range(self.niter):
         err = 0
         for xi, target in zip(X, y):
            delta_w = self.rate * (target - self.predict(xi))
            self.weight[1:] += delta_w * xi
            self.weight[0] += delta_w
            err += int(delta_w != 0.0)
         self.errors.append(err)
      return self

   def net_input(self, X):
      """Calculate net input"""
      return np.dot(X, self.weight[1:]) + self.weight[0]

   def predict(self, X):
      """Return class label after unit step"""
      return np.where(self.net_input(X) >= 0.0, 1, -1)

bogotobogo.com site search:
Preprocessing Iris data set

To test our perceptron implementation, we will load the two flower classes Setosa and Versicolor from the Iris data set. The perceptron rule is not restricted to two dimensions, however, we will only consider the two features sepal length and petal length for visualization purposes.

Now, we will use the pandas library to load the Iris data set into a DataFrame object:

>>> import pandas as pd
>>> df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)
>>> 
>>> df.tail()
       0    1    2    3               4
145  6.7  3.0  5.2  2.3  Iris-virginica
146  6.3  2.5  5.0  1.9  Iris-virginica
147  6.5  3.0  5.2  2.0  Iris-virginica
148  6.2  3.4  5.4  2.3  Iris-virginica
149  5.9  3.0  5.1  1.8  Iris-virginica
>>> 
>>> df.iloc[145:150, 0:5]
       0    1    2    3               4
145  6.7  3.0  5.2  2.3  Iris-virginica
146  6.3  2.5  5.0  1.9  Iris-virginica
147  6.5  3.0  5.2  2.0  Iris-virginica
148  6.2  3.4  5.4  2.3  Iris-virginica
149  5.9  3.0  5.1  1.8  Iris-virginica

Next, we extract the first 100 class labels that correspond to the 50 Iris-Setosa and 50 Iris-Versicolor flowers, respectively:

>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> 
>>> y = df.iloc[0:100, 4].values
>>> y
array(['Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',
       'Iris-setosa', 'Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor',
       'Iris-versicolor', 'Iris-versicolor', 'Iris-versicolor'], dtype=object)
>>>

The we want to convert the class labels into the two integer class labels 1 (Versicolor) and -1 (Setosa) that we assign to a vector y where the values method of a pandas DataFrame yields the corresponding NumPy representation.

>>> y = np.where(y == 'Iris-setosa', -1, 1)
>>> y
array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1])

Also, we need to extract the first feature column (sepal length) and the third feature column (petal length) of those 100 training samples and assign them to a feature matrix X:

>>> X = df.iloc[0:100, [0, 2]].values
>>> X
array([[ 5.1,  1.4],
       [ 4.9,  1.4],
           ...
       [ 5.1,  3. ],
       [ 5.7,  4.1]])
>>>

We can visualize via a two-dimensional scatter plot using the matplotlib:

>>> plt.scatter(X[:50, 0], X[:50, 1], color='red', marker='o', label='setosa')
>>> plt.scatter(X[50:100, 0], X[50:100, 1], color='blue', marker='x', label='versicolor')
>>> plt.xlabel('petal length')
>>> plt.ylabel('sepal length')
>>> plt.legend(loc='upper left')
>>> plt.show()

Training the perceptron model

Picture from “Python Machine Learning by Sebastian Raschka, 2015”

Now we can train our perceptron algorithm on the Iris data subset that we extracted in the previous section.

We will plot the misclassification error for each epoch to check if the algorithm converged and found a decision boundary that separates the two Iris flower classes:

>>> # import Perceptron from perceptron.py
>>> from perceptron import Perceptron
>>> pn = Perceptron(0.1, 10)
>>> pn.fit(X, y)
>>> plt.plot(range(1, len(pn.errors) + 1), pn.errors, marker='o')
>>> plt.xlabel('Epochs')
>>> plt.ylabel('Number of misclassifications')
>>> plt.show()

We can see the plot of the misclassification errors versus the number of epochs as shown below:

Our perceptron converged after the sixth epoch (iteration). Now we’re able to classify the training samples perfectly.

Visualize the decision boundaries

To visualize the decision boundaries for our 2D datasets, let’s implement a small convenience function:

from matplotlib.colors import ListedColormap
def plot_decision_regions(X, y, classifier, resolution=0.02):
   # setup marker generator and color map
   markers = ('s', 'x', 'o', '^', 'v')
   colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
   cmap = ListedColormap(colors[:len(np.unique(y))])

   # plot the decision surface
   x1_min, x1_max = X[:,  0].min() - 1, X[:, 0].max() + 1
   x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
   xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
   np.arange(x2_min, x2_max, resolution))
   Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
   Z = Z.reshape(xx1.shape)
   plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
   plt.xlim(xx1.min(), xx1.max())
   plt.ylim(xx2.min(), xx2.max())

   # plot class samples
   for idx, cl in enumerate(np.unique(y)):
      plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
      alpha=0.8, c=cmap(idx),
      marker=markers[idx], label=cl)

In the code above, we define a number of colors and markers and create a color map from the list of colors via ListedColormap.

Then, we determine the minimum and maximum values for the two features and use those feature vectors to create a pair of grid arrays xx1 and xx2 via the NumPy meshgrid function.

Since we trained our perceptron classifier on two feature dimensions, we need to flatten the grid arrays and create a matrix that has the same number of columns as the Iris training subset so that we can use the predict method to predict the class labels Z of the corresponding grid points.

After reshaping the predicted class labels Z into a grid with the same dimensions as xx1 and xx2 , we can now draw a contour plot via matplotlib’s contourf function that maps the different decision regions to different colors for each predicted class in the grid array:

>>> plot_decision_regions(X, y, classifier=pn)
>>> plt.xlabel('sepal length [cm]')
>>> plt.ylabel('petal length [cm]')
>>> plt.legend(loc='upper left')
>>> plt.show()

As shown in the following figure, we can now see a plot of the decision regions.

The perceptron learned a decision boundary that was able to classify all flower samples in the Iris training subset perfectly.

Although the perceptron classified the two Iris flower classes perfectly, convergence is one of the biggest problems of the perceptron.
Frank Rosenblatt proved mathematically that the perceptron learning rule converges if the two classes can be separated by a linear hyperplane.
However, if classes cannot be separated perfectly by such a linear decision boundary, the weights will never stop updating unless we set a maximum number of epochs.
From “Python Machine Learning by Sebastian Raschka, 2015”.

Robotics

What is artificial neural network(ANN) and and how to applied it in programming

Posted on November 3, 2020December 19, 2020 by thanhnguyen118

The Neural network structure is: one, input layer, one output layer, and layers between are hidden layer.

Introduction

An Artificial Neural Network (ANN) is an interconnected group of nodes, similar to the our brain network.

Here, we have three layers, and each circular node represents a neuron and a line represents a connection from the output of one neuron to the input of another.

The first layer has input neurons which send data via synapses to the second layer of neurons, and then via more synapses to the third layer of output neurons.

Jupyter Notebook

This tutorial uses IPython’s Jupyter notebook.

If you don’t have it, please visit iPython and Jupyter – Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github.

This tutorial consists of 7 parts and each of it has its own notebook. Please check out Jupyter notebook files at Github.

Reference

We’ll follow Neural Networks Demystified throughout this series of articles.

Problem

Suppose we want to predict our test score based on how many hours we sleep and how many hours we study the night before.

In other words, we want to predict output value yy which are scores for a given set of input values XX which are hours of (sleep, study).

XX (sleep, study)	y (test score)
(3,5)	75
(5,1)	82
(10,2)	93
(8,3)	?

In our machine learning approach, we’ll use the python to store our data in 2-dimensional numpy arrays.

We’ll use the data to train a model to predict how we will do on our next test.

This is a supervised regression problem.

It’s supervised because our examples have outputs(yy).

It’s a regression because we’re predicting the test score, which is a continuous output.

If we we’re predicting the grade (A,B, etc.), however, this is going to be a classification problem but not a regression problem.

We may want to scale our data so that the result should be in [0,1].

Now we can start building our Neural Network.

We know our network must have 2 inputs(XX) and 1 output(yy).

We’ll call our output y^y^, because it’s an estimate of yy.

Neural-Network-3-Layers-Red-Green-Hours-Sleep-Study-y-hat-Cleared.png

Any layer between our input and output layer is called a hidden layer. Here, we’re going to use just one hidden layer with 3 neurons.

Neurons synapses

As explained in the earlier section, circles represent neurons and lines represent synapses.

Synapses have a really simple job.

They take a value from their input, multiply it by a specific weight, and output the result. In other words, the synapses store parameters called “weights” which are used to manipulate the data.

Neurons are a little more complicated.

Neurons’ job is to add together the outputs of all their synapses, and then apply an activation function.

Certain activation functions allow neural nets to model complex non-linear patterns.

For our neural network, we’re going to use sigmoid activation functions.

Home

DOCKER NGINX WEB SERVER – FILE SHARING/COPYING & DOCKERFILE

Posted on November 3, 2020November 18, 2020 by thanhnguyen118

Note

This will be a very brief tutorial on Docker: we’ll take a “nginx” docker image and build a simple web server on Docker container. In that process, we can get a quick taste of how Docker is working.

We’ll learn:

How to get an official image.
How to use the files on host machine from our container.
How to copy the files from our host to the container.
How to make our own image – in this tutorial, we’ll make a Dockerfile with a page for render.

Getting Nginx official docker image

To create an instance of Nginx in a Docker container, we need to search for and pull the Nginx official image from Docker Hub. Use the following command to launch an instance of Nginx running in a container and using the default configuration:

$ docker container run --name my-nginx-1 -P -d nginx
f0cea39a8bc38b38613a3a4abe948d0d1b055eefe86d026a56c783cfe0141610

The command creates a container named “my-nginx-1” based on the Nginx image and runs it in “detached” mode, meaning the container is started and stays running until stopped but does not listen to the command line. We will talk about this later how to interact with the container.

The Nginx image exposes ports 80 and 443 in the container and the -P option tells Docker to map those ports to ports on the Docker host that are randomly selected from the range between 49153 and 65535.

We do this because if we create multiple Nginx containers on the same Docker host, we may induce conflicts on ports 80 and 443. The port mappings are dynamic and are set each time the container is started or restarted.

If we want the port mappings to be static, set them manually with the -p option. The long form of the “Container Id” will be returned.

We can run docker ps to verify that the container was created and is running, and to see the port mappings:

$ docker container ps
CONTAINER ID IMAGE COMMAND                 CREATED       STATUS       PORTS                                        NAMES
f0cea39a8bc3 nginx "nginx -g 'daemon off"  5 minutes ago Up 5 minutes 0.0.0.0:32769->80/tcp   my-nginx-1

We can verify that Nginx is running by making an HTTP request to port 32769 (reported in the output from the preceding command as the port on the Docker host that is mapped to port 80 in the container), the default Nginx welcome page appears:

$ curl http://localhost:32769
<!DOCTYPE html>
<html>
<head>
...
</html>

We can stop the container and remove:

$ docker container stop f0cea39a8bc3
f0cea39a8bc3

$ docker container ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ docker container rm f0cea39a8bc3

If we want to map the container’s port 80 to a specific port of the host, we can use -p with host_port:docker_port:

$ docker container run --name my-nginx-1 -p8088:80 -d nginx
bd39cc4ed6dbba92e85437b0ab595e92c5ca2e4a87ee3e724a0cf04aa9726044

$ docker container ps
CONTAINER ID  IMAGE  COMMAND                 CREATED         STATUS         PORTS                  NAMES
bd39cc4ed6db  nginx  "nginx -g 'daemon of…"  18 seconds ago  Up 18 seconds  0.0.0.0:8088->80/tcp   my-nginx-1

More work on the Nginx Docker Container

Now that we have a working Nginx Docker container, how do we manage the content and the configuration? What about logging?

It is common to have SSH access to Nginx instances, but Docker containers are generally intended to be for a single purpose (in this case running Nginx) so the Nginx image does not have OpenSSH installed and for normal operations there is no need to get shell access directly to the Nginx container. We will use other methods supported by Docker rather than using SSH.

Keep the Content and Configuration on the Docker Host

When the container is created, we can tell Docker to mount a local directory on the Docker host to a directory in the container.

The Nginx image uses the default Nginx configuration, so the root directory for the container is /usr/share/nginx/html. If the content on the Docker host is in the local directory /tmp/nginx/html, we run the command:

$ docker container run --name my-nginx-2 \
-v /tmp/nginx/html:/usr/share/nginx/html:ro -P -d nginx

$ docker container ps
CONTAINER ID  IMAGE  COMMAND                CREATED       STATUS       PORTS                   NAMES
35a4c74ff073  nginx  "nginx -g 'daemon off" 7 seconds ago Up 2 seconds 0.0.0.0:32770->80/tcp   my-nginx-2

Now any change made to the files in the local directory, /tmp/nginx/html on the Docker host are reflected in the directories /usr/share/nginx/html in the container. The :ro option causes these directors to be read only inside the container.

Copying files from the Docker host

Rather than using the files that kept in the host machine, another option is to have Docker copy the content and configuration files from a local directory on the Docker host when a container is created.

Once a container is created, the files are maintained by creating a new container when the files change or by modifying the files in the container.

A simple way to copy the files is to create a Dockerfile to generate a new Docker image, based on the Nginx image from Docker Hub. When copying files in the Dockerfile, the path to the local directory is relative to the build context where the Dockerfile is located. For this example, the content is in the MyFiles directory under the same directory as the Dockerfile.

Here is the Dockerfile:

FROM nginx
COPY MyFiles /usr/share/nginx/html

We can then create our own Nginx image by running the following command from the directory where the Dockerfile is located:

$ docker image build -t my-nginx-image-1 .
Sending build context to Docker daemon  4.096kB
Step 1/2 : FROM nginx
 ---> be1f31be9a87
Step 2/2 : COPY MyFiles /usr/share/nginx/html
 ---> 507c7d77c45f
Successfully built 507c7d77c45f
Successfully tagged my-nginx-image-1:latest

Note the period (“.”) at the end of the command. This tells Docker that the build context is the current directory. The build context contains the Dockerfile and the directories to be copied.

Now we can create a container using the image by running the command:

$ docker container run --name my-nginx-3 -P -d my-nginx-image-1
150fe5eddb31acde7dd432bed73f8daba2a7eec3760eb272cb64bfdd2809152d

$ docker container ps
CONTAINER ID  IMAGE             COMMAND                  CREATED        STATUS        PORTS                   NAMES
150fe5eddb31  my-nginx-image-1  "nginx -g 'daemon of…"   7 seconds ago  Up 7 seconds  0.0.0.0:32772->80/tcp   my-nginx-3

Editing (Sharing) files in Docker container : volumes

Actually, the volume of Docker belongs to an advanced level. So, in this section, we only touch the topic briefly.

As we know, we are not able to get SSH access to the Nginx container, so if we want to edit the container files directly we can use a helper container that has shell access.

In order for the helper container to have access to the files, we must create a new image that has the proper volumes specified for the image.

Assuming we want to copy the files as in the example above, while also specifying volumes for the content and configuration files, we use the following Dockerfile:

FROM nginx
COPY content /usr/share/nginx/html
VOLUME /myVolume-1

We then create the new Nginx image (my-nginx-image-2) by running the following command:

$ docker image build -t my-nginx-image-2 .
Sending build context to Docker daemon  4.096kB
Step 1/3 : FROM nginx
 ---> be1f31be9a87
Step 2/3 : COPY MyFiles /usr/share/nginx/html
 ---> Using cache
 ---> 2bddb692d90d
Step 3/3 : VOLUME /myVolume-1
 ---> Running in 699a41ac23c9
Removing intermediate container 699a41ac23c9
 ---> 1037d1f75691
Successfully built 1037d1f75691
Successfully tagged my-nginx-image-2:latest

Now we create an Nginx container (my-nginx-4) using the image by running the command:

$ docker container run --name my-nginx-4 -P -d my-nginx-image-2
8c84cf08dc08a0b00bd04c9b7047ff65f1ec5c910aa6b096a743b18652e128f9

We then start a helper container with a shell and access the content and configuration directories of the Nginx container we created in the previous example by running the command:

$ docker container run -i -t --volumes-from my-nginx-4 --name my-nginx-4-with-volume debian /bin/bash
root@64052ed5e1b3:/#

This creates an interactive container named my-nginx-4-with-volume that runs in the foreground with a persistent standard input (-i) and a tty (-t) and mounts all the volumes defined in the container my-nginx-4 as local directories in the my-nginx-4-with-volume container.

After the container is created, it runs the bash shell, which presents a shell prompt for the container that we can use to modify the files as needed.

Now, we can check the volume has been mounted:

root@64052ed5e1b3:/# ls           
bin  boot  dev	etc  home  lib	lib64  media  mnt  myVolume-1  opt  proc  root	run  sbin  srv	sys  tmp  usr  var
root@64052ed5e1b3:/#

To check the volume on local machine we can go to /var/lib/docker/volumes. On Mac system, we need to take an additional step to:

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Or for newer system:

screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Home

DOCKER IMAGE AND CONTAINER VIA DOCKER COMMANDS (SEARCH, PULL, RUN, PS, RESTART, ATTACH, AND RM)

Posted on November 3, 2020November 18, 2020 by thanhnguyen118

LinuX Containers (LXC)

The problem with Virtual Machines built using VirtualBox or VMWare is that we have to run entire OS for every VM. That’s where Docker comes in. Docker virtualizes on top of one OS so that we can run Linux using technology known as LinuX Containers (LXC). LXC combines cgroups and namespace support to provide an isolated environment for applications. Docker can also use LXC as one of its execution drivers, enabling image management and providing deployment services.

Docker allows us to run applications inside containers. Running an application inside a container takes a single command: docker run.

Docker Registry – Repositories of Docker Images

We need to have a disk image to make the virtualization work. The disk image represents the system we’re running on and they are the basis of containers.

Docker registry is a registry of already existing images that we can use to run and create containerized applications.

There are lots of communities and works already been done to build the system. Docker company supports and maintains its registry and the community around it.

We can search images within the registry hub, for example, the sample picture is the result from searching “flask”.

Docker search

The Docker search command allows us to go and look at the registry in search for the images that we want.

$ docker search --help

Usage: docker search [OPTIONS] TERM

Search the Docker Hub for images

  --automated=false    Only show automated builds
  --no-trunc=false     Don't truncate output
  -s, --stars=0        Only displays with at least x stars

If we do the same search, Jenkins, we get exactly the same result as we got from the web:

$ docker search ubuntu
NAME                      DESCRIPTION                                    STARS  OFFICIAL   AUTOMATED
ubuntu                    Ubuntu is a Debian-based Linux operating s...  5969     [OK]       
rastasheep/ubuntu-sshd    Dockerized SSH service, built on top of of...  83                   [OK]
ubuntu-upstart            Upstart is an event-based replacement for ...  71       [OK]       
ubuntu-debootstrap        debootstrap --variant=minbase --components...  30       [OK]       
torusware/speedus-ubuntu  Always updated official Ubuntu docker imag...  27                   [OK]
nuagebec/ubuntu           Simple always updated Ubuntu docker images...  20                   [OK]
...

We got too many outputs, so we need to filter it out items with more than 10 stars:

$ docker search --filter=stars=20 ubuntu
NAME                      DESCRIPTION                                    STARS   OFFICIAL   AUTOMATED
ubuntu                    Ubuntu is a Debian-based Linux operating s...  5969      [OK]       
rastasheep/ubuntu-sshd    Dockerized SSH service, built on top of of...  83                   [OK]
ubuntu-upstart            Upstart is an event-based replacement for ...  71        [OK]       
ubuntu-debootstrap        debootstrap --variant=minbase --components...  30        [OK]       
torusware/speedus-ubuntu  Always updated official Ubuntu docker imag...  27                   [OK]
nuagebec/ubuntu           Simple always updated Ubuntu docker images...  20                   [OK]

Docker pull

Once we found the image we like to use it, we can use Docker’s pull command:

$ docker pull --help

Usage:	docker pull [OPTIONS] NAME[:TAG|@DIGEST]

Pull an image or a repository from a registry

Options:
  -a, --all-tags                Download all tagged images in the repository
      --disable-content-trust   Skip image verification (default true)
      --help                    Print usage

The pull command will go up to the web site and grab the image and download it to our local machine.

$ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
aafe6b5e13de: Pull complete 
0a2b43a72660: Pull complete 
18bdd1e546d2: Pull complete 
8198342c3e05: Pull complete 
f56970a44fd4: Pull complete 
Digest: sha256:f3a61450ae43896c4332bda5e78b453f4a93179045f20c8181043b26b5e79028
Status: Downloaded newer image for ubuntu:latest

The pull command without any tag will download all Ubuntu images though I’ve already done it. To see what Docker images are available on our machine, we use docker images:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu              latest              5506de2b643b        4 weeks ago         199.3 MB

So, the output indicates only one image is currently on my local machine. We also see the image has a TAG inside of it.

As we can see from the command below, docker pull centos:latest, we can also be more specific, and download only the version we need. In Docker, versions are marked with tags.

$ docker pull centos:latest
centos:latest: The image you are pulling has been verified
5b12ef8fd570: Pull complete 
ae0c2d0bdc10: Pull complete 
511136ea3c5a: Already exists 
Status: Downloaded newer image for centos:latest

Here is the images on our local machine.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
centos              latest              ae0c2d0bdc10        2 weeks ago         224 MB
ubuntu              latest              5506de2b643b        4 weeks ago         199.3 MB

The command, docker images, returns the following columns:

REPOSITORY: The name of the repository, which in this case is “ubuntu”.
TAG: Tags represent a specific set point in the repositories’ commit history. As we can see from the list, we’ve pulled down different versions of linux. Each of these versions is tagged with a version number, a name, and there’s even a special tag called “latest” which represents the latest version.
IMAGE ID: This is like the primary key for the image. Sometimes, such as when we commit a container without specifying a name or tag, the repository or the tag is <NONE>, but we can always refer to a specific image or container using its ID.
CREATED: The date the repository was created, as opposed to when it was pulled. This can help us assess how “fresh” a particular build is. Docker appears to update their master images on a fairly frequent basis.
VIRTUAL SIZE: The size of the image.

Docker run

Now we have images on our local machine. What do we do with them? This is where docker run command comes in.

$ docker run  --help

Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

  -a, --attach=[]            Attach to STDIN, STDOUT or STDERR.
  --add-host=[]              Add a custom host-to-IP mapping (host:ip)
  -c, --cpu-shares=0         CPU shares (relative weight)
  --cap-add=[]               Add Linux capabilities
  --cap-drop=[]              Drop Linux capabilities
  --cidfile=""               Write the container ID to the file
  --cpuset=""                CPUs in which to allow execution (0-3, 0,1)
  -d, --detach=false         Detached mode: run the container in the background and print the new container ID
  --device=[]                Add a host device to the container (e.g. --device=/dev/sdc:/dev/xvdc)
  --dns=[]                   Set custom DNS servers
  --dns-search=[]            Set custom DNS search domains
  -e, --env=[]               Set environment variables
  --entrypoint=""            Overwrite the default ENTRYPOINT of the image
  --env-file=[]              Read in a line delimited file of environment variables
  --expose=[]                Expose a port from the container without publishing it to your host
  -h, --hostname=""          Container host name
  -i, --interactive=false    Keep STDIN open even if not attached
  --link=[]                  Add link to another container in the form of name:alias
  --lxc-conf=[]              (lxc exec-driver only) Add custom lxc options --lxc-conf="lxc.cgroup.cpuset.cpus = 0,1"
  -m, --memory=""            Memory limit (format: <number><optional unit>, where unit = b, k, m or g)
  --name=""                  Assign a name to the container
  --net="bridge"             Set the Network mode for the container
                               'bridge': creates a new network stack for the container on the docker bridge
                               'none': no networking for this container
                               'container:<name|id>': reuses another container network stack
                               'host': use the host network stack inside the container.  Note: the host mode gives the container full access to local system services such as D-bus and is therefore considered insecure.
  -P, --publish-all=false    Publish all exposed ports to the host interfaces
  -p, --publish=[]           Publish a container's port to the host
                               format: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort | containerPort
                               (use 'docker port' to see the actual mapping)
</name|id></optional></number>

docker run --help is a rather big help, and we have more:

  --privileged=false         Give extended privileges to this container
  --restart=""               Restart policy to apply when a container exits (no, on-failure[:max-retry], always)
  --rm=false                 Automatically remove the container when it exits (incompatible with -d)
  --security-opt=[]          Security Options
  --sig-proxy=true           Proxy received signals to the process (even in non-TTY mode). SIGCHLD, SIGSTOP, and SIGKILL are not proxied.
  -t, --tty=false            Allocate a pseudo-TTY
  -u, --user=""              Username or UID
  -v, --volume=[]            Bind mount a volume (e.g., from the host: -v /host:/container, from Docker: -v /container)
  --volumes-from=[]          Mount volumes from the specified container(s)
  -w, --workdir=""           Working directory inside the container

Currently we are on Ubuntu 14.04.1 LTS machine (local):

$ cat /etc/issue
Ubuntu 14.04.1 LTS \n \l

Now we’re going to Docker run centos image. This will create container based upon the image and execute the bin/bash command. Then it will take us into a shell on that machine that can continue to do things:

$ docker run -it centos:latest /bin/bash
[root@98f52715ecfa /]#

By executing it, we’re now on a bash. If we look at /etc/redhat-release:

[root@98f52715ecfa /]# cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)

We’re now on CentOS 7.0 on top of my Ubuntu 14.04 machine. We have an access to yum:

[root@98f52715ecfa /]# yum
Loaded plugins: fastestmirror
You need to give some command
Usage: yum [options] COMMAND

List of Commands:

check          Check for problems in the rpmdb
check-update   Check for available package updates
...

Let’s make a new file in our home directory:

[root@98f52715ecfa /]# ls
bin  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  selinux  srv  sys  tmp  usr  var
[root@98f52715ecfa /]# cd /home
[root@98f52715ecfa home]# ls
[root@98f52715ecfa home]# touch bogotobogo.txt
[root@98f52715ecfa home]# ls
bogotobogo.txt
[root@98f52715ecfa home]# exit
exit
k@laptop:~$

Docker ps – list containers

After making a new file on our Docker container, we exited from there, and we’re back to our local machine with Ubuntu system.

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

The docker ps lists containers but currently we do not have any. That’s because nothing is running. It shows only running containers.

We can list all containers using -a option:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
98f52715ecfa        centos:latest       "/bin/bash"         12 minutes ago      Exited (0) 5 minutes ago                       goofy_yonath        
f8c5951db6f5        ubuntu:latest       "/bin/bash"         4 hours ago         Exited (0) 4 hours ago                         furious_almeida

Docker restart

How can we restart Docker container?

$ docker restart --help

Usage: docker restart [OPTIONS] CONTAINER [CONTAINER...]

Restart a running container

  -t, --time=10      Number of seconds to try to stop for before killing the container. Once killed it will then be restarted. Default is 10 seconds.

We can restart the container that’s already created:

$ docker restart 98f52715ecfa
98f52715ecfa

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
98f52715ecfa        centos:latest       "/bin/bash"         20 minutes ago      Up 10 seconds                           goofy_yonath

Now we have one active running container, and it already executed the /bin/bash command.

Docker attach

The docker attach command allows us to attach to a running container using the container’s ID or name, either to view its ongoing output or to control it interactively. We can attach to the same contained process multiple times simultaneously, screen sharing style, or quickly view the progress of our daemonized process.

$ docker attach --help

Usage: docker attach [OPTIONS] CONTAINER

Attach to a running container

  --no-stdin=false    Do not attach STDIN
  --sig-proxy=true    Proxy all received signals to the process (even in non-TTY mode). SIGCHLD, SIGKILL, and SIGSTOP are not proxied.

We can attach to a running container:

$ docker attach 98f52715ecfa
[root@98f52715ecfa /]# 

[root@98f52715ecfa /]# cd /home
[root@98f52715ecfa home]# ls
bogotobogo.txt

Now we’re back to the CentOS container we’ve created, and the file we made is still there in our home directory.

Docker rm

We can delete the container:

[root@98f52715ecfa home]# exit
exit

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
98f52715ecfa        centos:latest       "/bin/bash"         30 minutes ago      Exited (0) 14 seconds ago                       goofy_yonath        
f8c5951db6f5        ubuntu:latest       "/bin/bash"         5 hours ago         Exited (0) 5 hours ago                          furious_almeida   

$ docker rm f8c5951db6f5
f8c5951db6f5

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
98f52715ecfa        centos:latest       "/bin/bash"         32 minutes ago      Exited (0) 2 minutes ago                       goofy_yonath

We deleted the Ubuntu container and now we have only one container, CentOS.

Remove all images and containers

We use Docker, but working with it creates lots of images and containers. So, we may want to remove all of them to save disk space.

To delete all containers:

$ docker rm $(docker ps -a -q)

To delete all images:

$ docker rmi $(docker images -q)

Here the -a and -q do this:

-a: Show all containers (default shows just running)
-q: Only display numeric IDs

Home

DOCKER – ELK 7.6 : KIBANA PART 1

Posted on November 3, 2020November 18, 2020 by thanhnguyen118

Install Kibana 7.6 with Docker

Kibana is available as Docker images. The images use centos:7 as the base image.

A list of all published Docker images and tags is available at www.docker.elastic.co.

Pulling the image

Issue a docker pull command against the Elastic Docker registry:

$ docker pull docker.elastic.co/kibana/kibana:7.6.2

Running Kibana on Docker for development

Kibana can be quickly started and connected to a local Elasticsearch container for development or testing use with the following command:

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
e49fee1e070a        docker.elastic.co/logstash/logstash:7.6.2             "/usr/local/bin/dock…"   12 minutes ago      Up 12 minutes                                                        friendly_antonelli
addeb5426f0a        docker.elastic.co/beats/filebeat:7.6.2                "/usr/local/bin/dock…"   11 hours ago        Up 11 hours                                                          filebeat
caa1097bc4af        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   2 days ago          Up 2 days           0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   nifty_mayer   

$ docker run --link caa1097bc4af:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.6.2
...
{"type":"log","@timestamp":"2020-03-31T16:33:14Z","tags":["listening","info"],"pid":6,"message":"Server running at http://0:5601"}
{"type":"log","@timestamp":"2020-03-31T16:33:14Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at http://0:5601"}


$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
620c39470d7f        docker.elastic.co/kibana/kibana:7.6.2                 "/usr/local/bin/dumb…"   2 hours ago         Up 2 hours          0.0.0.0:5601->5601/tcp                           dazzling_chatterjee
e49fee1e070a        docker.elastic.co/logstash/logstash:7.6.2             "/usr/local/bin/dock…"   3 hours ago         Up 3 hours                                                           friendly_antonelli
addeb5426f0a        docker.elastic.co/beats/filebeat:7.6.2                "/usr/local/bin/dock…"   14 hours ago        Up 14 hours                                                          filebeat
caa1097bc4af        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   2 days ago          Up 2 days           0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   nifty_mayer

Accessing_Kibana

Kibana is a web application that we can access through port 5601. All we need to do is point our web browser at the machine where Kibana is running and specify the port number. For example, localhost:5601 or http://YOURDOMAIN.com:5601. If we want to allow remote users to connect, set the parameter server.host in kibana.yml to a non-loopback address.

When we access Kibana, the Discover page loads by default with the default index pattern selected. The time filter is set to the last 15 minutes and the search query is set to match-all (\*).

Specify an index pattern that matches the name of one or more of our Elasticsearch indices. The pattern can include an asterisk (*) to matches zero or more characters in an index’s name. When filling out our index pattern, any matched indices will be displayed.

Click “Create index pattern” to add the index pattern. This first pattern is automatically configured as the default. When you have more than one index pattern, we can designate which one to use as the default by clicking on the star icon above the index pattern title from Management > Index Patterns.

All done! Kibana is now connected to our Elasticsearch data. Kibana displays a read-only list of fields configured for the matching index.

Adding Kibana Sample Data from elastic.co

The following sections are based on https://www.elastic.co/guide/en/kibana/7.6/tutorial-sample-data.html

Now, let’s run ELK stack using ELK Stack with docker compose.

Before we do that, let’s modify the setup for xpack in “elasticsearch/config/elasticsearch.yml” to set “xpack.security.enabled: true”. Otherwise, we may the following error:

 {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

$ docker-compose up -d
Creating network "einsteinish-elk-stack-with-docker-compose_elk" with driver "bridge"
Creating einsteinish-elk-stack-with-docker-compose_elasticsearch_1 ... done
Creating einsteinish-elk-stack-with-docker-compose_kibana_1        ... done
Creating einsteinish-elk-stack-with-docker-compose_logstash_1      ... done

$ docker-compose ps
                          Name                                         Command               State                                        Ports                                      
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
einsteinish-elk-stack-with-docker-compose_elasticsearch_1   /usr/local/bin/docker-entr ...   Up      0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                  
einsteinish-elk-stack-with-docker-compose_kibana_1          /usr/local/bin/dumb-init - ...   Up      0.0.0.0:5601->5601/tcp                                                          
einsteinish-elk-stack-with-docker-compose_logstash_1        /usr/local/bin/docker-entr ...   Up      0.0.0.0:5000->5000/tcp, 0.0.0.0:5000->5000/udp, 5044/tcp, 0.0.0.0:9600->9600/tcp
$

On the Kibana home page, click the link underneath Add sample data.

On the Sample flight data card, click Add data.

Once the data is added, click View data > Dashboard.

Now, we are on the Global Flight dashboard, a collection of charts, graphs, maps, and other visualizations of the the data in the kibana_sample_data_flights index.

Filtering the Sample Data

In the Controls visualization, set an Origin City and a Destination City.
Click Apply changes. The OriginCityName and the DestCityName fields are filtered to match the data we specified. For example, this dashboard shows the data for flights from London to Oslo.
To add a filter manually, click Add filter in the filter bar, and specify the data we want to view.
When we are finished experimenting, remove all filters.

Querying the Data

To find all flights out of Rome, enter this query in the query bar and click Update:OriginCityName:Rome
For a more complex query with AND and OR, try this: OriginCityName:Rome AND (Carrier:JetBeats OR “Kibana Airlines”)
When finished exploring the dashboard, remove the query by clearing the contents in the query bar and clicking Update.

Discovering the Data

In Discover, we have access to every document in every index that matches the selected index pattern. The index pattern tells Kibana which Elasticsearch index we are currently exploring. We can submit search queries, filter the search results, and view document data.

In the side navigation, click Discover.
Ensure kibana_sample_data_flights is the current index pattern. We might need to click New in the menu bar to refresh the data.
To choose which fields to display, hover the pointer over the list of Available fields, and then click add next to each field we want include as a column in the table. For example, if we add the DestAirportID and DestWeather fields, the display includes columns for those two fields.

Editing Visualization

We have edit permissions for the Global Flight dashboard, so we can change the appearance and behavior of the visualizations. For example, we might want to see which airline has the lowest average fares.

In the side navigation, click Recently viewed and open the Global Flight Dashboard.
In the menu bar, click Edit.
In the Average Ticket Price visualization, click the gear icon in the upper right.
From the Options menu, select Edit visualization.

Create a bucket aggregation

In the Buckets pane, select Add > Split group.
In the Aggregation dropdown, select Terms.
In the Field dropdown, select Carrier.
Set Descending to 4.
Click Apply changes apply changes button.

Save the Visualization

In the menu bar, click Save.
Leave the visualization name as is and confirm the save.
Go to the Global Flight dashboard and scroll the Average Ticket Price visualization to see the four prices.
Optionally, edit the dashboard. Resize the panel for the Average Ticket Price visualization by dragging the handle in the lower right. We can also rearrange the visualizations by clicking the header and dragging. Be sure to save the dashboard.

Inspect the data

Seeing visualizations of our data is great, but sometimes we need to look at the actual data to understand what’s really going on. We can inspect the data behind any visualization and view the Elasticsearch query used to retrieve it.

In the dashboard, hover the pointer over the pie chart, and then click the icon in the upper right.
From the Options menu, select Inspect. The initial view shows the document count.
To look at the query used to fetch the data for the visualization, select View > Requests in the upper right of the Inspect pane.

Remove the sample data set

When we’re done experimenting with the sample data set, we can remove it.

Go to the Sample data page.
On the Sample flight data card, click Remove.

Note: Continues to Docker – ELK 7.6 : Kibana on Centos 7 Part 2.

In the tutorial, we’ll build our own dashboard composed of 4 visualization panels as the following:

	NguyenHai on Sơ lược(Sumary) Robot Operatin…
	Nguyễn Văn Hải on Sơ lược(Sumary) Robot Operatin…
	Lam on Vagrant là gì? Làm thế nào để…
	Huy Trịnh on Sơ lược(Sumary) Robot Operatin…
	vo xuan loc on Sơ lược(Sumary) Robot Operatin…