Python Automation and Machine Learning for EM and ICs

An Online Book, First Edition by Dr. Yougui Liao (2019)

Practical Electron Microscopy and Database - An Online Book

Table of Contents/Index

Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix


K-means clustering and PCA for failure analysis	Introduction
Clusters (Kubernetes, Apache Mesos, Spark Standalone, Apache Hadoop YARN) in Apache Spark	Introduction
Apache Spark applications to a Kubernetes cluster	Introduction
Apache Kudu	Introduction
Google Kubernetes Engine (GKE)	Introduction
Ingesting data in Hadoop (Sqoop, Flume, Kafka, NiFi)	Introduction
Filters (Kernels) in ML	Introduction
Knowledge Engineering in ML	Introduction
Knowledge base (repository)	Introduction
Keyword arguments	Introduction

Keras	Introduction
Comparison between Keras and Estimators (tf.estimators)	Introduction
3 ways to create a Keras model with TensorFlow
Sequential API to create a Keras model with TensorFlow	Introduction
Functional API to create a Keras model with TensorFlow	Introduction
Model Subclassing to create a Keras model with TensorFlow	Introduction
tf.keras API	Introduction
Keras preprocessing layers	Introduction
keras.layers.normalization	Introduction
tf.keras.layers.CategoryEncoding	Introduction
tf.keras.layers.Discretization	Introduction
tf.keras.layers.Hashing	Introduction
tf.keras.layers.StringLookup	Introduction
tf.keras.layers.IntegerLookup	Introduction
tf.keras.model.save()	Introduction
tf.keras.optimizers.Adam	Introduction
tf.keras.layers.TextVectorization	Introduction
k-means algorithm	Introduction
K-Means clustering for images	Introduction
sklearn.cluster.KMeans()	Introduction
Double click of mouse	Introduction
Mouse right-click	Introduction
Mouse left-click	Introduction
Left click a specific position	Introduction
Right click a specific position	Introduction
Double click a specific position	Introduction
Find the values of the keys	Introduction
Save key and escape (ESC) key	code. code.
Keyword search function/check whether or not a string is within another string (a space is included as a string character)	Introduction
Text/keyword classification/sort/prediction, training/test e.g. Youtube spam	Introduction
Knowledge-based agents	Introduction
K-Fold Cross-Validation	Introduction
Knuth-Morris-Pratt (KMP) algorithm (a string-searching algorithm)	Introduction
Indicator function/Kronecker delta function	Introduction
Linear correlation between two variables with Pearson Correlation Coefficient, Spearman Rank Correlation Coefficient, Kendall's Tau, Linear Regression, Coefficient of Determination and Correlation Ratio	Introduction
Kernel density estimation (KDE)	Introduction
Bandwidth parameter (τ) in LWR and KDE	Introduction
Print the files and keyword occurrence which have been searched from a ppt file	Introduction
from keyboard import is_pressed (Esc, check pressed key)	Introduction
Check if a key exists in a dictionary	Introduction
Create dictionary from nested (sublist) list and get the values with keys	Introduction
Kernel tricks and kernel function	Introduction
Codes: Automation of Mouse Movements and Clicks, and keyboard control (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli)	Introduction
Principle and troubleshooting: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli)	Introduction
Kendall Tau Rank Correlation Coefficient	Introduction
kfp.dsl package versus pipelines and components	Introduction
K-Nearest Neighbours (KNN algorithm):sklearn, model_selection, train_test_split, preprocessing, StandardScaler, transform(), fit_transform(), .fit(), neighbors, KNeighborsClassifier, KNeighborsClassifier(), predict(), metrics, accuracy_score(), and classification_report	Introduction

Keywords	Introduction
Keyword extraction methods	Introduction
Keyword extraction methods from documents in Natural Language Processing (NLP)	Introduction
Rapid Automatic Keyword Extraction (RAKE)	Introduction
Stopwords/stoplist	Introduction
Candidate keywords	Introduction
Keyword module in Python	Introduction
Build own/customized keyword candidates	Introduction
Keyword scores	Introduction
Put the keywords in a grouped string into the first available cells in the corresponding columns in a csv file	Introduction
keep_default_na	CSV: whether to include the default set of missing values in addition to the ones specified in na_values
keep_date_col	CSV: if True, then date component columns passed into parse_dates will be retained in the output (False by default).
keep='first'/keep='last'	CSV: (code)
Euclidean distance	Introduction
Accuracy/precision test	Introduction
Get response of the nearest neighbors	Introduction
Get the nearest neighbors	Introduction
random.random()	Introduction
f1_score()	f1_score: float or array of float, shape = [n_unique_labels] F1 score of the positive class in binary classification or weighted average of the F1 scores of each class for the multiclass task. (code)
from selenium.webdriver.common.keys import Keys	(code)
pyautogui.click(x, y, t)	x and y: coordinates, and t: times of clicks. Introduction.
.rightClick()	.rightClick(x=moveToX, y=moveToY) (Code)
.middleClick()	.middleClick(x=moveToX, y=moveToY) (Code)
.doubleClick()	.doubleClick(x=moveToX, y=moveToY) (Code)
.tripleClick()	.tripleClick(x=moveToX, y=moveToY) (Code)
confusion_matrix	(code)
Keyboard
write()	Save as a text file. Instruction. (Code)
.typewrite()	E.g. .typewrite('Hello world!\n', interval=secs_between_keys); .typewrite(['a', 'b', 'c', 'left', 'backspace', 'enter', 'f1'], interval=secs_between_keys). (Code)
enter	(Code)
ESC	(Code)
F1	(Code)
F2	(Code)
F3	(Code)
F4	(Code)
F5	(Code)
F6	(Code)
F7	(Code)
F8	(Code)
F9	(Code)
F10	(Code)
F11	(Code)
F12	(Code)
keyDown()	Find the values of the keys. Several key strings which will be pressed down in order, and then released in reverse order. Strings can be passed to this function: '\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '\|', '}', '~', 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', 'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab', 'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', 'command', 'option', 'optionleft', 'optionright'. (code)
keyUp()
press()
hold()	Holds a key conveniently and passes a string from the pyautogui.KEYBOARD_KEYS such as shift, ctrl, alt. (code)
GetKeyState(VK_NUMLOCK)	Turn off or on Num Lock. .press("numlock") (code).
GetKeyState(VK_CAPITAL)	Caps Lock, .press("capslock"). (code).
hotkey()	Several key strings which will be pressed down in order, and then released in reverse order. Strings can be passed to this function: '\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '\|', '}', '~', 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', 'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab', 'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', 'command', 'option', 'optionleft', 'optionright'. (code)
.scroll()	.scroll(amount_to_scroll, x=moveToX, y=moveToY). (code).
.keys()	Return the dictionary's keys. General.

**kwargs	Allows you to pass multiple, varying, keyworded variable length of arguments to a function. It should be used if you want to handle named arguments in a function. Introduction.
kron(a, b)	Kronecker product of two arrays.
__kwdefaults__/dict	name/type: default values for the keyword-only formal parameters
kind	Can be 'area', 'bar', 'barh', 'density', 'hist', 'kde', 'line', 'pie'
hotkey('k')	Introduction
.kill()	(code).
Variable length arguments (args and *kwargs)	Introduction

K-means clustering and PCA for failure analysis

Introduction

Clusters (Kubernetes, Apache Mesos, Spark Standalone, Apache Hadoop YARN) in Apache Spark

Introduction

Apache Spark applications to a Kubernetes cluster

Introduction

Apache Kudu

Introduction

Google Kubernetes Engine (GKE)

Introduction

Ingesting data in Hadoop (Sqoop, Flume, Kafka, NiFi)

Introduction

Filters (Kernels) in ML

Introduction

Knowledge Engineering in ML

Introduction

Knowledge base (repository)

Introduction

Keyword arguments

Introduction

Keras

Introduction

Comparison between Keras and Estimators (tf.estimators)

Introduction

3 ways to create a Keras model with TensorFlow

Sequential API to create a Keras model with TensorFlow

Introduction

Functional API to create a Keras model with TensorFlow

Introduction

Model Subclassing to create a Keras model with TensorFlow

Introduction

tf.keras API

Introduction

Keras preprocessing layers

Introduction

keras.layers.normalization

Introduction

tf.keras.layers.CategoryEncoding

Introduction

tf.keras.layers.Discretization

Introduction

tf.keras.layers.Hashing

Introduction

tf.keras.layers.StringLookup

Introduction

tf.keras.layers.IntegerLookup

Introduction

tf.keras.model.save()

Introduction

tf.keras.optimizers.Adam

Introduction

tf.keras.layers.TextVectorization

Introduction

k-means algorithm

Introduction

K-Means clustering for images

Introduction

sklearn.cluster.KMeans()

Introduction

Double click of mouse

Introduction

Mouse right-click

Introduction

Mouse left-click

Introduction

Left click a specific position

Introduction

Right click a specific position

Introduction

Double click a specific position

Introduction

Find the values of the keys

Introduction

Save key and escape (ESC) key

code. code.

Keyword search function/check whether or not a string is within another string (a space is included as a string character)

Introduction

Text/keyword classification/sort/prediction, training/test e.g. Youtube spam

Introduction

Knowledge-based agents

Introduction

K-Fold Cross-Validation

Introduction

Knuth-Morris-Pratt (KMP) algorithm (a string-searching algorithm)

Introduction

Indicator function/Kronecker delta function

Introduction

Linear correlation between two variables with Pearson Correlation Coefficient, Spearman Rank Correlation Coefficient, Kendall's Tau, Linear Regression, Coefficient of Determination and Correlation Ratio

Introduction

Kernel density estimation (KDE)

Introduction

Bandwidth parameter (τ) in LWR and KDE

Introduction

Print the files and keyword occurrence which have been searched from a ppt file

Introduction

from keyboard import is_pressed (Esc, check pressed key)

Introduction

Check if a key exists in a dictionary

Introduction

Create dictionary from nested (sublist) list and get the values with keys

Introduction

Kernel tricks and kernel function

Introduction

Codes: Automation of Mouse Movements and Clicks, and keyboard control (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli)

Introduction

Principle and troubleshooting: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli)

Introduction

Kendall Tau Rank Correlation Coefficient

Introduction

kfp.dsl package versus pipelines and components

Introduction

K-Nearest Neighbours (KNN algorithm):sklearn, model_selection, train_test_split, preprocessing, StandardScaler, transform(), fit_transform(), .fit(), neighbors, KNeighborsClassifier, KNeighborsClassifier(), predict(), metrics, accuracy_score(), and classification_report

Introduction

Keywords

Introduction

Keyword extraction methods

Introduction

Keyword extraction methods from documents in Natural Language Processing (NLP)

Introduction

Rapid Automatic Keyword Extraction (RAKE)

Introduction

Stopwords/stoplist

Introduction

Candidate keywords

Introduction

Keyword module in Python

Introduction

Build own/customized keyword candidates

Introduction

Keyword scores

Introduction

Put the keywords in a grouped string into the first available cells in the corresponding columns in a csv file

Introduction

keep_default_na

CSV: whether to include the default set of missing values in addition to the ones specified in na_values

keep_date_col

CSV: if True, then date component columns passed into parse_dates will be retained in the output (False by default).

keep='first'/keep='last'

CSV: (code)

Euclidean distance

Introduction

Accuracy/precision test

Introduction

Get response of the nearest neighbors

Introduction

Get the nearest neighbors

Introduction

random.random()

Introduction

f1_score()

f1_score: float or array of float, shape = [n_unique_labels]
F1 score of the positive class in binary classification or weighted average of the F1 scores of each class for the multiclass task. (code)

from selenium.webdriver.common.keys import Keys

(code)

pyautogui.click(x, y, t)

x and y: coordinates, and t: times of clicks. Introduction.

.rightClick()

.rightClick(x=moveToX, y=moveToY) (Code)

.middleClick()

.middleClick(x=moveToX, y=moveToY) (Code)

.doubleClick()

.doubleClick(x=moveToX, y=moveToY) (Code)

.tripleClick()

.tripleClick(x=moveToX, y=moveToY) (Code)

confusion_matrix

(code)

Keyboard

write()

Save as a text file. Instruction. (Code)

.typewrite()

E.g. .typewrite('Hello world!\n', interval=secs_between_keys); .typewrite(['a', 'b', 'c', 'left', 'backspace', 'enter', 'f1'], interval=secs_between_keys). (Code)

enter

(Code)

ESC

(Code)

F10

(Code)

F11

(Code)

F12

(Code)

keyDown()

Find the values of the keys. Several key strings which will be pressed down in order, and then released in reverse order. Strings can be passed to this function: '\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~', 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', 'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab', 'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', 'command', 'option', 'optionleft', 'optionright'. (code)

keyUp()

press()

hold()

Holds a key conveniently and passes a string from the pyautogui.KEYBOARD_KEYS such as shift, ctrl, alt. (code)

GetKeyState(VK_NUMLOCK)

Turn off or on Num Lock. .press("numlock") (code).

GetKeyState(VK_CAPITAL)

Caps Lock, .press("capslock"). (code).

hotkey()

Several key strings which will be pressed down in order, and then released in reverse order. Strings can be passed to this function: '\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~', 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', 'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab', 'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', 'command', 'option', 'optionleft', 'optionright'. (code)

.scroll()

.scroll(amount_to_scroll, x=moveToX, y=moveToY). (code).

.keys()

Return the dictionary's keys. General.

**kwargs

Allows you to pass multiple, varying, keyworded variable length of arguments to a function. It should be used if you want to handle named arguments in a function. Introduction.

kron(a, b)

Kronecker product of two arrays.

__kwdefaults__/dict

name/type: default values for the keyword-only formal parameters

kind

Can be 'area', 'bar', 'barh', 'density', 'hist', 'kde', 'line', 'pie'

hotkey('k')

Introduction

.kill()

(code).

Variable length arguments (*args and **kwargs)

Introduction