Electron microscopy
Python Automation and Machine Learning for ICs: Chapter N
- Python Automation and Machine Learning for ICs -
- An Online Book: Python Automation and Machine Learning for ICs by Yougui Liao -
Python Automation and Machine Learning for ICs                                                        http://www.globalsino.com/ICs/        

Table of Contents/Index 
Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix



Skills Network Labs (SN Labs, IBM) Introduction
Mask R-CNN (Mask Region-based Convolutional Neural Network)  Introduction
Google Cloud Natural Language API Introduction
Semantic Segmentation Using U-Net with EfficientNet and Pixelshuffle Introduction
Attention-Guided Neural Network (AGNN)  Introduction
ResNet (Residual Network)   Introduction
Parametric and non-parametric learning algorithms Introduction
Ingesting data in Hadoop (Sqoop, Flume, Kafka, NiFi) Introduction
Hadoop MapReduce used by Google, Netflix, Amazon and Machine Learning Introduction
Impact of corpus narrowness on language model training Introduction
Comparison between Naive Bayes algorithms and Bayesian machine learning techniques Introduction
n-grams Introduction
Feedforward neural network Introduction
Biological (human brain) and AI neural networks Introduction
Nearest-neighbor (NN) classification Introduction
Node consistency Introduction
Convert a number type to another Introduction
Try a number of times before exception or fail Introduction
Check updated new files in a folder Introduction
Find nearest white pixel to a given pixel location on an binary image Introduction
Dot notation Introduction
Non-linearity in machine learning Introduction
t-SNE (t-distributed stochastic neighbor embedding, from sklearn.manifold import TSNE) Introduction
Trainable and non-trainable layers Introduction
.norm() (Taxicab Norm, Manhattan Norm, Euclidian Norm and Vector Max Norm) Introduction
.new() Introduction
tf.keras.layers.normalization Introduction
(Deep) neural network Introduction
Neural networks with TensorFlow Introduction
(Deep) Convolutional neural networks (CNN) Introduction
Use __name__ to control execution of the code code
Open a new tab in an application window Introduction
Create a new presentation Introduction
Add a new slide into an existing ppt or a created ppt file Introduction
Launch the existing opened application if there is or start a new one if there is not Introduction
Check if all the (and how many, length of a string) characters in the text are digits/numbers (Code)
Convert capital alphabet letters/characters to number Introduction
Get the name of the current/most front window Introduction
Notepad window Introduction
Check to see if or get a window with a name containing specific titles or texts Introduction
Option/selection/choice methods ("pop-up windows of Yes and No ") Introduction
Calculate/pass the arbitrary (any) number of variables or input arguments Introduction
Convert strings to number (integers/float) Introduction
Merge/combine two text files into a new text file, add a new line to the beginning of a text file Introduction
Sum a list of numbers in any length Introduction
Mixing of using numbers and strings by conversions Introduction
Build databases with different/uncertain number of members Introduction
Cheatsheet of numpy Introduction
Numpy: Access the element at the second row, the third entry, access a specific row or a column, access some elements (submatrix), or replace/modify an element in the array, print a transfer of an array, access array under conditions or filtering Introduction
Calculations in DataFrame: Add a column, calculate for a new column, delete a column, all the rows with values greater than 30 in "Score A" column Introduction
Handle NaN value in DataFrame, replace empty cells with ...
Count the numbers of uppercase letters, lowercase letters and spaces in a string and then swap the cases of the letters.
Swap two numbers code
Copy images to a different folder/Save an image in a new folder code, code. code. code.
Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values CSV: Introduction. code.
Search position of numbers code
Loop through numbers in a range code
Find/search birthyear by name code
Find the greatest of three numbers code1, code2, code3.
Reverse the digits of a given number code
Handle "No Results", "Not Found" (Error vs. Exceptions) Introduction
Check if file exists or not (Cannot find a specific file?) Introduction
Check if an item/element is in a list or not Introduction
Add a new slide into an existing ppt, or work on existing slides Introduction
Check file existence with partial filename Introduction
Print/obtain a specific digit in a number, e.g. in a cell value in cvs Introduction
Empty and None Introduction
Check whether a file is empty or not Introduction
Find files with a specific file extension/type or with file names ending with specific characters Introduction
Watchdog for monitoring specific file or files with specific extension and/or file names Introduction
Move(remove) all files from original folder in a directory to a new directory Introduction
Check whether or not a cell value in a column of a CSV file matchs a value in a column of another CSV file, then do something: e.g. add a value to another column of a csv file  Introduction
Check if CSV cell value is NaN Introduction
Time and date used as a file/folder name stamp (e.g. duplicate a file in the same folder) Introduction
Count how many (number) files and folders in a directory Introduction
Limit event/action numbers in the event List, then stop Introduction
Modify file path/directory by changing folder names by merging a list Introduction
Find the file names of the images in a pptx Introduction
Merge/combine two pptx files into one, including merging the pptx files with the words in a sentence as file names (not all words has pptx files) Introduction
Libraries used to convert incident documents into numerical vectors Introduction
Natural Language Processing (NLP) Introduction
Keyword extraction methods from documents in Natural Language Processing (NLP)
Count occurrence/nubmer of words/phrase in a text file Introduction
Extract pdf pages to form new pdf files Introduction
Generate text file with the bank of collecting all words, characters and strings from news Introduction
Remove \n in string or new line in txt/text file Introduction
Support-vector machines(SVM)/support-vector networks(SVN) Introduction
Nearest-neighbor (NN) cluster removal Introduction
Denoising/remove noise in images Introduction
Non-zero (Nonzero) pixel values from an image Introduction
List of notations for machine learning application to wafers Introduction
Evaluation (Precision and Recall) in Text classification with Naive Bayes Introduction
Precision, Recall, False Positive Rate, and False Negative Rate (Miss Rate or False Negative Proportion) Introduction
Text classification with Naive Bayes Introduction
 Stationary and Non-Stationary State Transitions in Markov Decision Process (MDP) Introduction
Q-Learning with Function Approximation (Deep Q-Network - DQN) Introduction
Grid world navigation Introduction
Nonlinear extensions of Independent Component Analysis (ICA) Introduction
Bayesian networks Introduction
Maximum Likelihood Estimation (MLE) of single Gaussian (normal) distribution Introduction
Number (size) of features in ML Introduction
Normalizing (normalization of) input in neural network Introduction
Logistic regression as a one-neuron/single-layer neural network (connection between linear & activation parts) Introduction
Number of neurons and layers in neural network Introduction
Edge detection of images in neural network Introduction
Neural network vs. end-to-end learning vs. black box model
Softmax regression (multinomial logistic regression)/softmax multi-class network/softmax classifier Introduction
Neuron (= linear + activation) introduction Introduction
No Free Lunch Theorems Introduction
"Norm" of parameters, and L1 Norm (Manhattan Norm) and L2 Norm (Euclidean Norm) Introduction
Logistic regression and Naive Bayes Introduction
Apple Neural Engine (ANE) Introduction
Feature vector and number of features Introduction
Single Naive Bayes (Gaussian Naive Bayes) versus Multinomial Naive Bayes Introduction
Artificial Neural Networks (ANNs) Introduction
Gaussian distribution and standard gaussian distribution (multivariate normal distribution) Introduction
K-Nearest Neighbours (KNN algorithm):sklearn, model_selection, train_test_split, preprocessing, StandardScaler, transform(), fit_transform(), .fit(), neighbors, KNeighborsClassifier, KNeighborsClassifier(), predict(), metrics, accuracy_score(), and classification_report Introduction
Normalized ratios Introduction
Negative log likelihood (NLL) Introduction
Probability density function (PDF): comparisons between (normal (gaussian) distribution, uniform distribution, exponential distribution and poisson distribution) Introduction
Exponential Family: Parameter, Sufficient Statistic, Natural Parameter, Base Measure and Log-Partition Function (Bernoulli distribution and Gaussian distribution) Introduction
Newton's method Introduction
Newton's method versus gradient descent Introduction
Non-parametric learning algorithm Introduction
Big O notation Introduction
Epsilon cover/ε-cover/epsilon-net Introduction
CycleGAN (Cycle-Consistent Adversarial Networks) Introduction
Symbols/notations used in ML Introduction
Nonasymptotic Analysis Introduction
Two-dimensional neural network Introduction
One-dimensional neural network Introduction
Multinomial Naive Bayes algorithm Introduction
Strong machine learning and NLP departments in universities Introduction
Percentages of information received through different senses (eye, nose, ear and hand feeling) Introduction
Elastic Net Introduction
Various names or terms that describe similar concepts or techniques in ML Introduction
Nonasymptotic versus asymptotic analysis Introduction
Natural Language Processing (NLP) approaches in addressing the Failure Analysis (FA) search problem Introduction
Generative Adversarial Network (GAN) technologies Introduction
Natural Language Processing (NLP) versus Text Introduction
Recurrent Neural Networks (RNN) Introduction
(Single) Naive Bayes/Gaussian Naive Bayes Introduction
NLTK (Natural Language Toolkit) Introduction
Nearest/most similar lyrics of a sentence to a CSV file Introduction
Natural language inference Introduction
Get directory/path/file name partially Introduction
Generate a file name, folder name. {}{}....format. Manipulation of file and folder names (rename file name and folder name): i) Create a new folder (e.g. with os.mkdir) and then copy all files from a folder to the new folder and rename the file, and then open the file. If the folder exists, then no file will be copied, but the file will still be opened. ii) Print and export the folder names and file names (with or without extensions) from a folder into a text file. iii) csv2image filename. Introduction
Count number of lines in a text file Introduction
Modify HTML webpage (e.g. with graph network by adding text/hyperlink in) Introduction
Check if a variable is a number, string or integer Introduction
Count the number of the pages in a single multi-page/frame image Introduction
Check if all the (and how many, length of a string) characters in the text are digits/numbers Introduction
Create dictionary from nested (sublist) list and get the values with keys Introduction
Skip/replace empty cells/NaN value from DataFrame/CSV file Introduction
Convert between numpy array and string Introduction
Continue script execution no matter whether some try fails or not (finally)
Get header/column names from DataFrame Introduction
Iterate over rows in a DataFrame/read and print row by row (number of columns and rows, df.shape[0]/df.shape[1]) Introduction
Remove rows if (multiple) NaN is more than a number in DataFrame Introduction
Read columns with numeric values/numbers only in dataframe Introduction
Change/rename a column name/header in a CSV file Introduction
Feature importance for Multinomial Naive Bayes algorithm Introduction
Get the latest/newest/most recent file in a folder within certain time/days Introduction
Check if a string is empty, NaN value or space only Introduction
Check and drop negative from dataframe pandas Introduction
y axis values are not ordered (disordered) Introduction
Compare (pattern/ratio of) two different columns, check whether column values match in DataFrame Introduction
Check whether one column contains number only and another column contains letters only or mixture of numbers and letters in DataFrame Introduction
Check the difference between two columns in DataFrame Introduction
Delete the column/row in a CSV file if they are empty or less than a number (or header/index only) Introduction
networkx for network Introduction
igraph for clustering and network Introduction
Networkit: a network/link/landscape tool Introduction
Pyvis: An interactive geometric graph network/link/landscape Introduction
na_values: NA or N/A values = missing values. optional list of strings to recognize as NaN (missing values), either in addition to or in lieu of the default set. To completely override the default values that are recognized as missing, specify keep_default_na = False. The default NaN recognized values are [’-1.#IND’, ’1.#QNAN’, ’1.#IND’, ’-1.#QNAN’, ’#N/A’,’N/A’, ’NA’, ’#NA’, ’NULL’, ’NaN’, ’-NaN’, ’nan’, ’-nan’]. read_csv(path, na_values=[5]) = the default values, in addition to 5 , 5.0 when interpreted as numbers are recognized as NaN;
read_csv(path, keep_default_na=False, na_values=[""]) = only an empty field will be NaN; read_csv(path, keep_default_na=False, na_values=["NA", "0"]) = only NA and 0 as strings are NaN; read_csv(path, na_values=["Nope"]) = the default values, in addition to the string "Nope" are recognized as NaN.
__add__, __call__, __contains__, __delitem__, __delattr__, __eq__, __enter__, __ge__, __getattribute__, __getnewargs__, __getattr__, __getitem__, __gt__, __hash__, __reduce__, __iadd__, __imul__, __init_subclass__, __index__, __int__, __invert__, __new__, __neg__, __reduce_ex__, __reversed__, __rmul__, __radd__, __rand__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __ror__, __round__,__rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __dir__, __doc__, __divmod__, __iter__, __le__, __lt__, __len__, __ne__, __repr__, __setattr__, __setitem__, __sizeof__, __lshift__, __sub__, __subclasshook__, __str__ Introduction
next() Introduction
Select specific columns from a DataFrame to form a new DataFrame Introduction
Add letter/commas/numbers/characters to the end/beginning of strings in a list Introduction
Replace the lines between two lines “xx” and “yy” in a text file with new lines Introduction
Selecting only numeric/number columns, and then select two specific columns for plot Introduction
Mean (average, .mean())/.sum()/maximum(.max())/minimum(.min())/number of non-null values(.count())/.median()/variance(.var())/standard deviation(.std()/pstdev()) Introduction
Plot workflow: Create new empty column in DataFrameMove the cells in a column to another column under certain conditionSelect specific columns for scatter plot
Cheatsheet of numbers Introduction
First N largest and smallest numbers in a list Introduction
Search/extract all the 4-digit numbers (with and without extension) from a given text Introduction
Jupyter notebooks Introduction
Normalize data in dataframe Introduction


names CSV: List of column names to use as column names. To replace header existing in file, explicitly pass header=0.


nrows CSV: Number of rows to read out of the file. Useful to only read a small portion of a large file
na_rep CSV: A string representation of a missing value (default ‘’)
names CSV: Names in headers of csv files. code. code.
nunique() CSV: (code)
neighbors (code)
Num Lock .press("numlock") (code).
Represent one of two values: True or False. code. code. code.
not     Condition must be false for the statement to be true
if ... not in ... Introduction. code.
if not (code)
if ... is None code.
.name (code)
name A string used for identification purposes only.
It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.
on_thread_start() Override this method instead of start(). start() calls this method.
This method is called right before this thread is started and this object’s run() method is invoked.
on_thread_stop() Override this method instead of stop(). stop() calls this method.
This method is called immediately after the thread is signaled to stop.
if __name__ == '__main__' Introduction. The processes starts reading the current file in order to execute the function specified. Without this clause, the import would first execute more process start calls, before getting to the function execution. code. code. code. code. Application example: run the page4853main3 program (as a module) through page4853main4 program. A similar example with defined functions is page4853main5.py. (code).
name() E.g. name[0] and name[1] prints the first and second letter of the string, respectively. However, negative indexing, -n, in a string refers to the character present at the nth position beginning from the end. Example code
Python has three types of numbers: integer, floating point and complex.
Integer/fractions/round /decimal/digits/floating
It does not have any fractional part. Introduction. int: Example code.
Floating Point It can store number with a fractional part
Complex It can store real and imaginary parts
Decimal Those having fixed precision
Rational Those having a numerator and a denominator
Sets Introduction, and data structures. Abstraction of a mathematical set
scipy.linalg.norm This function is able to return one of eight different matrix norms, or one of an infinite number of vector norms, depending on the value of the ord parameter. For tensors with rank different from 1 or 2, only ord=None is supported. It is for an old release of SciPy (version 0.14.0).
from numpy.linalg import norm (code).
from scipy.linalg import norm code
Norms and other numbers
linalg.norm(x[, ord, axis, keepdims]) Matrix or vector norm.
linalg.cond(x[, p]) Compute the condition number of a matrix.
linalg.det(a) Compute the determinant of an array.
linalg.matrix_rank(M[, tol, hermitian]) Return matrix rank of array using SVD method
linalg.slogdet(a) Compute the sign and (natural) logarithm of the determinant of an array.
trace(a[, offset, axis1, axis2, dtype, out]) Return the sum along diagonals of the array.
__name__/str name/type: Introduction. the function name. Example code
cv2.namedWindow code.
nrows Number of rows of subplots
ncols Number of columns of subplots
NumPy Introduction. Has advanced math functions and a rudimentary scientific computing package. It is used for all things "numbers and Python." In image processing, we are mainly making use of Numpy's array functionality. code1, code2.
Speed comparison with and without numpy Introduction
.ndim (code)
shape/shape[] (code)
size (code)
nbytes (code)
numpy.identity(n, dtype = None) Return a identity matrix i.e. a square matrix with ones on the main diagonal. code.
Numpy splicing draw a grey or color point or shape in an image. code.
.sort(axis=0) code.
import numpy Actually means import numpy as numpy.
import numpy as np Is equivalent to: (Array, code)
import numpy
np = numpy
del numpy
.random.permutation() code.
.randint() Random integer. code.
numpy.transpose() Reverse or permute the axes of an array; returns the modified array. For an array a with two axes, transpose(a) gives the matrix transpose of swapping. code.
.argwhere() (code)
np.linspace() Introduction. General, code. code.
np.full_like Return a full array with the same shape and type as a given array. Code.
.array General, number array . for image. code. in csv.
hist() (code)
.xlabel() (code)
.ylabel() (code)
.title() (code)
.text() (code)
.xlim() (code)
.ylim() (code)
grid() (code)
.arange() Introduction. Can change with an increment. General, code. code.
reshape() numpy.reshape(a, newshape, order='C', 'F', or 'A' - optional)[source]. Introduction. in csv.
numpy.ones(shape, dtype=None, order='C', *, like=None) Return a new array of given shape and type, filled with ones. General, code.
np.eye() General.

Create a Van der Monde matrix.

numpy.linalg This function is able to return one of eight different matrix norms, or one of an infinite number of vector norms, depending on the value of the ord parameter.
random.rand() General.
random.random() Introduction.
random.randn(d0, d1, ..., dn) Return a sample (or samples) from the “standard normal” distribution. Introduction.
.hstack General, code.
.vstack General, code.
random.default_rng Random generator: General. code.
Count duplicates/occurrence and show unique values in csv files CSV: Introduction
Merge two csv files CSV: Introduction
Split columns and merge in csv CSV: Split columns and then merge the splits in a csv file. Introduction
Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values CSV: Introduction. code.
Search/print/output the rows CSV: Print the rows if their cell values are greater than a specific value, in the csv file with numbers only; output the rows if the cell value is in a specific range. Introduction. code.
Sort a csv file with column CSV: Instruction: used "key = operator.itemgetter()".
Calculation in csv CSV: Compares with the ones which cannot be used for math calculation, find the maximum in a column. Introduction
Convert between rows and columns from csv CSV: Convert resulting row from CSV search into a column. Introduction
Replace/change to new headers in a csv file CSV: Introduction
Remove duplicate cell values from a csv file CSV: Introduction
Skip rows and/or columns in csv CSV: Introduction.
Nose Delivers an alternate test discovery and running process for unittest. This intends to mimic py.test’s behavior as much as it can.
NuPIC The Numenta Platform for Intelligent Computing (NuPIC) is a platform which aims to implement an HTM learning algorithm and make them public source as well. It is the foundation for future machine learning algorithms based on the biology of the neocortex.
Comparison between spaCy and Natural Language Toolkit (NLTK)
GetKeyState(VK_NUMLOCK) Turn off or on Num Lock. .press("numlock") (code).
hotkey('n') (code)
split() Introduction. Split a string by dots, split a file name by dots, split a file name from its extension. code. (code).
driver.get(“http://globalsino.com”) Navigate to URL
Navigate to URL
driver.navigate().refresh() Refresh page
driver.navigate().forward() Navigate forwards in browser history
driver.navigate().back() Navigate backward in browser history
NoSuchElementException() (code)
os.path.normcase() (code).
os.path.normpath() (code).
os.path.normpath(): Normalize/format the path string into a proper string for the OS (code)
np.nan (code)
numpy.all/np.all Introduction
.DataFrame() Introduction. .drop(), index, columns, axes, dtypes, size, shape, ndim, empty, T, values
Switches simulations  
Diodes simulations