Python Automation and Machine Learning for EM and ICs

An Online Book, First Edition by Dr. Yougui Liao (2019)

Practical Electron Microscopy and Database - An Online Book

Table of Contents/Index

Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix

   
LASSO (Least Absolute Shrinkage and Selection Operator)  
Skills Network Labs (SN Labs, IBM)  
Leveraging precision, speed, and automation: Integrating Mask R-CNN and YOLOv8  
"Extract, Transform, Load" (ETL) and " Extract, Load, Transform" (ELT) processes  
YOLOv8 (You Only Look Once, version 8)  
Comparison btween data lake and data warehouse  
Using proxy labels, building a labeling system, and utilizing a labeling service when historical labeled data is unavailable for ML projects  
OLS (Ordinary Least Squares) regression model  
Big data lifecycle  
Impact of corpus narrowness on language model training  
Guess and check algorithm with a combination of a for loop and an if statement  
Labor cost of data analysis with and without automation and ML techniques  
Trade-off between minimizing loss and minimizing complexity  
Mean squared error (MSE) (L2loss function, Euclidean loss)  
Python libraries for Bayesian ML techniques  
L1Loss (Absolute Loss or Mean Absolute Error (MAE))  
Language Model  
Replaces a symbol/character/letter in a string  
Plot with letters/words/character as x-/y-axis  
Linear Programming (LP) algorithm  
Colored Line Detection and Dot Annotation Transfer with Coordinate Export in Grayscale Images   
Constraint ("limit"/"range") satisfaction  
Local Search in ML  
Likelihood weighting  
First-order logic (FOL)  
Logical statements  
Modus ponens (a logical inference rule)  
Logic Puzzle in ML  
Propositional Logic Algorithms in ML  
Cost/loss function versus reward function  
Linearization  
Linear Quadratic Regulation (LQR)  
Q-Learning with Function Approximation (Deep Q-Network - DQN)  
Nonlinear extensions of Independent Component Analysis (ICA)  
Maximum Likelihood Estimation (MLE) of single Gaussian (normal) distribution  
Learning algorithm (ensemble learning) and pipeline  
Logistic regression as a one-neuron/single-layer neural network (connection between linear & activation parts)  
Number of neurons and layers in neural network  
Comparison among sigmoid, hyperbolic tangent (tanh) and rectified linear unit (ReLU) functions  
Neuron (= linear + activation) introduction  
Gini Loss  
Misclassification loss in decision trees  
Linear model versus polynomial model  
No Free Lunch Theorems  
Learning theory  
Leave-One-Out Cross-Validation (LOOCV)  
"Norm" of parameters, and L1 Norm (Manhattan Norm) and L2 Norm (Euclidean Norm)  
Logistic regression and Naive Bayes  
Open datasets, and open-source tools and libraries for ML practice  
Learning theory  
Generalization  
Bias and variance, and bias-variance trade-off in ML  
Model Complexity  
Convergence and Optimization  
Sample Complexity  
Probably Approximately Correct (PAC) learning  
Margin Theory  
No Free Lunch Theorems  
Layers  
Categorical features preprocessing layers  
Keras preprocessing layers  
tf.keras.layers.Hashing  
tf.keras.layers.CategoryEncoding  
tf.keras.layers.Discretization  
keras.layers.normalization  
tf.keras.layers.StringLookup  
tf.keras.layers.IntegerLookup  
tf.keras.layers.TextVectorization  
Output the web links obtained by Google Search  
Mirror/reflect image from left to right/from top to bottom  
Non-linearity in machine learning  
(Find) file size/find the largest/smallest file in a directory/folder  
.pack(side=LEFT)/.pack(side=RIGHT)/.place(x=, y=) --- position of the buttons  
Locate/find the center of a bright (maximum/highest intensity) spot in an image & find nearest white pixel to a given/specifical pixel location on an binary image  
Linear algebra  
.LabelEncoder()  
Rectified Linear Units (ReLUs)  
Least squares fit  
Long short-term memory (LSTM)  
Script execution limited by retry time  
Line detection on image  
Automatically review, scroll, click webpage and its link  
Mouse left single click  
Left click  
Left click a specific position  
Lock desktop  
Launch the existing opened application if there is or start a new one if there is not  
Loops (e.g. for loop) for 2D (two-dimensional) plot  
Convert capital alphabet letters/characters to number  
Type capital letters  
Modify/replace the line in a text file if a line contains specific string  
Move the mouse/cursor to the left or right  
Bind/link multiple commands to buttons  
Copy and then store it into memory and it can be pasted for use later  
Get the latest/newest/most recent file in a folder within certain time/days  
Select/input a folder/directory/path for later to be called to use  
Close file after reading a file: avoid file locking  
watchdog to look for filesystem changes  
Check the letters and symbols starting and ending with  
Calibrate and put a scale bar, and draw a line segment on an image  
Calculator of length accuracy in 3D structure  
Merge/combine two text files into a new text file, add a new line to the beginning of a text file  
Count the numbers of uppercase letters, lowercase letters and spaces in a string and then swap the cases of the letters.  
Draw a line in a image  
Draw lines manually and then label them with arrows  
Remove letters or characters on either side (both left and right sides) and stops when neither such letters no characters on either side  
Calculate the coordinates of a point in a given rectangle and the distance of a given point to a line  
Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values  
Bail out/terminate of a loop  
Loop through numbers in a range  
Load/launch images and ColorMixing in DigitalMicrograph  
Draw lines and arrows in a ppt  
Linear correlation between two variables with Pearson Correlation Coefficient, Spearman Rank Correlation Coefficient, Kendall's Tau, Linear Regression, Coefficient of Determination and Correlation Ratio  
.LINE  
Stability/reliability of locateCenterOnScreen()  
Find latitude and longitude of a place in a map  
Print and set file path as a variable (e.g. convert all characters in the pathname to lowercase)  
Check if a directory is empty; find the position/index of a particular file/folder in a file directory/path; remove folder or file level by level (or layer by laer) from its directory/path  
matplotlib.pyplot axis/text color (title, xticks, rotation, xlabel, ylabel, title, fontsize, grid(), legend(), show())  
top and left for pptx  
break and functions to exit a loop: Stop the loop, e.g. while loop immediately.  
check if a letter is in a string  
Plot pixel intensities (histogram) along a line of an image  
Global access to a local variable inside a function from outside of the function externally  
Top (ranking, best, must know) Python libraries/modules  
Measure length/distance on an image w/o calibrated bar  
Rotate (alignment) an image by line along the x- or y-axis  
Get pixel location/coordinates on an image using mouse click/events  
Crop/snip part of a image with definition by a pixel line  
Quit/exit/stop a process (including by pressing a letter)  
Supervised learning  
Extract the index of a string element in a list  
Unsupervised learning  
Reinforcement learning  
Break/exit/skip a function/code line after a certain time  
Libraries used to convert incident documents into numerical vectors  
Linear Support Vector Classifier (Linear SVC)  
Change/capitalize the case of the first letter of a string (first letter: uppercase; other letters: lowercase)  
Ranking/most popular programming languages for data analysts  
Remove \n in string or new line in txt/text file  
Remove duplicate/same lines in a text file  
Data labeling and annotation in supervised machine learning  
Global defects and local defects identified by defect denoising  
Laplacian filtering  
Clustering of Laplacian  
Draw circles/lines on images  
Table of applications of Python and its libraries  
L2 regularization/Ridge/ridge regularization/Tikhonov regularization  
Support Vector Machines (SVM) and Logistic Regression  
Hidden layer in deep learning neural network  
Convolutional Layers (CONV) in Deep Learning  
Fully Connected Layers (FC) in Deep Learning  
Laplace smoothing/Laplace correction/add-one smoothing  
Comparison between Poisson distribution, Gaussian (normal) distribution and logistic regression  
Logistic regression versus Gaussian discriminant analysis  
Joint likelihood  
Cross entropy (log loss/logistic loss)  
Softmax regression (multinomial logistic regression)/softmax multi-class network/softmax classifier  
Canonical response function/canonical link function  
Learning rule in ML  
GLM (Generalized Linear Model)  
Negative log likelihood (NLL)  
Exponential Family: Parameter, Sufficient Statistic, Natural Parameter, Base Measure and Log-Partition Function (Bernoulli distribution and Gaussian distribution)  
Perceptron algorithm and logistic regression  
Update parameters θj using gradient of the loss function  
Logistic function/sigmoid function  
Logistic regression  
Logistic regression versus linear regression  
Linear regression versus classification  
Comparison between mean squared error (MSE), absolute error (L1 Loss) and fourth-power loss  
Comparison between L1 Regularization and L1 Loss (absolute loss or mean absolute error (MAE))  
Convolution and convolutional layers  
Latent features and latent variables  
Bandwidth parameter (τ) in LWR and KDE  
Parametric learning algorithm  
Non-parametric learning algorithm  
Locally Weighted Regression (LWR)  
Learning Algorithm (estimator)  
Learning rate  
Linear regression and its algorithm  
Generalization Error/Generalization Loss/Test Error/Expected Error of Hypothesis/Risk  
Lipschitzness/Lipschitz continuity  
Check existence of phrase on text file line-by-line  
Expected risk (population risk, expected value of loss or error)  
Central Limit Theorem (CLT)  
Empirical loss/training loss  
Empericial loss versus population loss  
Linear Discriminant Analysis  
IDLE (integrated development and learning environment) and integrated development environment (IDE)  
Natural Language Processing (NLP) approaches in addressing the Failure Analysis (FA) search problem  
Natural Language Processing (NLP) versus Text  
Bayes' theorem (Bayes rule or Bayes law) in machine learning  
Analysis of papers/publications/literature in machine learning and Python applications  
Multiple linear regression  
Likelihood and maximum Likelihood estimation (MLE)  
"Label space" in machine learning  
Predicted label  
"True label" ("observed label") in machine learning  
Predicted label versus predictor (feature)  
Loss (risk, cost, objective) function  
Supervised, unsupervised and reinforcement learning  
Read line-by-line from a text file  
Extract the first or last N letters from a string  
Nearest/most similar lyrics of a sentence/text to a CSV file  
Natural language inference  
NLTK (Natural Language Toolkit)  
Large and small datasets in ML  
(Text and image) contrastive learning  
Remove/reload/unload an imported module/function/script  
Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc: localhost, insert rows, update, count updated, delete rows, comparision between extract data by Python and SQL itself  
Count number of lines in a text file  
Pyvis: An interactive geometric graph network/link/landscape  
Networkit: a network/link/landscape tool  
Loop through a Python dictionary  
Delete the column/row in a CSV file if they are empty or less than a number (or header/index only)  
scipy.optimize.linprog function  
Plot multiple images on the same figure by hiding x- and y-(tick) labels on axis  
Fairness Analysis and Python Libraries  
Convert set into a list and vice versa  
Lock a file to prevent deleting, and then release the file once job is done  
Lock a file to prevent deleting, and then release the file once job is done  
Exception LookupError (string index)  
OverflowError (too large to store)  
Calculation with combinations of variables from lists  
Remove empty strings from list of strings  
Count how many empty strings in a list  
Check if two lists are same/identical  
Remove the substring after the first or last character "::" in a given string, or extract the substring between the first and last "::"  
Penalized regression (Lasso and Ridge)  
Compare (pattern/ratio of) two different columns, check whether column values match in DataFrame  
Check whether one column contains number only and another column contains letters only or mixture of numbers and letters in DataFrame  
Check the difference between two columns in DataFrame  
Get the last line in a text file  
Check if Windows/PC screen is locked  
Check all the imported/current modules/libraries  
Check if all the (and how many, length of a string) characters in the text are digits/numbers  
Different behavior of automation execution (e.g. pyautogui) locally or remotely through internet  
Data plot with labels  
Leetcode for Google/Amazon  
Create a log (log.log) file to monitor script execution  
Last n days/weeks/months (.to_datetime(x), .set_index(y), .last(z), .reset_index(), and .max() in pandas)  
Lists/list(). A list is a collection of objects. List is denoted with square brackets []. Declaration: myList = list(). Unlike strings, lists are mutable, a list can also contain list(s). Lists in memory are considered to be an object in memory.  
Check if one list is subset of another  
Check if a list is empty or not  
Check if an item/element is in a list or not  
Check if a file/folder exists or not (Cannot find a specific file/folder? a specific folder in the path? select specific folders to form a string, split a dos path into its components, and then print the list, or check files with extension)  
Check if an element is in a sublist of a list  
Get/list immediate subdirectories/subfolders; get only the last part of a path/folder/drive; split a dos path into its components, and then print the list  
Access elements in a list and sublist  
Replace/substitute a item in a list  
List all files and directories which has specific files or files with specific extensions  
Get/list immediate subdirectories/subfolders; get only the last part of a path  
Find minimum and maximum values in a list  
Change/swap values in a list  
Modify a list (e.g. add/insert an item between items)  
Plot a list of x, y coordinates to an image  
Sort a list  
Split a sentence/string into list of words, remove all special characters from a sentence  
Modify file path/directory by changing folder names by merging a list  
Limit event/action numbers in the event List, then stop  
Loop through a list (e.g. for loop)  
Remove an item/element/duplicates from a list  
Get the list of the methods for a function  
index("") In list.  
Extract elements from a list (different way from removing elements to get part of the list)  
Convert csv/dataframe column to a list or vice versa  
Create dictionary from nested (sublist) list and get the values with keys  
Convert between numpy array, string or list of string  
Check if two lists have the same elements  
Extract the least/most frequency/duplicate/occurrence element in a list  
Find common/different elements/items between two lists/sets  
Collect the file list in a folder into a csv file  
os.listdir(). Output the file list of images in a folder, but only returns the names.  
Data structures (Data science, and comparison between list, tuple, set, dictionary)  
Split a list into columns  
Comparison between =, ==, .copy(), copy.copy() for "list": changes of "list"  
count() in csv/list  
Sum a list of numbers in any length  
Convert a list to a matrix  
Convert/change the case of all letters/word into uppercase (capital) or lowercase in a list of strings  
Find duplicate items in a list  
Form a list of strings from an old string with all the 6 digits by removing all special characters or spaces  
Plot a figure with a colored arrow between text lines/steps  
__add__, __call__, __contains__, __delitem__, __delattr__, __eq__, __enter__, __ge__, __getattribute__, __getnewargs__, __getattr__, __getitem__, __gt__, __hash__, __reduce__, __iadd__, __imul__, __init_subclass__, __index__, __int__, __invert__, __new__, __neg__, __reduce_ex__, __reversed__, __rmul__, __radd__, __rand__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __ror__, __round__,__rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __dir__, __doc__, __divmod__, __iter__, __le__, __lt__, __len__, __ne__, __repr__, __setattr__, __setitem__, __sizeof__, __lshift__, __sub__, __subclasshook__, __str__  
Reverse a list  
locals()  
Convert a sentence/text to a list  
Add letter/commas/numbers/characters to the end/beginning of strings in a list  
Replace the lines between two lines “xx” and “yy” in a text file with new lines  
Remove string 0s from the back/end of a list until non-zero values  
Duplicate/repeat the same words/elements in a string/list  
Extract the last column as subdataframe  
Optimizing failure analysis processes in semiconductor labs using machine learning  
Hide x-axis tick labels (only show some labels) where x values are under certain conditions  
Font size of tick labels in plot  
Cheatsheet of list  
Set logarithmic scale (exponential) for y-axis in plots  
Populate the table with logarithmic format in pptx  
PermissionError (E.g. file cannot be written when it is open/locked)  
Output data if any or same element in a string are in two lists  
HTTP(Hypertext Transfer Protocol)/URL (Uniform Resource Locator)  
.Label()  
Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values  
.press(Button.left)  
.release(Button.left)  
.click(Button.left, x): x clicks of mouse.  
hotkey('l')  
hotkey('left')  
.topleft  
.top/.bottom/.left/.right  
'xyz'.isalpha(): Check if string is alphabet (letter, or one type of character)  
import logging  
.format.line  
os.path.lexists()  
.set_xticklabels()  
Loop through a string from left to right  
Reference list items by position code1, code2
lineterminator CSV: string (length 1), default None, Character to break file into lines. Only valid with C parser
lineterminator CSV: code.
left_index=, right_index= CSV: (code)
.loc[] and .iloc[] Introduction
line CSV: Print line by line from a CSV file: code.
Locators
driver.get("")  
   
driver.findElement(By.linkText("NextPage")).click();
driver.findElement(By.id("")).sendKeys("")      
driver.findElement(By.xpath("")).click()
driver.findElement(By.xpath("").sendKeys("")
driver.findElement(By.id(""))
chooseFile.sendKeys("")
driver.findElement(By.name("q")).sendKeys ("")
dom =document.getElementById("")
driver.FindElement(By.CssSelector(""))
driver.findElement(By.className(""))
driver.findElement(By.tagName("select")).Click()
driver.findElement(By.partialLinkText(" NextP")).click()
 
.locateOnScreen() .locateOnScreen('looksLikeThis.png') returns (left, top, width, height) on the image which the screenshot is taken from. Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. This function mostly is useless. Introduction. (code)
top and left for locateOnScreen() Introduction
.locateCenterOnScreen() Uses pyscreeze. x, y = MySearch_img to get the x- and y-coordinates of centers of the feature. Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. Introduction. (code)
.locateAllOnScreen() Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. Introduction. (code)
   
lambda Introduction. Lambda functions can be used in places where we expect variables. code.
class watchdog.events.LoggingEventHandler Bases: watchdog.events.FileSystemEventHandler
Logs all the events captured.
class watchdog.events.LoggingEventHandler

Bases: watchdog.events.FileSystemEventHandler
Logs all the events captured.

Insert/add text or line Insert text or new text lines to a specific position, at the end of a line, in a file. Introduction
math.log(x,y) Returns the natural logarithm of x to base y.
math.log2(x) Returns the base-2 logarithm of x.
lower() Returns the string by converting all the characters of the string to lower case. Introduction. code.
LoggingEventHandler (code)
len() Is a function to get the length of a collection. It returns the length of the string. Plus: The last character of a given string can also be printed. Example code. copy method. all kinds of len().
   
LightGBM Is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages:
Faster training speed and higher efficiency.
Lower memory usage.
Better accuracy.
Support of parallel, distributed, and GPU learning.
Capable of handling large-scale data.
scipy.linalg Linear algebra routines and matrix decompositions extending beyond those provided in numpy.linalg. scipy.linalg contains all the functions that are in numpy.linalg. Additionally, scipy.linalg also has some other advanced functions that are not in numpy.linalg. scipy.linalg operations can be applied equally to numpy.matrix or to 2D numpy.ndarray objects. code.
scipy.linalg.norm This function is able to return one of eight different matrix norms, or one of an infinite number of vector norms, depending on the value of the ord parameter. For tensors with rank different from 1 or 2, only ord=None is supported. It is for an old release of SciPy (version 0.14.0).
from scipy.linalg import norm code
scipy.linalg.block_diag Create a block diagonal matrix from the provided arrays.
scipy.linalg.circulant Create a circulant matrix.
scipy.linalg.companion Create a companion matrix.
scipy.linalg.convolution_matrix Create a convolution matrix.
scipy.linalg.dft Create a discrete Fourier transform matrix.
scipy.linalg.fiedler Create a symmetric Fiedler matrix.
scipy.linalg.fiedler_companion Create a Fiedler companion matrix.
scipy.linalg.hadamard Create an Hadamard matrix.
scipy.linalg.hankel Create a Hankel matrix.
scipy.linalg.helmert Create a Helmert matrix.
scipy.linalg.hilbert Create a Hilbert matrix.
scipy.linalg.invhilbert Create the inverse of a Hilbert matrix.
scipy.linalg.leslie Create a Leslie matrix.
scipy.linalg.pascal Create a Pascal matrix.
scipy.linalg.invpascal Create the inverse of a Pascal matrix.
scipy.linalg.toeplitz Create a Toeplitz matrix.
linalg.multi_dot(arrays, *[, out]) Compute the dot product of two or more arrays in a single function call, while automatically selecting the fastest evaluation order.
linalg.matrix_power(a, n) Raise a square matrix to the (integer) power n.
linalg.cholesky(a) Cholesky decomposition.
linalg.qr(a[, mode]) Compute the qr factorization of a matrix.
linalg.svd(a[, full_matrices, compute_uv, …]) Singular Value Decomposition.
linalg.norm(x[, ord, axis, keepdims]) Matrix or vector norm.
linalg.cond(x[, p]) Compute the condition number of a matrix.
linalg.det(a) Compute the determinant of an array.
linalg.matrix_rank(M[, tol, hermitian]) Return matrix rank of array using SVD method
linalg.slogdet(a) Compute the sign and (natural) logarithm of the determinant of an array.
linalg.solve(a, b) Solve a linear matrix equation, or system of linear scalar equations.
linalg.tensorsolve(a, b[, axes]) Solve the tensor equation a x = b for x.
linalg.lstsq(a, b[, rcond]) Return the least-squares solution to a linear matrix equation.
linalg.inv(a) Compute the (multiplicative) inverse of a matrix. code.
linalg.pinv(a[, rcond, hermitian]) Compute the (Moore-Penrose) pseudo-inverse of a matrix.
linalg.tensorinv(a[, ind]) Compute the ‘inverse’ of an N-dimensional array.
from scipy.linalg import eigh Print "selected eigenvalues" and "complex ndarray": code.
linalg.LinAlgError Generic Python-exception-derived object raised by linalg functions.
skimage.measure.label(input[, neighbors, ...]) Label connected regions of an integer array.
skimage.measure.LineModel() Total least squares estimator for 2D lines.
skimage.measure.LineModelND() Total least squares estimator for N-dimensional lines.
smtp.login() code.
(min_val, max_val, min_loc, max_loc) = cv2.minMaxLoc() Returns the max and min intensity values as an array
that includes the location of these intensities. Takes the correlation result and returns a 4-tuple which includes the minimum correlation value, the maximum correlation value, the (x, y)-coordinate of the minimum value, and the (x, y)-coordinate of the maximum value, respectively. Max_Val is the location with the highest intensity in the image, corresponding to the best matching input image with regard to the defined template.. code. code. code.
cv2.line() Draw a line in a image. (code). code
   
label Label for plot legend
logy Use logarithmic scaling on the y-axis
legend Add a subplot legend (True by default)
'ls' code.
np.linspace() Introduction. General, code. code.
numpy.linalg This function is able to return one of eight different matrix norms, or one of an infinite number of vector norms, depending on the value of the ord parameter. (code).