Electron microscopy
Python Automation and Machine Learning for ICs: Chapter A
- Python Automation and Machine Learning for ICs -
- An Online Book: Python Automation and Machine Learning for ICs by Yougui Liao -
Python Automation and Machine Learning for ICs                                                       http://www.globalsino.com/ICs/        

Table of Contents/Index 
Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix



LASSO (Least Absolute Shrinkage and Selection Operator) Introduction
Software/interface/API (Application Programming Interface) used in data science and machine learning Introduction
Comparison between CNN, CNN with Attention and Autoencoder Introduction
Attention-Guided Neural Network (AGNN)  Introduction
Memory resources in Apache Spark applications Introduction
Clusters (Kubernetes, Apache Mesos, Spark Standalone, Apache Hadoop YARN) in Apache Spark Introduction
Comparison between Clouds (Amazon, IBM, Google ...) Introduction
Deploy modes for driver process in Apache Spark: client mode and cluster mode Introduction
Apache Spark applications to a Kubernetes cluster Introduction
Run an Apache Spark application  Introduction
Apache Spark architecture Introduction
Aggregating data in SparkSQL Introduction
Apache Data Ingestion Frameworks (ADIF) for CSV to DataFrame Conversion  Introduction
Generate automated reports using Python Introduction
Apache Flink Introduction
Apache Kudu  Introduction
Apache Impala  Introduction
Personalizing applications with ML Introduction
Principles of ethical and responsible ML (selection bias, confirmation bias, automation bias, model fairness) Introduction
Data labeling and annotation in supervised ML Introduction
Example of Vertex AutoML Vision Introduction
Parametric and non-parametric learning algorithms Introduction
Machine learning algorithms Introduction
Directed Acyclic Graph (DAG) Introduction
Spark Core of Apache Spark Introduction
Comparisons among SparkML, MLlib, and AutoML Introduction
Apache HBase Introduction
Analyzing Data in Hadoop (HDFS, YARN, Apache Hive, Pig, HBase, Spark) Introduction
Hadoop MapReduce used by Google, Netflix, Amazon and Machine Learning Introduction
Apache Spark Introduction
Apache hadoop and hadoop ecosystem Introduction
Mathematical algorithms of artificial intelligence for semiconductor industry Introduction
Analytics and Technology Automation (ATA) Introduction
Calculation of Principal Component Analysis (PCA) Introduction
Principal Component Analysis (PCA) versus Uniform Manifold Approximation and Projection (UMAP) Introduction
Uniform Manifold Approximation and Projection (UMAP) Introduction
QuAM (Question-Answering Machine) Introduction
Attention in ML Introduction
AutoML (Automated Machine Learning) versus Generative AI Introduction
Virtual reality (VR), augmented reality (AR), and mixed reality (MR) Introduction
Exploratory data analysis (EDA) Introduction
Default mutable argument Introduction
Abstraction in Python programming Introduction
Computer hardware architecture Introduction
Labor cost of data analysis with and without automation and ML techniques Introduction
Function approximation Introduction
L1 Loss (Absolute Loss or Mean Absolute Error (MAE)) Introduction
Self-attention in ML Introduction
Plot with letters/words as x-/y-axis Introduction
Maintaining arc-consistency Introduction
Arc consistency Introduction
Simulated Annealing Introduction
Markov assumption Introduction
Sampling Methods for Approximate Inference Introduction
Approximate inference Introduction
Knowledge-based agents Introduction
A* (A-star) Search Introduction
Action in ML Introduction
Agent in ML Introduction
State-action rewards in Markov Decision Process (MDP) Introduction
Q-Learning with Function Approximation (Deep Q-Network - DQN) Introduction
Algorithm sensitivity to zero values Introduction
Credit assignment problem in reinforcement learning Introduction
Nonlinear extensions of Independent Component Analysis (ICA) Introduction
Mixture of Gaussians (MoG) versus Factor Analysis (FA) Introduction
Factor Analysis Model Introduction
Plot pixel intensity (histogram) along a line (row/column/x-axis/y-axis) of an image Introduction
Isolation forest algorithm Introduction
Anomaly detection Introduction
Learning algorithm (ensemble learning) and pipeline Introduction
Example of building robot (self-driving) systems with automated ML: helicopter Introduction
Weighted accuracy in ML Introduction
Close file after reading a file: avoid file locking Introduction
Logistic regression as a one-neuron/single-layer neural network (connection between linear & activation parts) Introduction
Batch Gradient Descent (BGD), Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, Batch Stochastic Gradient Descent, Momentum, (Adagrad, Adadelta, RMSprop), and Adam (Adaptive Moment Estimation) Introduction
Model = architecture + parameters Introduction
Neuron (= linear + activation) introduction Introduction
AdaBoost (Adaptive Boosting) Model Introduction
Experiences of developing machine learning algorithms Introduction
Bagging (Bootstrap Aggregating) Introduction
Additive structure/additive model in ML Introduction
Comparisons among Manual Search, Vertex Vizier, AutoML and Early stopping on google cloud Introduction
Difference between estimation and approximation errors Introduction
Example of ML debugging: Anti-Spam Introduction
Approximation error Introduction
Assumptions related to distribution of data in ML Introduction
Probably Approximately Correct (PAC) learning Introduction
CIFAR (Canadian Institute for Advanced Research) (CIFAR-10 and CIFAR-100) Introduction
Minimum A Priori (MAP) Introduction
Mean Average Precision (MAP) Introduction
Maximum A Posteriori (MAP) Introduction
Knuth-Morris-Pratt (KMP) algorithm (a string-searching algorithm) Introduction
Apple Neural Engine (ANE) Introduction
Custom AI/ML chips/ICs Introduction
Laplace smoothing/Laplace correction/add-one smoothing Introduction
Comparisons among artificial intelligence (AI), machine learning (ML) and quantum machine learning (QML) Introduction
Discriminative algorithms versus generative models Introduction
Artificial Neural Networks (ANNs) Introduction
Discriminative algorithms/discriminative models Introduction
Comparison between mean squared error (MSE), absolute error (L1 Loss) and fourth-power loss
Perceptron algorithm Introduction
Perceptron algorithm and logistic regression Introduction
Classification and algorithms for classification Introduction
Comparison between L1 Regularization and L1 Loss (absolute loss or mean absolute error (MAE)) Introduction
Autoencoders Introduction
Parametric learning algorithm Introduction
Non-parametric learning algorithm Introduction
Iterative algorithms Introduction
Algorithms for directly finding the global optimum Introduction
Learning Algorithm (estimator) Introduction
Linear regression and its algorithm Introduction
Gradient descent algorithm (for updating θ) Introduction
Actual Probability of Deviation Introduction
Send a variable from one script (back) to another script with a function Introduction
Rapid Automatic Keyword Extraction (RAKE) Introduction
Question answering retrieval Introduction
Leetcode for Google/Amazon Introduction
Check/find/get a file name/all file names or the last folder name (e.g. from a path/directory) Introduction
Compare dates (x days after or before a date), and difference between two dates in days Introduction
Multinomial Naive Bayes algorithm Introduction
Feature analysis/feature importance analysis
Scalability in automation and machine learning projects Introduction
Autonomous vehicles/cars and machine learning Introduction
Train/Test versus Model Accuracy Introduction
RSquare (R^2) versus RASE (Root Average Squared Error) Introduction
Adjusted R-squared values of two or more regression models Introduction
Nonasymptotic versus asymptotic analysis Introduction
any() Introduction
Add an item to a dictionary Introduction
Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc: localhost, insert rows, update, count updated, delete rows, comparision between extract data by Python and SQL itself Introduction
CycleGAN (Cycle-Consistent Adversarial Networks) Introduction
Class Activation Mapping (CAM) Introduction
AI/machine learning algorism for text analysis Introduction
Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc Introduction
Check if all the (and how many, length of a string) characters in the text are digits/numbers Introduction
Calculating the area fraction of each circle overlapped by a square grid and build wafer map Introduction
Extract any substrings with any pattern (e.g. dot (.)) Introduction
Convert all elements of specific column or in entire dataframe into strings Introduction
Trick: pd.concat() for merging/adding (two) columns Introduction
Automatically run a file in an application Introduction
Click a menus of an application Introduction
Different behavior of automation execution (e.g. pyautogui) locally or remotely through internet
Trick: generic code/script templates for complex automation Introduction
Generative Adversarial Network (GAN) technologies Introduction
Asymptotic analysis Introduction
Well-specified case of "asymptotic approach" Introduction
Compare (pattern/ratio of) two different columns, check whether column values match in DataFrame Introduction
Check whether one column contains number only and another column contains letters only or mixture of numbers and letters in DataFrame Introduction
Check the difference between two columns in DataFrame Introduction
Check if a file exists again (double check) Introduction
ArithmeticError (arithmeticException) Introduction
Avoid duplicates when creating text file Introduction
Automatically restart script execution after it breaks/fails/error Introduction
Call and run another script in a different/any (parent or children) directory/path/subfolder from a script Introduction
Modify HTML webpage (e.g. with graph network by adding/inserting text/hyperlink in) Introduction
Convert/change the case of all letters/word into uppercase (capital) or lowercase in a list of strings Introduction
Check if a variable does exist/is assigned/defined Introduction
Check all the imported/current modules/libraries Introduction
Plot multiple datasets on the same scatter graph with different x- and y-axis values Introduction
Write special/certain rows (row-by-row) of one csv file to another csv file Introduction
y axis values are not ordered (disordered) Introduction
matplotlib.pyplot to plot/generate images (with axis/colored text or annotation) Introduction
Avoid two or multiple plots being wrongly/incorrectly/unnecessarily mixed/overlap Introduction
Check if one list is subset of another Introduction
Remove/reload/unload (all) imported module/function/script Introduction
Plot graph/figure/image from CSV file/DataFrame by removing/hiding blank/empty cells with axis range (plt.xlim()) Introduction
Copy a file or all files (with os.mkdir) to save to somewhere (create a directory first if it does not exist) Introduction
Separately plot data into the same graph/figure/image from different csv files for each category (import multiple CSV files and concatenate into one DataFrame): append row-by-row or column-by-column Introduction
sort_values(by=... ascending/descending order) Introduction
Plot multiple images on the same figure by hiding x- and y-labels on axis Introduction
Convert between numpy array and string Introduction
Continue script execution no matter whether some try fails or not (finally, always executes) or else Introduction
Plot a figure with a colored arrow between text lines/steps Introduction
__add__, __call__, __contains__, __delitem__, __delattr__, __eq__, __enter__, __ge__, __getattribute__, __getnewargs__, __getattr__, __getitem__, __gt__, __hash__, __reduce__, __iadd__, __imul__, __init_subclass__, __index__, __int__, __invert__, __new__, __neg__, __reduce_ex__, __reversed__, __rmul__, __radd__, __rand__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __ror__, __round__,__rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __dir__, __doc__, __divmod__, __iter__, __le__, __lt__, __len__, __ne__, __repr__, __setattr__, __setitem__, __sizeof__, __lshift__, __sub__, __subclasshook__, __str__ Introduction
Get the date and time (a past date) of N days ago Introduction
Check if all/any values are true or false in a range of data Introduction
Merge/append rows/columns of a csv file into an old csv file if the rows/columns are not in the old csv file Introduction
Add letter/commas/numbers/characters to the end/beginning of strings in a list Introduction
Plot workflow: Create new empty column in DataFrameMove the cells in a column to another column under certain conditionSelect specific columns for scatter plot
Analytics and Technology Automation (ATA) Introduction
Aggregate duplicates in columns of data Introduction
.apply(pd.Series) Introduction
.apply(tuple, axis=1) Introduction
Output data if any or same element in a string are in two lists  Introduction
Overcoming automation challenges and forward-looking suggestions Introduction
Aggregation functions in data manipulation and database queries Introduction

accuracy_score() (code)
Accuracy/precision test Introduction
.axes[0] and .axes [1] CSV: for csv. (code)
alert() alert(text='', tilte='', button='Ok'). (code)
3 types of logical operators/booleans Represent one of two values: True or False. code. code. code.
and    /  &     The output is 'true' or 'false', when both the conditions are 'true' or 'false'. Both/all conditions must be true for the statement to be true. "and" is binary, but not is unary. Introduction. Example code.
import ... as ...  
.append() Introduction. Appends an element to the end of the list. copy method. Add a list/item at the end of the list.
A = A[ : :-1] Reverse an array
sys.argv Introduction
from ... import ... as ...  
import ... as ... E.g. code for opening an image. , code2
from tkinter.filedialog import askopenfilename code. code.
with ... as ... code. code.
.shapes.add_textbox (code)
slides.add_slide() (code)
.shapes.add_picture() (code)
.add_run() (code)
assert() code.
watchdog.observers.api: Immutables  
class watchdog.observers.api.ObservedWatch(path, recursive) Parameters: path – Path string. recursive – True if watch is recursive; False otherwise.
class watchdog.observers.api.EventQueue(maxsize=0)
class watchdog.observers.api.EventEmitter(event_queue, watch, timeout=1)
class watchdog.observers.api.EventDispatcher(timeout=1)
class watchdog.observers.api.BaseObserver(emitter_class, timeout=1)
watchdog.observers.api.EventDispatcher Base observer.
add_handler_for_watch(event_handler, watch) Adds a handler for the given watch.
Parameters: event_handler (watchdog.events.FileSystemEventHandler or a subclass) – An event handler instance that has appropriate event handling methods which will be called by the observer in response to file system events.
watch (An instance of ObservedWatch or a subclass of ObservedWatch) – The watch to add a handler for.
watchdog.observers.api.BaseObserver Platform-independent observer that polls a directory to detect file system changes.
watchdog.observers.api.BaseObserver File system independent observer that polls a directory to detect changes.
math.acos() Returns the arc cosine of x in radians.
math.atan() Returns the arc tangent of x, in radians.
math.atan2() Returns atan(y / x), in radians.
with mss.mss() as sct (code)
Multiple assignment E.g. a, b = 4, 3
*args "*" is the unpacking operator, which is not a list but a tuple. *args allows you to pass multiple, varying arguments or keyword arguments to a function. Note that args is just a name. You’re not required to use the name args. You can choose any name that you prefer. Introduction.
asctime() code
auto-py-to-exe (code)
.axis() Introduction
__annotations__/dict name/type: parameter and return annotations
skimage.measure.approximate_polygon(coords, ...) Approximate a polygonal chain with the specified tolerance.
asyncio Comparison between multithreading, multiprocessing and asyncio at page4797.
cv2.add code.
cv2.arrowedLine() cv2.arrowedLine(image, start_point, end_point, color[, thickness[, line_type[, shift[, tipLength]]]]) is used to draw arrow segment pointing from the start point to the end point. The parameters of the cv2.arrowedLine function are the same as those for cv2.line. code.
ax matplotlib subplot object to plot on; if nothing passed, uses active matplotlib subplot
alpha The plot fill opacity (from 0 to 1)
arrowprops= code.
.annotate() code.
arrowstyle code.
.argwhere() (code)
.array General, number array . for image. code. in csv.
.arange() Introduction. Can change with an increment. General, code. code.
hotkey('a') Introduction
alt Introduction
.active (code). (code).
.activate() Activate a window: with the active cursor in the window and the window is brought to the most front on the monitor. (code)
'xyz'.isalpha() Check if string is alphabet (letter, or one type of character)
auto_size_text=True (code).

Right-justify/align them so that they take up the same amount of space, whether the coordinate has one, two, three, or four letters or digits. (code).

add_connector() shapes.add_connector(MSO_CONNECTOR.STRAIGHT, Begin_x, Begin_y, End_x, End_y) (code).
.add_series() (code)
.add_chart() (code)
Comparison between *, .extend((), append(), =, ==, .copy() and copy.copy() for "list": changes of "list" Introduction
Image rotation in pptx (.rotation=) Introduction. E.g. Rotate an image around the top-left corner as an axis in a pptx file with the same lateral dimension
.at[] Introduction
os.path.abspath() (code).
Variable length arguments (*args and **kwargs) Introduction
numpy.all/np.all Introduction
.DataFrame() Introduction. .drop(), index, columns, axes, dtypes, size, shape, ndim, empty, T, values
Assert and assertion Introduction
Vertex AI Feature Store Introduction
Data Augmentation Introduction
Keyword arguments Introduction
Automation with programming
Reasons of automation and how to start Introduction
Generic (or generalized) robot programming (GRP) Introduction
Graphical user interface (GUI) Introduction
API (Application Programming Interface), e.g. weather, temperature Introduction
Keyboard and mouse automation  
  Functions of mouse Introduction
  Codes: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, pynput, ctypes, uiautomation and Sikuli) Introduction
    Principle and troubleshooting: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli) Introduction
  Automatically review, scroll, click webpage and its link Introduction
  Copy text into clipboard and then you can paste it a webpage, text/txt, word or powerpoint file automatically (code)
Automatically fill contexts into or select options on webpage Introduction
AutoML Introduction
Automation of EELS data extraction Introduction
Ranking/most popular automation testing tools Introduction
Ranking/most popular IT automation software tools Introduction
Comparison between Python, Blue Prism, UiPath, Automation Anywhere Introduction
Automated defect scanning in wafer map Introduction
Robots and Robotic Process Automation (RPA) Introduction
  Python Introduction
    X Reasons of using Python for automation Introduction
    X Examples of UiPath applications Introduction
    X Installation of UiPath and its packages and creation of new projects Introduction
    X Comparison between Python, Blue Prism, UiPath, Automation Anywhere Introduction
3 ways to create a Keras model with TensorFlow  
  Sequential API to create a Keras model with TensorFlow Introduction
  Functional API to create a Keras model with TensorFlow Introduction
  Model Subclassing to create a Keras model with TensorFlow Introduction
tf.data API Introduction
tf.keras API Introduction
Add padding/black/colored edge to images Introduction
except ... as -- Exception Introduction
Evaluation of accuracy in machine learning process Introduction
Hide/turn on/off axes/axis on matplotlib Introduction
Adam optimization algorithm Introduction
Activation functions in machine learning Introduction
Analysis process in ML Introduction
Linear algebra Introduction
Vertex AI Introduction
.astype() (Code)
after_run function Introduction
except AttributeError Introduction
TensorFlow APIs Introduction
Axis/dimension of tensor Introduction
adapt() Introduction
.as_default() Introduction
Table of applications of Python and its libraries Introduction
Open an application window through search at start Introduction
Add a new slide into an existing ppt or a created ppt file Introduction
Launch the existing opened application if there is or start a new one if there is not Introduction
Open and close any type of files with default programs/apps (e.g. word, excel, dm3, dm4, Digital Micrograph, powerpoint, internet explorer, chrome, and so on) in windows Introduction
Copy and apply formatting in Word and PowerPoint Introduction
Move the active window to make space for other apps Introduction
Go to the pointed tab on an app Introduction
Convert capital alphabet letters/characters to number Introduction
Minimize/maximize/restore/activate/resize/move/close Window objects Introduction
Bring/activate an application to most front/foreground Introduction
Typical training setup in Artificial intelligence (AI) Introduction
matplotlib.pyplot axis/text color Introduction
Convert a CSV file to an image with one column and another column as x-axis and y-axis, respectively. code.
Calculator of length accuracy in 3D structure Introduction
Swap the order of arguments in a function Introduction
Copy text into clipboard and then you can paste it anywhere Introduction
Calculate/pass the arbitrary (any) number of variables or input arguments Introduction
Access variable inside and outside a function Introduction
Access elements in a dictionary and subdictionary Introduction
Access elements in a list and sublist Introduction
Numpy: Access the element at the second row, the third entry, access a specific row or a column, access some elements (submatrix), or replace/modify an element in the array, print a transfer of an array, access array under conditions or filtering Introduction
.T (Transfer of array in Python) Introduction
Calculations in DataFrame: Add a column, calculate for a new column, delete a column, all the rows with values greater than 30 in "Score A" column Introduction
Markers (e.g. color cross, scatter, and circles) at specific coordinates with x- and y-axis Matplotlib
Keyword search function/check whether or not a string is within another string (a space is included as a string character) Introduction
Draw an arrow segment pointing from the start point to the end point in an image. code.
Draw lines manually and then label them with arrows code
Prevent other applications to modify the content until other Python script runs code.
Show/open images in any image viewer code, code.
Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values CSV: Introduction. code.
Sum two images after resizing them code

Create images with global, adaptive mean, adaptive Gaussian, binary, trunc, Tozero, and tozero thresholds.

Draw lines, elbow connectors and arrows in a ppt Introduction
Remove an/all/duplicate item(s)/elements from a list Introduction
Comparison between iteration algorithm and recursive algorithm: a function repeat itself Introduction
Add a new slide into an existing ppt, or work on existing slides Introduction
Work with (e.g. open) all/every files and subfolders/subdirectory in a folder (does not include any files or include all files in subfolders); get a list of all files and directories in the same folder where the Python script file is. Introduction
Insert all the images into a ppt file (one image per slide) (Introduction)
Mean (average, .mean())/.sum()/maximum(.max())/minimum(.min())/number of non-null values(.count())/.median()/variance(.var())/standard deviation(.std()) Introduction
Add markers on a map Introduction
Add/insert a column into an existing csv file Introduction
Call and then run your own functions and modules in different/other Python files; Python run another Python script
Run multiple Python files/scripts one after another Introduction
Monitor specific new files, and execute the file or another file (and then restart the monitoring program itself to continue its monitoring by standby, with watchdog; or observer runs in the background and pass the value of obtained events to a/another function call) Introduction
Watchdog for monitoring specific file or files with specific extension, and then run another file from watchdog Introduction
Launch script from another script using subprocess.run/subprocess.call Introduction
Simple ways to execute another python file when a new file has been uploaded Introduction
Move(remove) all files from original folder in a directory to a new directory Introduction
List all files and directories which has specific files or files with specific extensions Introduction
top and left for pptx (e.g. align the top-left corner of the image to the center of the slide no matter how the size of the images changes) Introduction
Subtract/minus one image from another image Introduction
Find contours in an image and their areas and coordinates Introduction
Global access to a local variable inside a function/class from outside of the function/class externally
if not hasattr/if hasattr (attribute) Introduction
Limit event/action numbers in the event List, then stop Introduction
Rotate (alignment) an image by line along the x- or y-axis Introduction
Shift/translate image along x-axis/y-axis Introduction
Modify a list (e.g. add/insert an item between items) Introduction
Artificial intelligence (AI) Introduction
Applications of artificial intelligence/machine learning in industry Introduction
Manual analysis of data Introduction
Ranking/most popular programming languages for data analysts Introduction
Write/save content to a text file/append a string into a text file. Introduction
Generate text file with the bank of collecting all words, characters and strings from news Introduction
Add/insert text to an image Introduction
Pass variables between functions/from one to another Introduction
Convolutional Autoencoder (CAE) Introduction
Apply a formatting function to all cells in a DataFrame Introduction
Hide x-axis tick labels (only show some labels) where x values are under certain conditions Introduction