Electron microscopy
Python Automation and Machine Learning for ICs: Chapter I
- Python Automation and Machine Learning for ICs -
- An Online Book: Python Automation and Machine Learning for ICs by Yougui Liao -
Python Automation and Machine Learning for ICs                                                        http://www.globalsino.com/ICs/        

Table of Contents/Index 
Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix



Spark installation Introduction
Data Ingestion in Apache Spark  Introduction
IoU (Intersection over Union) Introduction
Apache Spark User Interface (UI)  Introduction
Comparison between Clouds (Amazon, IBM, Google ...) Introduction
Skills Network Labs (SN Labs, IBM) Introduction
Apache Spark on IBM Cloud Introduction
Apache Data Ingestion Frameworks (ADIF) for CSV to DataFrame Conversion  Introduction
Apache Impala  Introduction
Feature Selection: Chi Square to select dependent and independent variables Introduction
Image generation with ML Introduction
Software/interface/API (Application Programming Interface) used in data science and machine learning Introduction
Image classification with ML Introduction
Identifying the business value of using ML Introduction
Ingesting data in Hadoop Introduction
Clean clipboard and/or check if clipboard is empty, text or image Introduction
Add padding/black/colored edge to images Introduction
Mirror/reflect image from left to right/from top to bottom Introduction
Folder for saving images from screenshot obtained by Snipping Tool on windows Introduction
Locate/find the center/coordinates of a bright (maximum/highest intensity) spot in an image & find nearest white pixel to a given/specifical pixel location on an binary image Introduction
Line detection on image Introduction
Retry a number of times/infinite retrying before exception or fail Introduction
Examples of matplotlib (image/data) visualizations Introduction
Save the image in clipboard to an image file Introduction
Analysis of ROI (region of interest)/portion of an image Introduction
issubclass Introduction
.isin() Introduction
Find duplicate items in a list Introduction
Feature ingestion Introduction
except ImportError Introduction
input_fn Introduction
tf.keras.layers.IntegerLookup Introduction

Match on images to find and to highlight unsimilar (threshold=0) to identical (threshold=1) regions of an image that match a template with a cross-correlation method


Match on images to find and to highlight unsimilar (threshold=0) to identical (threshold=1) regions of an image that match a template with a cross-correlation method

Coordinate at center of features, e.g. images Introduction
Infinite loops (e.g. stop infinite cycling of opening the same images) Introduction
Work (read, write, insert and delete rows and columns, and merge and unmerge cells) in Excel sheets Introduction
Work with/insert images into excel sheets Introduction
Work with/insert images into excel sheets Introduction
(Single and multiple enter/input) box for pop-up window Introduction

Thresholding with Match Template: Match on images to find and to highlight unsimilar (threshold=0) to identical (threshold=1) regions of an image that match a template with a cross-correlation method

Term Frequency-Inverse Document Frequency (TF-IDF) Introduction
Print colored text in Python IDLE's terminal Introduction
Copy text into clipboard and then use it immediately (one time clipboard) Introduction
API (Application Programming Interface), e.g. weather, temperature Introduction
Select/input a folder/directory/path for later to be called to use Introduction
Image matching with cross correlation and overlap of template edge. In this matching process, Normalized cross-correlation with those edge images is performed. code.
Cross correlation between two images code
Cross correlation between two images in any sizes. Multiscaling is used to avoid the issue caused by the different sizes of the template and original image, in order to find match in a original image, namely, the size of template is larger than the original image. code
Image similarity search code, introduction
Immediately invoked function expression code
Convert CSV to images, row by row, with pixel values: each row is an image code.
Convert a CSV file to an image with one column and another column as x-axis and y-axis, respectively. code.
Get a specific output for every input code.
Calibrate and put a scale bar, and draw a line segment on an image Introduction
Matrix conversion to image image.
Calculate/pass the arbitrary (any) number of variables or input arguments Introduction
Access variable inside and outside a function Introduction
Slicing and indexing in string
Popup windows with inputs Introduction
Draw an arrow segment pointing from the start point to the end point in an image. code.
Draw a line in a image code
Open webpages in internet browsers (Chrome, Microsoft edge, IE browser) Introduction
Open an image to file from URL (webpage), then it can be saved in PC code, code. open with color changed.
Open any files, e.g. text (.txt), image files, and so on. .txt file will be opened in Python program, e.g. IDLE shell. code
Copy images to a different folder/Save an image in a new folder code, code. code. code.
Set the output image to zero everywhere except my mask (color filter), and display red, green, and blue (RGB) channels of an image. code, code.
Show/open images in any image viewer code, code.
Creates an image from an colored images after remove the grey components (image conversion from color to gray involved) code
Extract three blue, green, red images from a color image or grey image, or convert RGB (color) image into three blue, green, red images code

Change/convert a colored image to a grey image(, and then show pixel values).

cv2, cv2, cv2/skimage. cv2/skimage. PIL. matplotlib.
Compute the difference between two images by using Structural Similarity Index with "pip install --upgrade imutils" code. code
Subtract (minus) two images after resizing them code, code.
Sum two images after resizing them code

Create images with global, adaptive mean, adaptive Gaussian, binary, trunc, Tozero, and tozero thresholds.

Find edges of images code
Reference list items by position code1, code2
Remove an item/element from a list Introduction
Check if an item/element is in a list or not Introduction
Comparison between iteration algorithm and recursive algorithm: a function repeat itself Introduction
Randomly open an image/file in a specific folder Introduction
Insert all the images into a ppt file (one image per slide) (Introduction)
Get/list immediate subdirectories/subfolders Introduction
Replace/substitute a item in a list Introduction
Get/list immediate subdirectories/subfolders; get only the last part of a path/folder Introduction
Get maximum and minimum value of column and its index Introduction
Add/insert a column into an existing csv file Introduction
Check if a directory is empty, find the position/index of a particular file/folder in a file directory/path Introduction
Placement/position of Python import statements Introduction
Watchdog ignore/skip and take pattern in directory/path Introduction
Find common elements/items between two lists Introduction
Invert the contrasts of black and white images (Code)
Plot pixel intensity (histogram) along a line (row/column/x-axis/y-axis) of an image Introduction
Global access to a variable inside a function from outside of the function
if not hasattr/if hasattr (attribute) Introduction
Search text on an image Introduction
Search/extract/find text on an image Introduction
Take a screenshot using a mouse click and drag method. For instance, take a screenshot, and then insert the image and/or a text into a ppt file. Introduction
Resize and then sum/mix/overlap two images Introduction
Measure length/distance on an image w/o calibrated bar Introduction
Rotate (alignment) an image by line along the x- or y-axis Introduction
Get pixel location/coordinates on an image using mouse click/events Introduction
Crop/snip part of a image with definition by a pixel line Introduction
Shift/translate image along x-axis/y-axis Introduction
Modify a list (e.g. add/insert/remove an item between items, merge all items) Introduction
Find the file names of the images in a pptx Introduction
Applications of artificial intelligence/machine learning in industry Introduction
Libraries used to convert incident documents into numerical vectors Introduction
Ranking and votes of essential/most important skills for data analysts Introduction
Ranking/most popular IT automation software tools Introduction
Installation of UiPath and its packages and creation of new projects Introduction
Extract a mask from an image with a threshold Introduction
Save generated images Introduction
Human inspection of defects in wafer map Introduction
Global defects and local defects identified by defect denoising Introduction
Image segmentation ("clustering") in color Introduction
Non-zero pixel values from an image Introduction
Mask an image with a threshold or with a color as a threshold Introduction
Pixel values on specific pixel in an image Introduction
Add/insert text to an image Introduction
Methods of data and information visualization Introduction
Convert images between Cartesian and Polar forms Introduction
Draw circles/lines on images Introduction
Plot a list of x, y coordinates to an image Introduction
Store images in pandas dataframe column Introduction
Smooth images (make image blurry) Introduction
self and __init__ method in Class Introduction
Image convolution Introduction
Global import Introduction
Guess and check algorithm with a combination of a for loop and an if statement Introduction
Independence (independent events) versus dependence (dependent events) in ML Introduction
Sampling Methods for Approximate Inference Introduction
Approximate inference Introduction
Inference and Inference Rules Introduction
Informed search algorithms/heuristic search algorithms Introduction
Intrinsic motivation in ML Introduction
Policy iteration versus value iteration Introduction
Policy iteration Introduction
Value iteration Introduction
Nonlinear extensions of Independent Component Analysis (ICA) Introduction
Jensen's inequality Introduction
Isolation forest algorithm Introduction
Weight initialization Introduction
Variance of input data for ML Introduction
Normalizing input in neural network Introduction
Impact of machine learning on ICs (integrated circuits) Introduction
Edge detection of images in neural network Introduction
Dependently Identically Distributed/Correlated Identically Distributed Introduction
Gini impurity Introduction
Bayes error/Bayes risk/Bayes rate/irreducible error Introduction
CIFAR (Canadian Institute for Advanced Research) (CIFAR-10 and CIFAR-100) Introduction
Mathematical equations, formulas and inequalities used in ML Introduction
Custom AI/ML chips/ICs Introduction
Indicator function/Kronecker delta function Introduction
Independent and identically distributed (i.i.d./IID) Introduction
Iterative algorithms Introduction
Input data (sample and feature) in ML Introduction
Independent Component Analysis (ICA) Introduction
Bayesian Probability, Bayesian Statistics (Distribution Over a Distribution), versus Bayesian Inference Introduction
Infinite Hypothesis Class Introduction
Union Bound/Boole's Inequality Introduction
Finite Hypothesis Class versus Infinite Hypothesis Class Introduction
Complementary inequality Introduction
Concentration inequality Introduction
Hoeffding Inequality Introduction
IR (Information Retrieval) Introduction
send_keys() and its uploading images to webpage Introduction
Fréchet Inception Distance (FID) coefficient Introduction
Feature analysis/feature importance analysis/weight of feature
Feature importance for Multinomial Naive Bayes algorithm Introduction
Percentages of information received through different senses (eye, nose, ear and hand feeling) Introduction
Variance Inflation Factors (VIFs) Introduction
"Input space" in machine learning Introduction
Check if a variable is a number, string or integer Introduction
except ValueError (exception if it is not an integer) Introduction
Find common/different elements/items between two lists/sets Introduction
Last n days/weeks/months (.to_datetime(x), .set_index(y), .last(z), .reset_index(), and .max() in pandas) Introduction
Check all the imported/current modules/libraries Introduction
Combine multiple images into a single multi-page image or vice versa Introduction
Count the number of the pages in a single multi-page/frame image Introduction
Plot graph/figure/image from CSV file Introduction
Plot graph/figure/image from CSV file/DataFrame by removing/hiding blank/empty cells with axis range (plt.xlim()) Introduction
Plot data into the same graph/figure/image from different csv files Introduction
Plot images with certain image size and in color Introduction
Plot multiple images on the same figure by hiding x- and y-labels Introduction
Inside/outside edges/margins of plotted images Introduction
Multimodal text and image similarity Introduction
Calculating the area fraction of each circle overlapped (filled with color) by a square grid and build wafer map with integration Introduction
Check if two circles intersect or overlap Introduction
Convert strings to number (integers/float) Introduction
Select/skip columns by index in DataFrame without changing the DataFrame itself, and change the order of of the selected columns Introduction
Check if two lists are same/identical Introduction
Different behavior of automation execution (e.g. pyautogui) locally or remotely through internet
Module import and execution are skipped during script execution Introduction
Detection of tables from an image Introduction
Natural language inference Introduction
Multimodal text and image search Introduction
Add an item to a dictionary Introduction
IDLE (integrated development and learning environment) and integrated development environment (IDE) Introduction
Image embeddings Introduction
(Text and image) contrastive learning Introduction
Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc: localhost, insert rows, update, count updated, delete rows, comparision between extract data by Python and SQL itself Introduction
Insert data/row into SQL Database on SSMS with pyodbc Introduction
Remove/reload/unload an imported module/function/script Introduction
Modify HTML webpage (e.g. with graph network by adding/inserting text/hyperlink in) Introduction
Insert paragraphs of texts into Python script Introduction
Python modules to interact with the operating system (os, platform, subprocess, shutils, glob and sys) Introduction
Pyvis: An interactive geometric graph network/link/landscape Introduction
Inside/outside edges/margins of plotted images Introduction
Avoid two or multiple plots being wrongly/incorrectly/unnecessarily mixed/overlap Introduction
Exception LookupError (string index) Introduction
Iterate over rows in a DataFrame/read and print row by row (number of columns and rows, df.shape[0]/df.shape[1]) Introduction
from keyboard import is_pressed (Esc) Introduction
Extract the index of a string element in a list Introduction
igraph for clustering, network and tree Introduction
Input a sentence and then output a sentence based on a dictionary obtained from csv Introduction
Delete the column/row in a CSV file if they are empty or less than a number (or header/index only) Introduction
__init__.py Introduction
__init__: Is short for initialize and is capable of creating objects from within a class. As a method, it’s just a function that’s defined inside of a class. But it must have the specific name __init__ — that’s two underscores followed by init followed by two more underscores. It is sometimes spoken as “dunder init.” The dunder part is short for double underline. Introduction
__init__(stat, listdir, polling_interval=1): Parameters: stat – stat function. See os.stat for details. listdir – listdir function. See os.listdir for details. polling_interval (float) – interval in seconds between polling the file system.  
self and __init__ method in Class Introduction
__add__, __call__, __contains__, __delitem__, __delattr__, __eq__, __enter__, __ge__, __getattribute__, __getnewargs__, __getattr__, __getitem__, __gt__, __hash__, __reduce__, __iadd__, __imul__, __init_subclass__, __index__, __int__, __invert__, __new__, __neg__, __reduce_ex__, __reversed__, __rmul__, __radd__, __rand__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __ror__, __round__,__rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __dir__, __doc__, __divmod__, __iter__, __le__, __lt__, __len__, __ne__, __repr__, __setattr__, __setitem__, __sizeof__, __lshift__, __sub__, __subclasshook__, __str__ Introduction
Python IDLE restart during execution Introduction
iter() Introduction
Reindex a DataFrame Introduction
Save dynamic graph as a movie/video or split a movie to image frames Introduction



inplace=True CSV: Change the items permanently. (code).
index_col CSV: column number, column name, or list of column numbers/names, to use as the index (row labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there is one more data column than there are headers, in which case the first column is taken as the index.
iterator CSV: If True, return a TextFileReader to enable reading a file into memory piece by piece
index CSV: whether to write row (index) names (default True)

CSV: Column label(s) for index column(s) if desired. If None (default), and header and index are True, then the index names are used. (A sequence should be given if the DataFrame uses MultiIndex).

line_terminator CSV: Character sequence denoting line end (default ‘\n’)
import pandas as pd CSV: code.
index_col CSV: Skip column index. code. code. code.
.loc[] and .iloc[] Introduction
inplace=True (code)
info() (code)
Return the dictionary's key-value pairs. General.
if if <test condition>:
<block if the test condition is true>
An if can have any number of if's nested within. Introduction. code.
if-else if <test condition>:
<block if the test condition is true>
<block if the test condition is not true>
If there are multiple else statements, then the second else is taken along with the nearest if. code1, code2, code3, code4.
Difference between if and if-else Group 1: if and if-else.
if else ladder "else" covers all the cases other than "if".
if <test condition>:
< The task to be performed if the condition 1 is true>
elif <test 2>:
<The task to be performed if the condition 2 is true>
elif <test 3>:
<The task to be performed if the condition 3 is true>
<The task to be performed if none of the above condition is true>
It is used when there are multiple conditions and the outcomes decide the action. Here, a switch is used in the case where different conditions lead to different actions..
If ... elif ... Introduction. "elif" is a short word of "else if" and can have as many as "elif" as you need. code.
if ... in ... E.g. find a letter in a string: code.
if ... not (in) ... Introduction. code.
if not (code)
if not hasattr/if hasattr (attribute) Introduction
if ... is None code.
if() Introduction
import <library name> import the entire library.
import ... as ...  
insert() Insert a list/item in a list.
index index of row (the first column). Introduction
index=False, index=True Keep or remove index column. (code)
image The Image module provides a class with the same name which is used to represent a PIL image. The module also provides a number of factory functions, including functions to load images from files, and to create new images. E.g. Open, rotate, and display an image, code.
import The import statement allows you to make use of data and functionality from a module which isn’t the current one.
import sys The sys module provides various functions and variables that are used to manipulate different parts of the Python runtime environment. It allows operating on the interpreter as it provides access to the variables and functions that interact strongly with the interpreter. sys can be used to exit from the script as well. code
import textwrap code.
from PIL import ImageTk  
.iconbitmap() (code)
ipadx= (code)
Instance One unit of data plus code generated from a class as an instance of that class. Each instance of a class is also called an object
from pptx.util import Inches Inches, Emu, Cm, Mm, Pt, and Px are base class for length classes, providing properties for converting length values to convenient units. (code)
from PIL import Image (code)
.insert_picture() (code)
ignore_patterns = None (code)
ignore_directories=False (code)
ignore_directories (Read-only) True if directories should be ignored; False otherwise.
ignore_patterns (Read-only) Patterns to ignore matching event paths.
watchdog.observers.api: Immutables  
is_recursive Determines whether subdirectories are watched for the path.
ident Thread identifier of this thread or None if it has not been started.
This is a nonzero integer. See the thread.get_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited.
isAlive() Return whether the thread is alive.
This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads.


Return whether the thread is alive.
This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads.

inode(path) Returns an id for path.
isupper() Returns the boolean True value if the all the characters of the string are in upper case. code.
islower() Returns the boolean True value if the all the characters of the string are in lower case. code.
isspace() Returns the boolean True value if there is a space in a string. code.
integrate.quad Numerical integration. code.
'123'.isdigit() Check if all the characters in the text are digits: "False" or "True". Superscript and subscripts (usually written using unicode) are also considered digit characters. The roman numerals, currency numerators and fractions (usually written using unicode) are considered numeric characters but not digits. Introduction. code.
'xyz'.isalpha() Check if string is alphabet (letter, or one type of character)
import mss (code)
import mss.tools (code)
Insert/add text or line Insert text or new text lines to a specific position, at the end of a line, in a file. Introduction
input() The function is used to gather data from the user. Introduction. code1, code2, code3, file path . code5. code6.
math.isnan(x) Returns True if x is not a number, otherwise returns False.
if __name__ == '__main__' The processes starts reading the current file in order to execute the function specified. Without this clause, the import would first execute more process start calls, before getting to the function execution. code. code. code. code. Application example: run the page4853main3 program (as a module) through page4853main4 program. A similar example with defined functions is page4853main5.py. (code).
isinstance() Returns True, if the first argument is an instance of that class. Multiple classes can be also be checked at once. E.g. isinstance(4, (int, float)). (code).
Integer/fractions/round /decimal/digits/floating
It does not have any fractional part. Introduction. int: Example code.
int() Converts a float number or a string to an integer, cast the number. code1
Identity The identity, in Python, refers to the address and does not change.
import datetime as ... code
isnumeric() Check if all the characters in the text are numeric or not: Introduction. Check if all the characters in the text are numeric. "-6" and "1.9" are NOT considered numeric values, because all the characters in the string must be numeric, and the - and the . are not. code.
isalnum() Returns True if all characters in the string are alphanumeric (either alphabets or numbers). code.
.index() It is an inbuilt function in Python, which searches for a given element from the start of the list and returns the lowest index where the element appears. The index of the first location is 0. Syntax: list_name.index(element, start, end). Introduction. code1, code2
.index(sub[start, [end]]) Same as ".find()", but it gives Error message if it does not find. code.
image[:,:,0] The : is used to get all values. For example, image[:, :, 0] means get all rows, all columns, and the first (at index 0) color channel, which is the red channel. code.
image[:,:,1] code.
image[:,:,2] code.
scipy.integrate Numerical integration
scipy.linalg.invhilbert Create the inverse of a Hilbert matrix.
scipy.linalg.invpascal Create the inverse of a Pascal matrix.
inner(a, b) Inner product of two arrays.
linalg.inv(a) Compute the (multiplicative) inverse of a matrix. code.
from scipy.linalg import eigh Print "selected eigenvalues" and "complex ndarray": code.
io (I/O) Core tools for working with streams and provides Python’s main facilities for dealing with various types of I/O, with three main types of I/O: text I/O, binary I/O and raw I/O. These are generic categories, and various backing stores can be used for each of them. A concrete object belonging to any of these categories is called a file object. Other common terms are stream and file-like object.
Text I/O f = open("myfile.txt", "r", encoding="utf-8"), f = io.StringIO("some initial text data").
Binary I/O, io.BytesIO f = open("myfile.jpg", "rb"), f = io.BytesIO(b"some initial binary data: \x00\x01"). Code.
Raw I/O Also called unbuffered I/O. f = open("myfile.jpg", "rb", buffering=0).
from PIL import Image code1, code2.
imageio.imread Is a library that provides an easy interface to read and write a wide range of image data, including animated images, volumetric data, and scientific formats. It runs on Python 3.5+.

Has an architecture that facilitates parallel and distributed computing. You can develop, execute, debug, and monitor parallel applications.

Image.open PIL.Image.open() Opens and identifies the given image file. code. image. code. code.
Image.save() Saves this image under the given filename. If no format is specified, the format to use is determined from the filename extension, if possible. code1, code2.
v2.rectangle(image, start_point, end_point, color of border line, border thickness) border. Compute the bounding box of the contour and then draw the bounding box on an image to represent where the ROI is. code. code. code.
cv2.imwrite() Save images. Introduction. code. code. code. code.
cv2.imshow() and "xyz"..imshow() A method is used to display an image in a window. The window automatically fits to the image size. First argument is a window name which is a string. Second argument is our image. Introduction. code. code. Code.
cv2.IMREAD_UNCHANGED = -1. reads the image as is from the source. If the source image is an RGB, it loads the image into array with Red, Green and Blue channels. If the source image is ARGB, it loads the image with three color components along with the alpha or transparency channel.
cv2.IMREAD_GRAYSCALE Reads the image as grey image. If the source image is color image, grey value of each pixel is calculated by taking the average of color channels. = 0, code.
cv2.IMREAD_COLOR Reads the image with RGB colors but no transparency channel. = 1
cv.THRESH_TRUNC If pixel intensity value is greater than threshold, it is truncated to the threshold. The pixel values are set to be the same as the threshold. All other values remain the same. code. code
cv.THRESH_TOZERO Pixel intensity is set to 0, for all the pixels intensity, less than the threshold value. code. code
cv.THRESH_TOZERO_INV Inverted or Opposite case of cv2.THRESH_TOZERO. code. code
cv::IMWRITE_EXR_TYPE = (3 << 4) + 0  
numpy.identity(n, dtype = None) Return a identity matrix i.e. a square matrix with ones on the main diagonal. code.
hotkey('i') Introduction
.insert_rows() (code)
.insert_cols For excel files: (code)
.isMaximized (code)
.isMinimized (code)
glob.iglob() (code)
.idf_ (code)
.implicitly_wait() Different from time.sleep(), implicitly_wait(time_to_wait) is to specify the amount of time the WebDriver instance i.e. the driver should wait when searching for an element if it is not immediately present in the HTML DOM in-terms of SECONDS when trying to find an element or elements if they are not immediately available. (code).
iframe in webpage <Introduction>
iat[] Introduction
os.path.isabs() (code)
os.path.isfile() (code)
os.path.isdir() (code)
os.path.islink() (code)
os.path.ismount() (code)
index("") In list. (code)
.DataFrame() Introduction. .drop(), index, columns, axes, dtypes, size, shape, ndim, empty, T, values