Python Automation and Machine Learning for EM and ICs

Batch Gradient Descent (BGD), Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, Batch Stochastic Gradient Descent, Momentum, (Adagrad, Adadelta, RMSprop), and Adam (Adaptive Moment Estimation)

Introduction

Model = architecture + parameters

Introduction

Neuron (= linear + activation) introduction

Introduction

AdaBoost (Adaptive Boosting) Model

Introduction

Experiences of developing machine learning algorithms

Introduction

Bagging (Bootstrap Aggregating)

Introduction

Additive structure/additive model in ML

Introduction

Comparisons among Manual Search, Vertex Vizier, AutoML and Early stopping on google cloud

Introduction

Difference between estimation and approximation errors

Introduction

Example of ML debugging: Anti-Spam

Introduction

Approximation error

Introduction

Assumptions related to distribution of data in ML

Introduction

Probably Approximately Correct (PAC) learning

Introduction

CIFAR (Canadian Institute for Advanced Research) (CIFAR-10 and CIFAR-100)

Introduction

Minimum A Priori (MAP)

Introduction

Mean Average Precision (MAP)

Introduction

Maximum A Posteriori (MAP)

Introduction

Knuth-Morris-Pratt (KMP) algorithm (a string-searching algorithm)

Introduction

Apple Neural Engine (ANE)

Introduction

Custom AI/ML chips/ICs

Introduction

Laplace smoothing/Laplace correction/add-one smoothing

Introduction

Comparisons among artificial intelligence (AI), machine learning (ML) and quantum machine learning (QML)

Introduction

Discriminative algorithms versus generative models

Introduction

Artificial Neural Networks (ANNs)

Introduction

Discriminative algorithms/discriminative models

Introduction

Comparison between mean squared error (MSE), absolute error (L1 Loss) and fourth-power loss

Introduction

Perceptron algorithm

Introduction

Perceptron algorithm and logistic regression

Introduction

Classification and algorithms for classification

Introduction

Comparison between L1 Regularization and L1 Loss (absolute loss or mean absolute error (MAE))

Introduction

Autoencoders

Introduction

Parametric learning algorithm

Introduction

Non-parametric learning algorithm

Introduction

Iterative algorithms

Introduction

Algorithms for directly finding the global optimum

Introduction

Learning Algorithm (estimator)

Introduction

Linear regression and its algorithm

Introduction

Gradient descent algorithm (for updating θ)

Introduction

Actual Probability of Deviation

Introduction

Send a variable from one script (back) to another script with a function

Introduction

Rapid Automatic Keyword Extraction (RAKE)

Introduction

Question answering retrieval

Introduction

Leetcode for Google/Amazon

Introduction

Check/find/get a file name/all file names or the last folder name (e.g. from a path/directory)

Introduction

Compare dates (x days after or before a date), and difference between two dates in days

Introduction

Multinomial Naive Bayes algorithm

Introduction

Feature analysis/feature importance analysis

Introduction

Scalability in automation and machine learning projects

Introduction

Autonomous vehicles/cars and machine learning

Introduction

Train/Test versus Model Accuracy

Introduction

RSquare (R^2) versus RASE (Root Average Squared Error)

Introduction

Adjusted R-squared values of two or more regression models

Introduction

Nonasymptotic versus asymptotic analysis

Introduction

any()

Introduction

Add an item to a dictionary

Introduction

Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc: localhost, insert rows, update, count updated, delete rows, comparision between extract data by Python and SQL itself

Introduction

CycleGAN (Cycle-Consistent Adversarial Networks)

Introduction

Class Activation Mapping (CAM)

Introduction

AI/machine learning algorism for text analysis

Introduction

Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc

Introduction

Check if all the (and how many, length of a string) characters in the text are digits/numbers

Introduction

Calculating the area fraction of each circle overlapped by a square grid and build wafer map

Introduction

Percentage Estimation of Pixels Within a Defined Intensity Contrast Range in Grayscale Images

Extract any substrings with any pattern (e.g. dot (.))

Convert all elements of specific column or in entire dataframe into strings

Trick: pd.concat() for merging/adding (two) columns

Automatically run a file in an application

Click a menus of an application

Different behavior of automation execution (e.g. pyautogui) locally or remotely through internet

Trick: generic code/script templates for complex automation

Generative Adversarial Network (GAN) technologies

Asymptotic analysis

Introduction

Well-specified case of "asymptotic approach"

Introduction

Compare (pattern/ratio of) two different columns, check whether column values match in DataFrame

Introduction

Check whether one column contains number only and another column contains letters only or mixture of numbers and letters in DataFrame

Introduction

Check the difference between two columns in DataFrame

Introduction

Check if a file exists again (double check)

Introduction

ArithmeticError (arithmeticException)

Introduction

Avoid duplicates when creating text file

Introduction

Automatically restart script execution after it breaks/fails/error

Introduction

Call and run another script in a different/any (parent or children) directory/path/subfolder from a script

Introduction

Modify HTML webpage (e.g. with graph network by adding/inserting text/hyperlink in)

Introduction

Convert/change the case of all letters/word into uppercase (capital) or lowercase in a list of strings

Introduction

Check if a variable does exist/is assigned/defined

Introduction

Check all the imported/current modules/libraries

Introduction

Plot multiple datasets on the same scatter graph with different x- and y-axis values

Introduction

Write special/certain rows (row-by-row) of one csv file to another csv file

Introduction

y axis values are not ordered (disordered)

matplotlib.pyplot to plot/generate images (with axis/colored text or annotation)

Avoid two or multiple plots being wrongly/incorrectly/unnecessarily mixed/overlap

Check if one list is subset of another

Remove/reload/unload (all) imported module/function/script

Plot graph/figure/image from CSV file/DataFrame by removing/hiding blank/empty cells with axis range (plt.xlim())

Copy a file or all files (with os.mkdir) to save to somewhere (create a directory first if it does not exist)

Separately plot data into the same graph/figure/image from different csv files for each category (import multiple CSV files and concatenate into one DataFrame): append row-by-row or column-by-column

sort_values(by=... ascending/descending order)

Introduction

Plot multiple images on the same figure by hiding x- and y-labels on axis

Introduction

Convert between numpy array and string

Introduction

Continue script execution no matter whether some try fails or not (finally, always executes) or else

Introduction

Plot a figure with a colored arrow between text lines/steps

Introduction

__add__, __call__, __contains__, __delitem__, __delattr__, __eq__, __enter__, __ge__, __getattribute__, __getnewargs__, __getattr__, __getitem__, __gt__, __hash__, __reduce__, __iadd__, __imul__, __init_subclass__, __index__, __int__, __invert__, __new__, __neg__, __reduce_ex__, __reversed__, __rmul__, __radd__, __rand__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __ror__, __round__,__rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __dir__, __doc__, __divmod__, __iter__, __le__, __lt__, __len__, __ne__, __repr__, __setattr__, __setitem__, __sizeof__, __lshift__, __sub__, __subclasshook__, __str__

Introduction

Get the date and time (a past date) of N days ago

Introduction

Check if all/any values are true or false in a range of data

Introduction

Merge/append rows/columns of a csv file into an old csv file if the rows/columns are not in the old csv file

Introduction

Add letter/commas/numbers/characters to the end/beginning of strings in a list

Introduction

Plot workflow: Create new empty column in DataFrame, Move the cells in a column to another column under certain condition, Select specific columns for scatter plot

Analytics and Technology Automation (ATA)

Introduction

Aggregate duplicates in columns of data

Introduction

.apply(pd.Series)

Introduction

.apply(tuple, axis=1)

Introduction

Output data if any or same element in a string are in two lists

Introduction

Overcoming automation challenges and forward-looking suggestions

Introduction

Aggregation functions in data manipulation and database queries

Introduction

accuracy_score()

(code)

Accuracy/precision test

Introduction

.axes[0] and .axes [1]

CSV: for csv. (code)

alert()

alert(text='', tilte='', button='Ok'). (code)

3 types of logical operators/booleans

Represent one of two values: True or False. code. code. code.

and / &

The output is 'true' or 'false', when both the conditions are 'true' or 'false'. Both/all conditions must be true for the statement to be true. "and" is binary, but not is unary. Introduction. Example code.

import ... as ...

.append()

Introduction. Appends an element to the end of the list. copy method. Add a list/item at the end of the list.

A = A[ : :-1]

Reverse an array

sys.argv

Introduction

from ... import ... as ...

import ... as ...

E.g. code for opening an image. , code2

from tkinter.filedialog import askopenfilename

code. code.

with ... as ...

code. code.

.shapes.add_textbox

(code)

slides.add_slide()

(code)

.shapes.add_picture()

(code)

.add_run()

(code)

assert()

code.

__abs__

watchdog.observers.api: Immutables

class watchdog.observers.api.ObservedWatch(path, recursive)

Parameters: path – Path string. recursive – True if watch is recursive; False otherwise.

class watchdog.observers.api.EventQueue(maxsize=0)

class watchdog.observers.api.EventEmitter(event_queue, watch, timeout=1)

class watchdog.observers.api.EventDispatcher(timeout=1)

class watchdog.observers.api.BaseObserver(emitter_class, timeout=1)

watchdog.observers.api.EventDispatcher

Base observer.

add_handler_for_watch(event_handler, watch)

Adds a handler for the given watch.
Parameters: event_handler (watchdog.events.FileSystemEventHandler or a subclass) – An event handler instance that has appropriate event handling methods which will be called by the observer in response to file system events.
watch (An instance of ObservedWatch or a subclass of ObservedWatch) – The watch to add a handler for.

watchdog.observers.api.BaseObserver

Platform-independent observer that polls a directory to detect file system changes.

watchdog.observers.api.BaseObserver

File system independent observer that polls a directory to detect changes.

math.acos()

Returns the arc cosine of x in radians.

math.atan()

Returns the arc tangent of x, in radians.

math.acosh()

math.atan2()

Returns atan(y / x), in radians.

math.asinh()

math.asin()

math.atanh()

as_integer_ratio

with mss.mss() as sct

(code)

Multiple assignment

E.g. a, b = 4, 3

*args

"*" is the unpacking operator, which is not a list but a tuple. *args allows you to pass multiple, varying arguments or keyword arguments to a function. Note that args is just a name. You’re not required to use the name args. You can choose any name that you prefer. Introduction.

asctime()

code

auto-py-to-exe

(code)

.axis()

Introduction

__annotations__/dict

name/type: parameter and return annotations

skimage.measure.approximate_polygon(coords, ...)

Approximate a polygonal chain with the specified tolerance.

asyncio

Comparison between multithreading, multiprocessing and asyncio at page4797.

cv2.add

code.

cv2.arrowedLine()

cv2.arrowedLine(image, start_point, end_point, color[, thickness[, line_type[, shift[, tipLength]]]]) is used to draw arrow segment pointing from the start point to the end point. The parameters of the cv2.arrowedLine function are the same as those for cv2.line. code.

matplotlib subplot object to plot on; if nothing passed, uses active matplotlib subplot

alpha

The plot fill opacity (from 0 to 1)

arrowprops=

code.

.annotate()

code.

arrowstyle

code.

.argwhere()

(code)

.array

General, number array . for image. code. in csv.

.arange()

Introduction. Can change with an increment. General, code. code.

hotkey('a')

Introduction

alt

Introduction

.active

(code). (code).

.activate()

Activate a window: with the active cursor in the window and the window is brought to the most front on the monitor. (code)

'xyz'.isalpha()

Check if string is alphabet (letter, or one type of character)

auto_size_text=True

(code).

.rjust()

Right-justify/align them so that they take up the same amount of space, whether the coordinate has one, two, three, or four letters or digits. (code).

add_connector()

shapes.add_connector(MSO_CONNECTOR.STRAIGHT, Begin_x, Begin_y, End_x, End_y) (code).

.add_series()

(code)

.add_chart()

(code)

Comparison between *, .extend((), append(), =, ==, .copy() and copy.copy() for "list": changes of "list"

Introduction

Image rotation in pptx (.rotation=)

Introduction. E.g. Rotate an image around the top-left corner as an axis in a pptx file with the same lateral dimension

.at[]

Introduction

os.path.abspath()

(code).

Variable length arguments (*args and **kwargs)

Introduction

numpy.all/np.all

Introduction

.DataFrame(): drop(), index, columns, axes, dtypes, size, shape, ndim, empty, T, values

Assert and assertion

Introduction

Vertex AI Feature Store

Introduction

Data Augmentation

Introduction

Keyword arguments

Introduction

Automation with programming

Introduction

Reasons of automation and how to start

Introduction

Generic (or generalized) robot programming (GRP)

Introduction

Graphical user interface (GUI)

Introduction

API (Application Programming Interface), e.g. weather, temperature

Introduction

Keyboard and mouse automation

Functions of mouse

Introduction

Codes: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, pynput, ctypes, uiautomation and Sikuli)

Introduction

Principle and troubleshooting: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli)

Introduction

Automatically review, scroll, click webpage and its link

Introduction

Copy text into clipboard and then you can paste it a webpage, text/txt, word or powerpoint file automatically

(code)

Automatically fill contexts into or select options on webpage

Introduction

AutoML

Introduction

Automation of EELS data extraction

Ranking/most popular automation testing tools

Introduction

Ranking/most popular IT automation software tools

Introduction

Comparison between Python, Blue Prism, UiPath, Automation Anywhere

Introduction

Automated defect scanning in wafer map

Introduction

Robots and Robotic Process Automation (RPA)

Introduction

Python

Introduction

Reasons of using Python for automation

Introduction

UiPath

Examples of UiPath applications

Introduction

Installation of UiPath and its packages and creation of new projects

Introduction

Comparison between Python, Blue Prism, UiPath, Automation Anywhere

Introduction

API

3 ways to create a Keras model with TensorFlow

Sequential API to create a Keras model with TensorFlow

Introduction

Functional API to create a Keras model with TensorFlow

Introduction

Model Subclassing to create a Keras model with TensorFlow

Introduction

tf.data API

Introduction

tf.keras API

Introduction

Add padding/black/colored edge to images

Introduction

except ... as -- Exception

Introduction

Evaluation of accuracy in machine learning process

Introduction

Hide/turn on/off axes/axis on matplotlib

Introduction

Adam optimization algorithm

Introduction

Activation functions in machine learning

Introduction

Analysis process in ML

Introduction

Linear algebra

Introduction

Vertex AI

Introduction

.astype()

(Code)

after_run function

Introduction

except AttributeError

Introduction

TensorFlow APIs

Introduction

Axis/dimension of tensor

Introduction

adapt()

Introduction

.as_default()

Introduction

Table of applications of Python and its libraries

Introduction

Open an application window through search at start

Introduction

Add a new slide into an existing ppt or a created ppt file

Introduction

Launch the existing opened application if there is or start a new one if there is not

Introduction

Open and close any type of files with default programs/apps (e.g. word, excel, dm3, dm4, Digital Micrograph, powerpoint, internet explorer, chrome, and so on) in windows

Introduction

Copy and apply formatting in Word and PowerPoint

Introduction

Move the active window to make space for other apps

Introduction

Go to the pointed tab on an app

Introduction

Convert capital alphabet letters/characters to number

Introduction

Minimize/maximize/restore/activate/resize/move/close Window objects

Introduction

Bring/activate an application to most front/foreground

Introduction

Typical training setup in Artificial intelligence (AI)

Introduction

matplotlib.pyplot axis/text color

Introduction

Convert a CSV file to an image with one column and another column as x-axis and y-axis, respectively.

code.

Calculator of length accuracy in 3D structure

Introduction

Swap the order of arguments in a function

Introduction

Copy text into clipboard and then you can paste it anywhere

Introduction

Calculate/pass the arbitrary (any) number of variables or input arguments

Introduction

Access variable inside and outside a function

Introduction

Access elements in a dictionary and subdictionary

Introduction

Access elements in a list and sublist

Introduction

Numpy: Access the element at the second row, the third entry, access a specific row or a column, access some elements (submatrix), or replace/modify an element in the array, print a transfer of an array, access array under conditions or filtering

Introduction

.T (Transfer of array in Python)

Introduction

Calculations in DataFrame: Add a column, calculate for a new column, delete a column, all the rows with values greater than 30 in "Score A" column

Introduction

Markers (e.g. color cross, scatter, and circles) at specific coordinates with x- and y-axis

Matplotlib

Keyword search function/check whether or not a string is within another string (a space is included as a string character)

Introduction

Draw an arrow segment pointing from the start point to the end point in an image.

code.

Draw lines manually and then label them with arrows

code

Prevent other applications to modify the content until other Python script runs

code.

Show/open images in any image viewer

code, code.

Count the number of lines (rows) and columns in a txt (and a csv) file, count different numbers in each region in a column, count missing or not available values

CSV: Introduction. code.

Sum two images after resizing them

code

Create images with global, adaptive mean, adaptive Gaussian, binary, trunc, Tozero, and tozero thresholds.

code

Draw lines, elbow connectors and arrows in a ppt

Introduction

Remove an/all/duplicate item(s)/elements from a list

Introduction

Comparison between iteration algorithm and recursive algorithm: a function repeat itself

Introduction

Add a new slide into an existing ppt, or work on existing slides

Introduction

Work with (e.g. open) all/every files and subfolders/subdirectory in a folder (does not include any files or include all files in subfolders); get a list of all files and directories in the same folder where the Python script file is.

Introduction

Insert all the images into a ppt file (one image per slide)

(Introduction)

Mean (average, .mean())/.sum()/maximum(.max())/minimum(.min())/number of non-null values(.count())/.median()/variance(.var())/standard deviation(.std())

Introduction

Add markers on a map

Introduction

Add/insert a column into an existing csv file

Introduction

Call and then run your own functions and modules in different/other Python files; Python run another Python script

Introduction

Run multiple Python files/scripts one after another

Introduction

Monitor specific new files, and execute the file or another file (and then restart the monitoring program itself to continue its monitoring by standby, with watchdog; or observer runs in the background and pass the value of obtained events to a/another function call)

Introduction

Watchdog for monitoring specific file or files with specific extension, and then run another file from watchdog

Introduction

Launch script from another script using subprocess.run/subprocess.call

Introduction

Simple ways to execute another python file when a new file has been uploaded

Introduction

Move(remove) all files from original folder in a directory to a new directory

Introduction

List all files and directories which has specific files or files with specific extensions

Introduction

top and left for pptx (e.g. align the top-left corner of the image to the center of the slide no matter how the size of the images changes)