Electron microscopy
 
Python Automation and Machine Learning for ICs: Chapter P
- Python Automation and Machine Learning for ICs -
- An Online Book: Python Automation and Machine Learning for ICs by Yougui Liao -
Python Automation and Machine Learning for ICs                                                                   http://www.globalsino.com/ICs/        


Table of Contents/Index 
Chapter/Index: Introduction | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Appendix

=================================================================================

 

 
                                                       
K-means clustering and PCA for failure analysis Introduction
Software/interface/API (Application Programming Interface) used in data science and machine learning Introduction
Spark driver program Introduction
Managing machine learning (ML) projects Introduction
Practice project of data processing using Spark Introduction
Personalizing applications with ML Introduction
Trade-offs between fairness and performance Introduction
Best practices for implementing ML in semiconductor manufacturing Introduction
Semantic Segmentation Using U-Net with EfficientNet and Pixelshuffle Introduction
Using proxy labels, building a labeling system, and utilizing a labeling service when historical labeled data is unavailable for ML projects Introduction
Parametric and non-parametric learning algorithms Introduction
Platform Security Engineering (PSE) and Machine Learning Introduction
Parallel computing and distributed computing Introduction
Analyzing Data in Hadoop (HDFS, YARN, Apache Hive, Pig, HBase, Spark) Introduction
Comparison between Apache Spark's MLlib and Python Introduction
Timeseries prediction in ML Introduction
Calculation of Principal Component Analysis (PCA) Introduction
Squared Pearson correlation coefficient Introduction
Principal Component Analysis (PCA) versus Uniform Manifold Approximation and Projection (UMAP) Introduction
Uniform Manifold Approximation and Projection (UMAP) Introduction
Proxy Introduction
Dataset and data preparation Introduction
Pre-training in ML Introduction
GPT (Generative Pre-trained Transformer) Introduction
Performance metrics Introduction
Max-pooling Introduction
TensorFlow Playground Introduction
Open datasets, and open-source tools and libraries for Python and ML practice Introduction
Python Tutor Introduction
Python Conference Introduction
Python introduction and application examples Introduction
Plot with letters/words as x-/y-axis Introduction
Fit and Smooth Plotted Curves Introduction
Constraint Satisfaction Problems (CSPs) as Search Problems Introduction
Correlations/similarity/dissimilarity/pair/match of two columns in csv data Introduction
Linear Programming (LP) algorithm Introduction
Evaluation (Precision and Recall) in Text classification with Naive Bayes Introduction
Precision and Recall Tradeoff Introduction
Precision, Recall, False Positive Rate, and False Negative Rate (Miss Rate or False Negative Proportion) Introduction
Search patterns in a text file Introduction
pgmpy Introduction
Unconditional probability Introduction
Distributive Property Introduction
Model Checking Algorithms and Modus Ponens Algorithms Introduction
Modus ponens (a logical inference rule) Introduction
Logic Puzzle in ML Introduction
Propositional Logic Algorithms in ML Introduction
Proposition symbols Introduction
Search problem in ML Introduction
Path and Path Cost in ML Introduction
POMDP (Partially Observable Markov Decision Process) Introduction
Policy search algorithms and "normal" reinforcement learning (RL) algorithms Introduction
Policy Search Algorithms Introduction
Stationary and Non-Stationary State Transitions in Markov Decision Process (MDP) Introduction
State transition function (probability) in reinforcement learning Introduction
Policy iteration versus value iteration Introduction
Policy iteration Introduction
Markov Decision Process (MDP) Introduction
Finding a correct loss (risk, objective) function for a specific problem Introduction
(Forward and backward) propagation equations Introduction
Linear correlation between two variables with Pearson Correlation Coefficient, Spearman Rank Correlation Coefficient, Kendall's Tau, Linear Regression, Coefficient of Determination and Correlation Ratio
Introduction
Model = architecture + parameters Introduction
Structure of PowerPoint reports Introduction
Data parallelism in distributed training Introduction
Linear model versus polynomial model Introduction
Posterior distribution Introduction
Distribution of θ (parameter distribution) in ML Introduction
Population Distribution in ML Introduction
Updating Hypothesis (ĥ) and/or Parameter θ^ in ML Introduction
Probably Approximately Correct (PAC) learning Introduction
Polynomial models in ML Introduction
Choice of parameters for training models Introduction
Minimum A Priori (MAP) Introduction
Mean Average Precision (MAP) Introduction
Maximum A Posteriori (MAP) Introduction
"Norm" of parameters, and L1 Norm (Manhattan Norm) and L2 Norm (Euclidean Norm) Introduction
Conditional probability Introduction
Posterior probability and prior probability Introduction
Comparison between Poisson distribution, Gaussian (normal) distribution and logistic regression Introduction
Single parameter estimation versus multiple parameter estimation Introduction
Parameterizations in ML Introduction
Exponential Family: Parameter, Sufficient Statistic, Natural Parameter, Base Measure and Log-Partition Function (Bernoulli distribution and Gaussian distribution) Introduction
Bayesian Probability, Bayesian Statistics (Distribution Over a Distribution), versus Bayesian Inference Introduction
Parameters, features and examples in ML Introduction
Comparison between mean squared error (MSE), absolute error (L1 Loss) and fourth-power loss
Introduction
Practice ML projects for beginners Introduction
Perceptron algorithm and logistic regression Introduction
Actual Probability of Deviation Introduction
Deviation Probability (Hoeffding Bound) Introduction
Core Steps/Procedure/Designing of Machine Learning Introduction
Pearson Correlation Coefficient/Pearson's r/Correlation Coefficient Introduction
Point-Biserial Correlation Introduction
Types of predictions with Supervised Learning Introduction
Cocktail party problem Introduction
Hypothesis (predicted output (h(x))) Introduction
Parametric learning algorithm Introduction
Non-parametric learning algorithm Introduction
Update parameters θj using gradient of the loss function Introduction
Perceptron algorithm Introduction
Bandwidth parameter (τ) in LWR and KDE Introduction
Probability density function (PDF): comparisons between (normal (gaussian) distribution, uniform distribution, exponential distribution and poisson distribution) Introduction
   
PySpice: Provides the interface to circuit simulators like Ngspice or Xyce. code.
import PySpice.Logging.Logging as Logging (code)
from PySpice.Spice.Netlist import Circuit (code)
from PySpice.Unit import * (code)
.simulator() (code)
NgSpice/PySpice for electrical circuit analysis Coding introduction
Save contents (download pdf files) in the webpages obtained by Google search into a text file Introduction
Analysis of ROI (region of interest)/portion of an image Introduction
Find nearest white pixel to a given/specifical pixel location on an binary image Introduction
Diversity prediction theorem Introduction
Categorical features preprocessing layers Introduction
Reasons of using Python for automation Introduction
Machine learning example step-by-step (prediction of house price) Introduction
.pack(side=LEFT)/.pack(side=RIGHT)/.place(x=, y=) --- position of the buttons (Code)
   
Add padding/black/colored edge to images Introduction
.pad() (code)
   
.paste() Introduction
Common values in two pandas series Introduction
Keras preprocessing layers Introduction
pywinauto Introduction
Machine learning workflow/procedure Introduction
Predictive/predict model (with "best"-option) Introduction
   
Pipelines in ML and Data Science Introduction
Pipelines in ML Introduction
Pipelines in Data Science Introduction
kfp.dsl package versus pipelines and components Introduction
Learning algorithm (ensemble learning) and pipeline Introduction
   
preprocessing. Introduction
PyTorch Introduction
Comparison between TensorFlow, PyTorch, Theano and OpenCV Introduction
Troubleshooting and problem solving in Python programming page4806
Heatmap with input from a csv/pkl file Introduction
Positions and colors of mouse/cursor and features Introduction
Left click a specific position Introduction
Right click a specific position Introduction
Double click a specific position Introduction
Print document Introduction
Search in a powerpoint file Introduction
Save files (e.g. powerpoint/ppt) Introduction
Summary of working on ppt Introduction
Analysis of papers/publications/literature in machine learning and Python applications Introduction
Plot workflow: Create new empty column in DataFrameMove the cells in a column to another column under certain conditionSelect specific columns for scatter plot
Add a new slide into an existing ppt, or work on existing slides, check the existence of a pptx file, if does not exist then create it Introduction
Table of powerPoint shortcut hotkeys Introduction
Table of PC/computer/Windows shortcut hotkeys Introduction
Comparison between Python and C/C++ Introduction
stopwatch and timing a process Introduction
Turn on and off with mouse press or a process Introduction
Quit/exit/stop a process (including by pressing a letter) Introduction
Periodic Table code
(Single and multiple enter/input) box for pop-up window Introduction
Copy and then store it into memory and it can be pasted for use later Introduction
Tricks in Python Programming and principles and practices in good programming Introduction
Text classification/sort/prediction, train/test e.g. Youtube spam Introduction
Print colored text in Python IDLE's terminal Introduction
API (Application Programming Interface), e.g. weather, temperature Introduction
Find/predict the best word similarity, e.g. car Introduction
Principal Component Analysis (PCA) Introduction
Select/input a folder/directory/path for later to be called to use Introduction
Pop-up windows/messages tkinter, ctypes, easygui
Convert CSV to images, row by row, with pixel values: each row is an image code.
Calibrate and put a scale bar, and draw a line segment on an image Introduction
Option/selection/choice methods ("pop-up windows of Yes and No ") Introduction
Copy text into clipboard and then you can paste it anywhere Introduction
Calculate/pass the arbitrary (any) number of variables or input arguments Introduction
File name, folder name. {}{}....format. Manipulation of file and folder names (rename file name and folder name): i) Create a new folder and then copy all files from a folder to the new folder and rename the file, and then open the file. If the folder exists, then no file will be copied, but the file will still be opened. ii) Print and export the folder names and file names (with or without extensions) from a folder into a text file. iii) csv2image filename. Introduction
Set default programs by file extensions and by file types and programs on Windows Introduction
Numpy: Access the element at the second row, the third entry, access a specific row or a column, access some elements (submatrix), or replace/modify an element in the array, print a transfer of an array, access array under conditions or filtering Introduction
Pint summary of the statistic data, change data format, sort/group columns Introduction
Popup windows with inputs Introduction
Popup windows with frames Introduction
Copy text into clipboard and then you can paste it a webpage, text/txt, word or powerpoint file automatically (code)
Draw an arrow segment pointing from the start point to the end point in an image. code.
Prevent other applications to modify the content until other Python script runs code.
Open an image to file from URL (webpage), then it can be saved in PC code, code. open with color changed.
Calculate the coordinates of a point in a given rectangle and the distance of a given point to a line code
Compute the difference between two images by using Structural Similarity Index with "pip install --upgrade imutils" code. code
Search position of numbers code
Reference list items by position code1, code2
Repeated printing the same string code
Palindrome repeat code
Selenium and pop-up windows Introduction
Check if a popup dialog is a window or not for Selenium app Introduction
Draw lines and arrows in a ppt Introduction
Find the color of a pixel on the screen Introduction
Create slides with text only Introduction
Draw lines, elbow connectors and arrows (shapes) in a pptx Introduction
Count and delete slides from ppt Introduction
Insert all the images into a ppt file (one image per slide) (Introduction)
Check file existence with partial filename Introduction
Get/list immediate subdirectories/subfolders; get only the last part of a path/folder/drive; split a dos path into its components, and then print the list Introduction
Move file(s) from one directory to another Introduction
Get maximum and minimum value of column and its index Introduction
Get the csv/pandas cell value with certain condition Introduction
Find latitude and longitude of a place in a map Introduction
Check/find/get a file name or the last folder name (e.g. from a path/directory) Introduction
Print and set file path as a variable (e.g. convert all characters in the pathname to lowercase) Introduction
Check if a file/folder/path/directory exists or not (Cannot find a specific file/folder?, or check files with extension, select specific folders to form a string, split a dos path into its components, and then print the list) Introduction
Check if a file/folder exists or not (Cannot find a specific file/folder? a specific folder in the path?) Introduction
Check if a directory is empty; find the position/index of a particular file/folder in a file directory/path; remove folder or file level by level (or layer by laer) from its directory/path Introduction
Placement/position of Python import statements Introduction
Watchdog ignore/skip and take pattern in directory/path Introduction
top and left, width and height for pptx (e.g. align the top-left corner of the image to the center of the slide no matter how the size of the images changes) Introduction
Extract text/check specific text from multiple powerpoint files (Some methods can extract text from most of the document extensions such as pptx and pptm) Introduction
Get mouse position/coordinates on click Introduction
Measure (real or pixel) length/distance on an image w/o calibrated bar Introduction
Resize (by real dimension or pixel) and then sum/mix/overlap two images Introduction
Get pixel location/coordinates on an image using mouse click/events Introduction
Crop/snip part of a image with definition by a pixel line Introduction
Modify file path/directory by changing folder names by merging a list Introduction
Email providers and their SMTP servers Introduction
Merge/combine two pptx files into one Introduction
Find the file names of the images in a pptx, (and then save/extract the image as a file) Introduction
Count occurrence/nubmer of words/phrase in a text file Introduction
Remove/repace (part) character(s) from string Introduction
Ranking/most popular programming languages for data analysts Introduction
Convert PDF file to text file Introduction
Extract pdf pages to form new pdf files Introduction
Ranking/most popular automation testing tools Introduction
Take a screenshot/snipping using a mouse click and drag method. For instance, take a screenshot, and then insert the image and/or a text into a ppt file. Introduction
Ranking/most popular IT automation software tools Introduction
Comparison between Python, Blue Prism, UiPath, Automation Anywhere Introduction
Robots and Robotic Process Automation (RPA) Introduction
Ranking/most popular machine learning frameworks used by data scientists Introduction
Wafer map failure pattern recognition (WMFPR) Introduction
Histogram for wafer analysis (e.g. percentage and frequency of grey level in the image) Introduction
Spatial defect patterns Introduction
Probe data Introduction
Non-zero pixel values from an image Introduction
Pixel values on specific pixel in an image Introduction
Public datasets for wafer map analysis Introduction
Pass variables between functions/from one to another Introduction
Methods of physical failure analysis (PFA)/root cause analysis of ICs Introduction
Convert images between Cartesian and Polar forms Introduction
Fault analysis/PFA (Physical Failure Analysis) time and efficiency Introduction
Patterns on wafers commonly reflecting specific process failure information Introduction
Print specific rows of a DataFrame Introduction
Table of applications of Python and its libraries Introduction
Detection procedures/processes of spatial defect patterns (bins) in wafers Introduction
Read/print a text/txt file Introduction
Store images in pandas dataframe column Introduction
Pickling and unpickling of pandas DataFrame (.pkl) Introduction
Probability bounds analysis (PBA) Introduction
Print the files and keyword occurrence which have been searched from a ppt file Introduction
Percentages of information received through different senses (eye, nose, ear and hand feeling) Introduction
Expected risk (population risk, expected value of loss or error) Introduction
Empericial loss versus population loss Introduction
Parameterized family and model parameters Introduction
Predicted label Introduction
Keyword Module in Python Introduction

Python drivers for SQL server (pyodbc, pymssql, PyMySQL, cx_Oracle and psycopg2)

Introduction
Remove unwanted/unnecessary parts from strings in a column of dataframe Introduction
Iterate over rows in a DataFrame/read and print row by row (number of columns and rows, df.shape[0]/df.shape[1]) Introduction
Trick: Output a portion of rows and columns from a csv file cell by cell into another csv file Introduction
Paraphrase mining Introduction
pyodbc for bridging SQL to Python Introduction
Create a Batch File to Run a Python Script Introduction
Read a frequently updated file periodically (similar to watchdog) Introduction
Codes: Automation of Mouse Movements and Clicks, and keyboard control (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, pynput, ctypes, uiautomation and Sikuli) Introduction
Principle and troubleshooting: Automation of Mouse Movements and Clicks (comparison among pyautogui, pygetwindow, pydirectinput, autoit, Quartz, platform, ctypes, uiautomation and Sikuli) Introduction
Trick: Get coordinate difference between mouse positions Introduction
Evaluation of Precision in Machine Learning Process Introduction
Probabilistic model Introduction
pyodbc for bridging SQL to Python Introduction
Access and use SQL Database on SSMS (Microsoft SQL Server Management Studio Express) with pyodbc: localhost, insert rows, update, count updated, delete rows, comparision between extract data by Python and SQL itself Introduction
Insert data/row into SQL Database on SSMS with pyodbc Introduction
Get directory/path/file name partially Introduction
Call and run another script in a different/any (parent or children) directory/path/subfolder from a script Introduction
Get the current directory/folder path Introduction
Python modules to interact with the operating system (os, platform, subprocess, shutils, glob and sys) Introduction
Change the current directory to any directory/path (e.g. with os.chdir) Introduction
Check existence of phrase on text file line-by-line Introduction
                                                       
"@echo off" and "pause" in Command Prompt Window Introduction
Last n days/weeks/months (.to_datetime(x), .set_index(y), .last(z), .reset_index(), and .max() in pandas) Introduction
Check if Windows/PC screen is locked Introduction
Combine multiple images into a single multi-page image or vice versa Introduction
Count the number of the pages in a single multi-page/frame image Introduction
Extract elements from a list (different way from removing elements to get part of the list) Introduction
Lock a file to prevent deleting, and then release the file once job is done Introduction
PermissionError (E.g. file cannot be written when it is open) Introduction
Extract/confirm any substrings with any pattern (e.g. dot (.)) Introduction
Get username and encoded password with getpass or or base64 Introduction
Find repeating patterns in columns, group them as cycles, and column correlations Introduction
Create table on pptx with certain rows and columns of strings Introduction
Create table on pptx with certain rows and columns in DataFrame Introduction
hotkey('h') in pyautogui Introduction
Difference/comparison between real mouse click and click from script/program, e.g. Pyautogui
Introduction
   
   
RegEx (Regular Expression) (characters to check if a string contains a specified search pattern, remove double spaces, and clean texts) Introduction
   
Compare (pattern/ratio of) two different columns, check whether column values match in DataFrame Introduction
Check whether one column contains number only and another column contains letters only or mixture of numbers and letters in DataFrame Introduction
Check the difference between two columns in DataFrame Introduction
   
   
   
csv workflow: Read into dataframeSelect a specific column from DataFrame, Select several specific columns to plot 
from keyboard import is_pressed (Esc, check pressed key) Introduction
Execute a command on Command Prompt of Windows Introduction
Check if one list is subset of another (partially (part of)) Introduction
Insert paragraphs of texts into Python script (f"""/f''') Introduction
Plot figures with date/month/year Introduction
Summary/templates/examples of pptx and PowerPoint format Introduction
.bat (batch) files for Command Prompt Windows Introduction
pd.concat() Introduction
                                                       
Penalized regression (Lasso and Ridge) Introduction
Joint probability distribution (p) Introduction
Predicted label versus predictor (feature) Introduction
Hypothesis class/hypothesis family/predictor class/model class/hypothesis family/predictor family/model family Introduction
Predicted values (ŷ, y^) Introduction
Phrase Introduction
Summary/templates of plotting graphs/figures Introduction
Plot from dictionary Introduction
Data plot with labels Introduction
Plot multiple images on the same figure by hiding x- and y-labels Introduction
Plot table Introduction
Plot graph/figure/image from CSV file/DataFrame Introduction
Plot graph/figure/image from CSV file/DataFrame by removing/hiding blank/empty cells with axis range (plt.xlim()) Introduction
Plot images with certain image size and in color Introduction
Plot multiple datasets on the same scatter graph with different x- and y-axis values Introduction
Separately plot data into the same graph/figure/image from different (multiple) csv files for each category Introduction
Plot a list of x, y coordinates to an image Introduction
|Plot distance between points calculated by coordinates Introduction
Plot pixel intensity (histogram) along a line (row/column/x-axis/y-axis) of an image Introduction
Plot histogram Introduction
Change/convert a colored image to a grey image(, and then show pixel values). cv2, cv2, cv2/skimage. cv2/skimage. PIL. matplotlib.
Plot curve/chart in pptx Introduction
Transparency of marker (e.g. for plots) Introduction
Loops (e.g. for loop) for 2D (two-dimensional) plot (or map) Introduction
plotly.graph_objects Introduction
Commands/concepts of plot with matplotlib.pyplot to plot/generate images: e.g. axis/colored text or annotation, ScaleBar() Introduction
plots (code)
.xticks() and .yticks() Introduction
code. code code
plt.plot(): Combine multiple plots and plot continuous curve, solid green ('-g'), dashed green ('--g'), dashdot ('.g'), dotted (':g'). Plot by different grouping and summing. code. Introduction
plt.scatter(): Plot scattered curves Introduction
plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None), or plt.subplots_adjust(wspace=0, hspace=0): Adjusting the spacing around subplots code
Plotly: Is used for plotting graphs easily. This works very well in interactive web applications. With this, we can make different types of basic charts like line, pie, scatter, heat maps, polar plots and so on. Whatever visualisation we can think of, we can easily plot a graph of it using Plotly.  
from matplotlib import pyplot as plt: == import matplotlib.pyplot as plt. (code)
bokeh.plotting.figure.circle()  
Inside/outside edges/margins of plotted images Introduction
Avoid two or multiple plots being wrongly/incorrectly/unnecessarily mixed/overlap Introduction
Recall (Sensitivity or True Positive Rate) in machine learning Introduction
Plot a figure with a colored arrow between text lines/steps Introduction
Python IDLE restart during execution Introduction
Box and Whisker plots Introduction
Get the date and time (a past date) of N days ago Introduction
std() and pstdev() standard deviation in csv/dataframe Introduction
Waterfall plot Introduction
Safely use credentials (username and password) in Python project Introduction
Tableau integration with Python Introduction
Percentage Introduction
Color the Tables in pptx (PowerPoint) Introduction
Define/measure the size of PowerPoint/pptx slides Introduction
Color and rotate/vertical text in pptx Introduction
Selecting only numeric/number columns, and then select two specific columns for plot Introduction
Hide x-axis tick labels (only show some labels) where x values are under certain conditions Introduction
Create table with merged cells on pptx Introduction
Plot images from different DataFrame in a single row Introduction
Cheatsheet of image plot Introduction
plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None), or plt.subplots_adjust(wspace=0, hspace=0) Introduction
Font size of tick labels in plot Introduction
PowerPoint (pptx) cheatsheet Introduction
Font size of a (single/multiple) cell in table in PowerPoint Introduction
Change the width of the cell in ppt Introduction
Only use the first 4 characters in the headers of the table for pptx/dataframe Introduction
Populate the table with logarithmic format in pptx Introduction
Cheatsheet of PySpark (for SparkSQL) and DataFrames Introduction
Comparative overview of multivariate statistical methods (Correlation Analysis, Regression Analysis, Factor Analysis, Cluster Analysis, Principal Component Analysis (PCA), Canonical Correlation Analysis, Discriminant Analysis, Path Analysis, Structural Equation Modeling (SEM), Multivariate Analysis of Variance (MANOVA), Analysis of Covariance (ANCOVA) ): purposes, variables, and outputs Introduction
HTTP(Hypertext Transfer Protocol)/URL (Uniform Resource Locator) Introduction
Cheatsheet of Python programming  Introduction
Cheatsheet of Pandas and DataFrame Introduction

 

Pandas

Introduction. Is a must for data-science since it is one of the most used packages for analyzing data, data exploration, and manipulation. It provides fast, expressive, and flexible data structures to easily (and intuitively) work with structured (tabular, multidimensional, potentially heterogeneous) and time-series data. Pandas can do well:
     Easy handling of missing floating point and non-floating point data
     Columns can be inserted and deleted from DataFrame and higher dimensional objects
     Automatic and explicit data alignment: objects can be aligned or be ignored to a set of labels
     Powerful, flexible grouping
     Convert ragged, differently-indexed data in other data structures into DataFrame objects
     Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
     Intuitive merging and joining data sets
     Reshaping and pivoting of data sets
     Hierarchical labeling of axes
     Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases
     Compatible with ultrafast HDF5 format
     Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging

import pandas as pd Introduction
pandas for CSV
Introduction
pandas.read_csv() CSV: Both read_csv() and read_table() use the same parsing code to intelligently convert tabular data into a DataFrame. pandas.read_csv(filepath_or_buffer, sep=NoDefault.no_default, delimiter=None, header='infer', names=NoDefault.no_default, index_col=None, usecols=None, squeeze=False, prefix=NoDefault.no_default, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal='.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, encoding_errors='strict', dialect=None, error_bad_lines=None, warn_bad_lines=None, on_bad_lines=None, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None, storage_options=None). code.
pandas.DataFrame.to_csv() CSV: DataFrame.to_csv(path_or_buf=None, sep=',', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, mode='w', encoding=None, compression='infer', quoting=None, quotechar='"', line_terminator=None, chunksize=None, date_format=None, doublequote=True, escapechar=None, decimal='.', errors='strict', storage_options=None). Write the contents of the Frame into a CSV file: code.
pandas.read_table() CSV: Both read_csv() and read_table() use the same parsing code to intelligently convert tabular data into a DataFrame. Read general delimited file into DataFrame. pandas.read_table(filepath_or_buffer, sep=NoDefault.no_default, delimiter=None, header='infer', names=NoDefault.no_default, index_col=None, usecols=None, squeeze=False, prefix=NoDefault.no_default, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal='.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, error_bad_lines=None, warn_bad_lines=None, on_bad_lines=None, encoding_errors='strict', delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None)[source]. code.
loc[] and iloc[] Introduction
describe() Print statistic summary of the data. (code)
.fillna() Replace empty cells with anything. (code)
.groupby() sort/group columns. (code). CSV: (code)
info() Print information in CSV. (code)
.to_datetime() Change data format. (code)
del     Introduction
head() (code)
path_or_buf CSV: A string path to the file to write or a StringIO
inplace=True (code)
.T (Transfer of array in Python) Introduction
predict() Introduction.
password() password(text='', title='', default='', mask=''). (code)
parse_dates CSV: if True then index will be parsed as dates (False by default). You can specify more complicated
options to parse a subset of columns or a combination of columns into a single date column (list of ints or names, list of lists, or dict) [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column [[1, 3]] -> combine columns 1 and 3 and parse as a single date column {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result ‘foo’
pop() Removes the item at the given index from the list and returns the removed item. (code)
sys.platform
(code)
sys.path
Is a list of strings that specifies the search path for modules. Basically this tells Python what locations to look in when it tries to import a module.  code.
sys.platform Is a platform identifier. You can use this to append platform specific modules to sys.path, import different modules depending on platform or run different pieces of code. code.
from PIL import ImageTk  
tkinter popup windows Introduction
.place() (code)
.pack() (code). (code).
PhotoImage() (code)
pady= (code)
padx= (code)
PyAutoGUI
pyautogui.position() (Code)
press() Several key strings which will be pressed down in order, and then released in reverse order. Strings can be passed to this function: '\t', '\n', '\r', ' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e','f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~', 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', 'down', 'end', 'enter', 'esc', 'escape', 'execute', 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', 'shift', 'shiftleft', 'shiftright', 'sleep', 'space', 'stop', 'subtract', 'tab', 'up', 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', 'command', 'option', 'optionleft', 'optionright', 'spacebar'. (code)
hotkey()
keyDown()
keyUp()
winleft and winright

Introduction

.pixelMatchesColor() Introduction
pyautogui.PAUSE = 1 Wait 1 second after performing an action
pyautogui.FAILSAFE "pyautogui.FAILSAFE = True": moving the mouse cursor to the upper-left corner of the screen will cause PyAutoGUI to raise the pyautogui .FailSafeException exception. "pyautogui.FAILSAFE = False": disable this feature.
enter Press the enter key or add "\n" in .typewrite function. Introduction
popup windows with pyautogui Introduction
pyautogui.pixel(x,y)[] e.g. pyautogui.pixel(30,60)[0] == 0, here, [0] = R (red) value, [1] = G (green) value, [2] = B (blue) value.
Mouse clicks (code)
mouseUp() Move the mouse and then release it. .mouseUp(x=moveToX, y=moveToY, button='left'). (code)

mouseDown()

Move the mouse and then release it. .mouseDown(x=moveToX, y=moveToY, button='left'). (code)
.middleClick() .middleClick(x=moveToX, y=moveToY) (Code)
dragTo() dragTo(x, y, duration=num_seconds) drags mouse to XY. (code).
dragRel() dragRel(xOffset, yOffset, duration=num_seconds) drags mouse relative to its current position. (code).
displayMousePosition() (code)
.moveRel() .moveRel(xOffset, yOffset, duration=num_seconds). (Code).
.moveTo(x, y, t) x and y: coordinates, and t: time. By default, pyautogui.MINIMUM_DURATION is 0.1. (Code)
.tripleClick() .tripleClick(x=moveToX, y=moveToY) (Code)
.typewrite() E.g. .typewrite('Hello world!\n', interval=secs_between_keys); .typewrite(['a', 'b', 'c', 'left', 'backspace', 'enter', 'f1'], interval=secs_between_keys). (Code)
.typewrite() E.g. .typewrite('Hello world!\n', interval=secs_between_keys); .typewrite(['a', 'b', 'c', 'left', 'backspace', 'enter', 'f1'], interval=secs_between_keys). (Code)
.rjust()

Right-justify/align them so that they take up the same amount of space, whether the coordinate has one, two, three, or four letters or digits. (code).

alert() alert(text='', tilte='', button='Ok'). (code)
confirm() confirm(text='', title='', buttons=['OK', 'Cancel']) (code)
prompt() prompt(text='', title='', default=''). (code)
.locateOnScreen() .locateOnScreen('looksLikeThis.png') returns (left, top, width, height) on the image which the screenshot is taken from. Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. This function mostly is useless. Introduction. (code)
.locateCenterOnScreen() Uses pyscreeze. x, y = MySearch_img to get the x- and y-coordinates of centers of the feature. Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. Introduction. (code)
.locateAllOnScreen() Difference between pyautogui.locateOnScreen("anImage") and pyautogui.locateOnScreen("anImage", minSearchTime=): minSearchTime = amount of time in seconds to repeat taking screenshots and trying to locate a match. (code)
write() (Code)
.screenshot() (code)
PyKeyboard Attributes  
'a', 'b', 'c', 'A', 'B', 'C', '1', '2', '3', '!', '@', '#', and so on The keys for single characters
'enter' (or 'return' or '\n') The enter key
'esc' The esc key
'shiftleft', 'shiftright' The left and right shift keys
'altleft', 'altright' The left and right alt keys
'ctrlleft', 'ctrlright'

The left and right ctrl keys

'tab' (or '\t') The tab key
'backspace', 'delete' The backspace and delete keys
'pageup', 'pagedown' The page up and page down keys
'home', 'end'

The home and end keys

'up', 'down', 'left', 'right' The up, down, left, and right arrow keys
'f1', 'f2', 'f3', and so on The F1 to F12 keys
'volumemute', 'volumedown', 'volumeup' The mute, volume down, and volume up keys (some keyboards do not have these keys, but your operating system will still be able to understand these simulated keypresses)
'pause'

The pause key

'capslock', 'numlock', 'scrolllock' The caps lock, num lock, and scroll lock keys
'insert' The ins or insert key
'printscreen' The prtsc or print screen key
'winleft', 'winright'

The left and right win keys (on Windows)

'command' The Command (z) key (on OS X)
'option' The option key (on OS X)
ppt
from pptx import Presentation (code)
from pptx.util import Inches (code)
from pptx.util import Pt (code)
from pptx.dml.color import RGBColor (code)
Add and work on a new slide
.slide_layouts[0] Introduction
.slide_layouts[1] Introduction
.slide_layouts[2] Introduction
.slide_layouts[3] Introduction
.slide_layouts[4] Introduction
.slide_layouts[5] Introduction
.slide_layouts[6] Simple pptx slide generation. Introduction
.slide_layouts[7] Introduction
.slide_layouts[8] Introduction
.slide_layouts[9] Introduction
.slide_layouts[10] Introduction
.slide_layouts[11]  
Presentation() (code)
paragraphs[] (code)
from pptx.enum.shapes import MSO_AUTO_SHAPE_TYPE (code)
Quit from Powerpoint show (code)
   
from pptx.chart.data import ChartData (code)
from pptx.enum.chart import XL_CHART_TYPE (code)
.categories (code)
.add_series() (code)
.add_chart() (code)
.LINE (code)
.chart (code)
.  
.series (code)
.format.line (code)
Image rotation in pptx (.rotation=) Introduction.
Text Format (e.g. ALIGN) in pptx PP_ALIGN has CENTER, DISTRIBUTE, JUSTIFY, JUSTIFY_LOW, LEFT, RIGHT, THAI_DISTRIBUTE, and MIXED, and MSO_VERTICAL_ANCHOR has TOP, MIDDLE, BOTTOM, MIXED. Introduction
MyPresentation.save() (code)
 
from PIL import Image (code)
.placeholders[] (code)
pass() Means "Do nothing": Tell Python that you’re just not quite ready to finish writing the class, so that it does't throw an error message. Instruction.
__pos__  
__pow__  
patterns (Read-only) Patterns to allow matching event paths.
path The path that this watch monitors.
class watchdog.observers.polling.PollingObserver(timeout=1)
class watchdog.observers.polling.PollingObserverVFS(stat, listdir, polling_interval=1)

path(id)

Returns path for id. None if id is unknown to this snapshot.

paths

Set of file/directory paths in the snapshot.

math.pi Returns the mathematical constant pi (3.141592 . . .).
math.pow(x, y) Returns x raised to the power y. code1, code2.
pow(x, y, z) x raise to the power y and reminder by z.
patterns=['*.*'] (code)
.PatternMatchingEventHandler (code)
os.path.join(path Introduction. code. code. (code)
os.path.exists() (code) (code)
path     code. (code)
.pack code. code.
Floating Point It can store number with a fractional part
property The word are synonymous with attribute. code. code. Change a attribute, code.
os.path.splitext() method Is used to split the path name into a pair root and ext. Here, ext stands for extension and has the extension portion of the specified path while root is everything except ext part. ext is empty if specified path does not have any extension. If the specified path has leading period (‘.’), it then will be ignored. Introduction. Examples are:
path name root ext
/Desktop/file.txt /home/User/Desktop/file .txt /Desktop /home/User/Desktop {empty}
file.py file .py
.txt .txt {empty}
pyperclip.paste() (code). (code).
Py2exe (code)
pyinstaller (code)
scipy.linalg.pascal Create a Pascal matrix.
linalg.pinv(a[, rcond, hermitian]) Compute the (Moore-Penrose) pseudo-inverse of a matrix.
Pyglet

Is an excellent choice for an object-oriented programming interface in developing games. In fact, it also finds use in developing other visually-rich applications for Mac OS X, Windows, and Linux. In the 90s, when people were bored, they resorted to playing Minecraft on their computers. Pyglet is the engine behind Minecraft.

PyGame Provides an extremely easy interface to the Simple Directmedia Library (SDL) platform-independent graphic, audio, and input libraries.
Python Twisted Is an event-driven networking engine, Twisted is written in Python, and licensed under the open-source MIT license.
PIL

Pillow is a friendly fork of PIL (Python Imaging Library), but is more user-friendly. If you work with images, then Pillow is your best friend. code1, code2.

from PIL import Image code1, code2.
skimage.measure.perimeter(image[, neighbourhood]) Calculate total perimeter of all objects in binary image.
skimage.measure.points_in_poly Test whether points lie inside a polygon.
skimage.measure.profile_line(img, src, dst) Return the intensity profile of an image measured along a scan line.
import pyperclip (code)
pywin32 Provides useful methods and class for interaction with Windows, as the name suggests.
PyGTK Lets you easily create programs with a GUI (Graphical User Interface) with Python.
Pipenv Is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. The cardinal purpose is to provide users with a working environment which is easy to set up. Pipenv, the “Python Development Workflow for Humans” was created by Kenneth Reitz for managing package discrepancies.
   
   
Pydot Is used for generating complex oriented and non-oriented graphs. It is specially used while developing algorithms based on neural networks and decision trees.
PyOD As the name suggests, it is a Python toolkit for detecting outliers in multivariate data. It provides access to a wide range of outlier detection algorithms. Outlier detection, also known as anomaly detection, refers to the identification of rare items, events or observations which differ from the general distribution of a population.
Pint  
Pynance Work wonders for a stock market trader. It is an open-source python package that retrieves, analyses and visualizes the data from stock market derivatives. With this library in hand you can generate labels and features for machine learning models.
Pyfolio Evaluates the trading performance with the help of Pyfolio. It provides risk analysis reports and performance results of financial portfolios based on the returns. This was developed by Quantopian and works well with Zipline which is a backtesting library (will discuss later). Pyfolio specializes in creating tear sheet models and bayesian analysis.
cv2.putText()

cv2.putText(image, text, org, font, fontScale, color[, thickness[, lineType[, bottomLeftOrigin]]]) is used to draw a text string on any image. The font types are FONT_HERSHEY_SIMPLEX = 0, FONT_HERSHEY_PLAIN = 1, FONT_HERSHEY_DUPLEX = 2, FONT_HERSHEY_COMPLEX = 3, FONT_HERSHEY_TRIPLEX = 4, FONT_HERSHEY_COMPLEX_SMALL = 5, FONT_HERSHEY_SCRIPT_SIMPLEX = 6, FONT_HERSHEY_SCRIPT_COMPLEX = 7, and FONT_ITALIC = 16. The thickness of the line is in pixel. lineType: This is an optional parameter.It gives the type of the line to be used. bottomLeftOrigin: This is an optional parameter. When it is true, the image data origin is at the bottom-left corner. Otherwise, it is at the top-left corner. code. (code)

   
   
   
   
   
   
   
.random.permutation() code.
Search/print/output the rows CSV: Print the rows if their cell values are greater than a specific value, in the csv file with numbers only; output the rows if the cell value is in a specific range. Introduction. code.
PyBrain Contains algorithms for neural networks that can be used by entry-level students yet can be used for state-of-the-art research. The goal is to offer simple, flexible yet sophisticated and powerful algorithms for machine learning with many pre-determined environments to test and compare your algorithms.
t.penup()/t.pu()/t.up() Pull the pen up – no drawing when moving. (code)
t.pendown()/t.pd()/t.down() Pull the pen down – drawing when moving. (code)
.getcanvas().postscript(file="") Picture saved in current working directory. (code)
.set_window_position() (code)
.position (code)
from pynput.mouse import Button (code)
from pynput.mouse import Controller (code)
.press(Button.left) (code)
.press(Button.right) (code)
space bar on keyboard Press space bar. Introduction
PyScreeze Is a simple, cross-platform screenshot module for Python 2 and 3.
hotkey('p') Introduction
press(pagedown) Introduction
press(pagedown) Introduction
PyGetWindow A simple, cross-platform module for obtaining GUI information on application's windows.
.Popen() Opens a pipe from a command. This pipe allows the command to send its output to another command. Introduction
from time import perf_counter (code)
.button_pressed() (code)
os.path.getctime (code)(code)
make_pipeline (code).
PySimpleGUI27 (code).
.switch_to. and .switch_to.parent_frame() Introduction
Pt E.g. run.font.size = Pt(15). Inches, Emu, Cm, Mm, Pt, and Px are base class for length classes, providing properties for converting length values to convenient units.
Px Inches, Emu, Cm, Mm, Pt, and Px are base class for length classes, providing properties for converting length values to convenient units.
.slide_width (code).
slide_height (code).
os.path.basename() (code).
os.path.abspath() (code).
os.path.dirname() (code).
os.path.expanduser() (code).
os.path.expandvars() (code).
os.path.normcase() (code).
os.path.normpath() Normalize/format the path string into a proper string for the OS (code).
os.path.lexists() (code)
os.path.isabs() (code)
os.path.isfile() (code)
os.path.isdir() (code)
os.path.islink() (code)
os.path.ismount() (code)
os.path.getatime() (code)
os.path.getmtime() (code)
os.path.samefile() (code)
os.path.getsize() (code)
   

 

=================================================================================