=================================================================================

The user can connect a computer running a GUI automation tool, such as SikuliX [1] or PyAutoGUI [2], to a target robot programming environment (e.g., an human-machine interfaces (HMIs) hosted on a teach pendant) to generate a program in the application environment. The Pyautogui is a cross-platform GUI automation Python module for human beings. The pyautogui module has the functions for simulating mouse clicks and movements, scrolling the mouse wheel, and typing keyboard (automation with keyboard and mouse). This saves time and allows for maintaining a unified code base, from which programs for proprietary target environments can easily be generated. PyAutoGUI is used for desktop GUIs. [4] In practice, stand-alone runtime scripts in pure python can be made using the PyAutoGUI package. The mouse can be controlled by using dedicated Python packages such as the PyAutoGUI library.

PyAutoGUI had been used to make lane change decision in autonomous driving. [10] The researchers found that closing any extra window other than the IDE was smooth in terms of the package’s performance. Having a dual monitor setup would sometime lead to the simulator opening in random co-ordinates in the screen. Therefore, they used single monitor for easier execution. Their package can automate mouse-clicks and keyboard strokes. The mouse functions of PyAutoGUI use x- and y-coordinates. For mouse clicks, they recorded the coordinates of the radio buttons that were to be clicked and passed it to PyAutoGUI for execution. The x-coordinates increase positively going to the right, and the y-coordinates increase positively going down. The resolution is how many pixels wide and tall your screen is. The screen’s resolutions are listed in page4484.

PyAutoGUI had been used to check the threshold limit of curves in pattern recognition. [11] In their applications, as soon as the threshold limit is crossed, the required mouse operation occur. This can be achieved with the mouse operations being connected with the python code. Operations such as scrolling, clicking, and traversing across the screen are also handled by this library.

In general, pyautogui can move the mouse and type keystrokes at an incredible speed. In some cases, it might be too fast for other programs to keep up with. Stopping the program can be difficult if the mouse is moving around on its own, preventing you from breaking the process. On the other hand, pyautogui’s performance can be low when replaying a large number of mouse movements so that it takes minutes to execute some routine jobs. [6]

In the operation of pyautogui script, a script can be told to wait with pyautogui.PAUSE or time.sleep after every function call. In the meantime, non-PyAutoGUI instructions will not have this pause.

In summary, PyAutoGUI can be used for the GUI automation below:
         i) Controlling mouse movement and interaction, including clicking, dragging and scrolling mouse, and getting the mouse position,
         ii) Analyzing the screenshot,
         iii) Image recognition,
         iv) Controlling the Keyboard,
         v) Hotkey combinations.

Table 4497. Application examples with PyAutoGUI.

======================================================

Positions of the mouse: code:
         
Output:         
           
         Clicked in the one in red:         
               

======================================================

Move the mouse to a position: code:
         

======================================================

Drag the mouse: code:
         

======================================================

Mouse clicks: code:
         

======================================================

Combination of mouse clicks and drag: code:
         

======================================================

Scroll the mouse wheel: code:
         

======================================================

Move the mouse: code:
         

======================================================

Drag the mouse: code:
         
Output:         
         

======================================================

Find a similar feature and then click it : code:
         
Output:         
         

[1] SikuliX. Available online: http://sikulix.com/ (accessed on 25 November 2020).
[2] PyAutoGUI. Available online: https://pyautogui.readthedocs.io/ (accessed on 25 November 2020).
[3] Anish Agarwal, Munther Dahleh, Devavrat Shah, Dylan Sleeper, Andrew Tsai, Madeline Wong, Zorro: A Model Agnostic System to Price Consumer Data, Computer Science: Social and Information Networks, arXiv preprint arXiv:1906.02420.
[4] Luca Carettoni. On insecure zip handling, rubyzip and metasploit rce (cve-2019-5624). https://blog. doyensec.com/2019/04/24/rubyzip-bug.html, Accessed March 19, 2020.
[5] Al Sweigart. 2019. PyAutiGUI documentation. https://pyautogui.readthedocs.io/en/latest/. Accessed: 2021-04-14.
[6] Siddhanth Venkateshwaran, Ellen Kidane, Luís Pina, Dynamic Software Updates for Unmodified Browsers through Multi-Version Execution, Programming Languages, arXiv:2106.04655.
[7] Tuo Zhou, Shih-Yuan Yu, Matthew Michaels, Fangzhou Du, Lawrence Kulinsky, Mohammad Abdullah Al Faruque, Establishing Digital Recognition and Identification of Microscopic Objects for Implementation of Artificial Intelligence (AI) Guided Microassembly, Computational Engineering, Finance, and Science, arXiv:2107.10823.
[8] T.-H. Chang, T. Yeh, and R. C. Miller, “GUI testing using computer vision,” in Proc. of the 28th Intl. Conf. on Human Factors in Computing Systems - CHI ’10. ACM Press, 2010, p. 1535.
[9] Kyle MacMillan, Tarun Mangla, James Saxon, Nick Feamster, Measuring the Performance and Network Utilization of Popular Video Conferencing Applications, arXiv:2105.13478v1.
[10] Mukesh Ghimire, Malobika Roy Choudhury, Guna Sekhar Sai Harsha Lagudu, Lane Change Decision-Making through Deep Reinforcement Learning, arXiv:1904.00231v2.
[11] Ajith Abraham, M. A. Jabbar, Sanju Tiwari, Isabel M. S. Jesus, Proceedings of the 11th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2019).
[12] Samarth Shah, Preliminary Wildfire Detection: A Dataset and Challenges, arXiv:2109.05083v1, 2021.
[13] "Pyautogui’s documentation".