Day 5

Python Control Mouse & Keyboard

Olly

Book

• Automate the Boring Stuff with Python
  • Translated Book
  • Online eBook
  • Online Course -Udemy
• Environment require Python 3.4+

Contents:

1. Basic python

• Statement Assignment
• print()
• str()
• for i in range(n)
• try: & exception:

2. Third-party modules

• pyautogui
• time

Install pyautogui Module

If you have native python you can install module as below :

cmd
pip install pyautogui

If you use conda you can install module as below :

cmd
conda install pyautogui

Check Modules Exist

You don't need to install time, because time is in standard library.

Without Error Massage!!

import pyautogui
import time

Let's Start!

pyautogui Module

• The pyautogui module can send virtual keypresses and mouse clicks to Windows

If pyautogui Too Fast/Out of Control

• Shutting down everything by logging out

How to Prevent

• Wait n second for each PyAutoGUI function call
• Moving the mouse cursor to the upper-left corner of the screen will raise the pyautogui.FailSafeException

  import pyautogui
  pyautogui.PAUSE = 1
  pyautogui.FAILSAFE = True
  

Controlling Mouse Movement

Find Cursor Location

• The upper-left corner of the screen (x, y) = (0, 0)
• The x-coordinates increase going to the right, and the y-coordinates increase going down
• All coordinates are positive integers

My Screen Size

pyautogui.size()

Return width & height of your screen

import pyautogui
pyautogui.size()

(1366, 768)

Store the width and height as variables for better readability in your programs

width, height = pyautogui.size()

Moving the Mouse

pyautogui.moveTo(x, y, duration=second)

pyautogui.moveRel(x_pixels, y_pixels, duration=second)

• duration = Move from point to point

import pyautogui
for i in range(5):
    pyautogui.moveTo(100, 100, duration=0.25)
    pyautogui.moveTo(200, 100, duration=0.25)
    pyautogui.moveTo(200, 200, duration=0.25)
    pyautogui.moveTo(100, 200, duration=0.25)

import pyautogui
for i in range(5):
    pyautogui.moveRel(100, 0, duration=0.25)
    pyautogui.moveRel(0, 100, duration=0.25)
    pyautogui.moveRel(-100, 0, duration=0.25)
    pyautogui.moveRel(0, -100, duration=0.25)

Getting the Mouse Position

pyautogui.position()

Return x and y of your cursor on screen now

pyautogui.position()

(791, 551)

Project : “Where Is the Mouse Right Now?”

Aim :

• Display the current x- and y-coordinates of the mouse cursor.
• Update these coordinates as the mouse moves around the screen.

Step 1: Import the Module

Step 2: Set Up the Quit Code and Infinite Loop

Step 3: Get and Print the Mouse Coordinates

Step 1: Import the Module

import pyautogui
print('Press Ctrl-C to quit.')

Press Ctrl-C to quit.

Step 2: Set Up the Quit Code and Infinite Loop

try:
    while True:
        # TODO: Get and print the mouse coordinates.
except KeyboardInterrupt:
    print('\nDone.')

Step 3: Get and Print the Mouse Coordinates

To erase text, print the \b backspace escape character

x, y = pyautogui.position()
positionStr = 'X: ' + str(x).rjust(4) + ' Y: ' + str(y).rjust(4)
print(positionStr, end='')
print('\b' * len(positionStr), end='', flush=True)

X:  122 Y:  446

For Overall

Pease donload WhereIsMouse.py

Excute WhereIsMouse.py with following instructions

Controlling Mouse Interaction

Clicking the Mouse

pyautogui.click(x, y, button='left')

pyautogui.mouseDown(x, y, button='left')

pyautogui.mouseUp(x, y, button='left')

pyautogui.doubleClick(x, y, button='left')

• Button could be left, right or middle to specify which you want

import pyautogui
pyautogui.click(100, 150, button='left')

Equal

import pyautogui
pyautogui.mouseDown(100, 150)
pyautogui.mouseUp(100, 150)

Dragging the Mouse

Drag		Move
pyautogui.dragTo(x, y, duration=0.25)	<==equal to==>	pyautogui.moveTo(x, y, duration=0.25)
pyautogui.dragRel(x_pixel, y_pixel, duration=0.25)	<==equal to==>	pyautogui.moveRel(x_pixel, y_pixel, duration=0.25)

Try to move your cursor on the chrome browser icon during program sleep

import pyautogui,time
time.sleep(3)
pyautogui.dragTo(500, 800, duration=1)

import pyautogui,time
time.sleep(3)
pyautogui.dragRel(500, 0, duration=1)

Before

After

Project : “Drawing with pyautogui”

Aim :

• Create Graphic Automatically

During program sleep :

Try to move the mouse cursor over the drawing program’s window and select pencial/brush

import pyautogui, time
time.sleep(10)
pyautogui.click()
distance = 200
while distance > 0:
    pyautogui.dragRel(distance, 0, duration=0.2)   # move right
    distance = distance - 5
    pyautogui.dragRel(0, distance, duration=0.2)   # move down   
    pyautogui.dragRel(-distance, 0, duration=0.2)  # move left
    distance = distance - 5
    pyautogui.dragRel(0, -distance, duration=0.2)  # move up

Scrolling the Mouse

The size of a scroll unit varies for each operating system and application

pyautogui.scroll(int)

import pyautogui
pyautogui.scroll(-300)

Oops!!! Something Wrong Above

We can modify the code as below to scroll the mouse successfully

import time, pyautogui
time.sleep(2); pyautogui.scroll(-300)

Working with the Screen

Getting a Screenshot

pyautogui.screenshot()

import pyautogui
im = pyautogui.screenshot()
im

Color of the Pixel at Coordinate in Image

.getpixel(tuple)

The return value from getpixel() is an RGB tuple of three integers

1. Amount of red in the pixel
2. Amount of green in the pixel
3. Amount of blue in the pixel

im.getpixel((0, 0))

(55, 44, 36)

Recognize Color as Eyes

pyautogui.pixelMatchesColor(x, y, (R, B, G))

If the pixel at the given x- and y-coordinates on the screen matches the given color,

pixelMatchesColor() function will return True

pyautogui.pixelMatchesColor(0, 0, (55, 44, 36))

True

pyautogui.pixelMatchesColor(0, 0, (56, 44, 36))

False

Project: "Extending the WhereIsMouse Program"

Aim

• Add Recognize function for WhereIsMouse program

Add below code into WhereIsMouse.py

pixelColor = pyautogui.screenshot().getpixel((x, y))
positionStr += ' RGB: (' + str(pixelColor[0]).rjust(3)
positionStr += ', ' + str(pixelColor[1]).rjust(3)
positionStr += ', ' + str(pixelColor[2]).rjust(3) + ')'

Download full code here WhereIsMouse_WithRBG.py

#! python
import pyautogui
print('Press Ctrl-C to quit.')
try:
    while True:
        # TODO: Get and print the mouse coordinates.
        x, y = pyautogui.position()
        positionStr = 'X: ' + str(x).rjust(4) + ' Y: ' + str(y).rjust(4)
        # Add Code Here!!!!!
        # Add Code Here!!!!!
        # Add Code Here!!!!!
        # Add Code Here!!!!!
        print(positionStr, end='')
        print('\b' * len(positionStr), end='', flush=True)
except KeyboardInterrupt:
    print('\nDone.')

Execute Program as Below

Controlling the Keyboard

Sending a String from the Keyboard

pyautogui.typewrite(String, second)

Try to open notepad editor during program sleep, pyautoguui will keyin Hello Olly!

import time, pyautogui
time.sleep(10)
pyautogui.click(100, 100)
pyautogui.typewrite('Hello Olly!', 0.25)

Key Names

pyautogui.typewrite(['KeyName',...], second)

Keyboard Key String	Meaning
'a', 'b', 'c', 'A', 'B', 'C', '1', '2', '3', '!', '@', '#', and so on	The keys for single characters
'left'	The left arrow keys

import time, pyautogui
time.sleep(10)
pyautogui.click(100, 100)
pyautogui.typewrite(['O', 'l', 'l', 'y', '!', '!', 'left', 'left', ' ', ',', 'H', 'I'], 0.25)

Keyboard Key String	Meaning
'a', 'b', 'c', 'A', 'B', 'C', '1', '2', '3', '!', '@', '#', and so on	The keys for single characters
'enter' (or 'return' or '\n')	The ENTER key
'esc'	The ESC key
'shiftleft', 'shiftright'	The left and right SHIFT keys
'altleft', 'altright'	The left and right ALT keys
'ctrlleft', 'ctrlright'	The left and right CTRL keys
'tab' (or '\t')	The TAB key
'backspace', 'delete'	The BACKSPACE and DELETE keys
'pageup', 'pagedown'	The PAGE UP and PAGE DOWN keys
'home', 'end'	The HOME and END keys
'up', 'down', 'left', 'right'	The up, down, left, and right arrow keys

Keyboard Key String	Meaning
'f1', 'f2', 'f3', and so on	The F1 to F12 keys
'volumemute', 'volumedown', 'volumeup'	The mute, volume down, and volume up keys (some keyboards do not have these keys, but your operating system will still be able to understand these simulated keypresses)
'pause'	The PAUSE key
'capslock', 'numlock', 'scrolllock'	The CAPS LOCK, NUM LOCK, and SCROLL LOCK keys
'insert'	The INS or INSERT key
'printscreen'	The PRTSC or PRINT SCREEN key
'winleft', 'winright'	The left and right WIN keys (on Windows)
'command'	The Command () key (on OS X) 'option' The OPTION key (on OS X)

Pressing and Releasing the Keyboard

pyautogui.keyDown(key)

pyautogui.keyUpd(key)

Below show how to hold the SHIFT key and press 4 to get "$"

import time, pyautogui
time.sleep(10)
pyautogui.click(100, 100)
pyautogui.keyDown('shift')
pyautogui.press('4')
pyautogui.keyUp('shift')

Hotkey Combinations

pyautogui.hotkey('ctrl', 'c',...)

Select ALL = Ctrl + a

Copy = Ctrl + c

import pyautogui
pyautogui.keyDown('ctrl')
pyautogui.keyDown('a')
pyautogui.keyDown('c')
pyautogui.keyUp('c')
pyautogui.keyUp('a')
pyautogui.keyUp('ctrl')

Equal

import pyautogui
pyautogui.hotkey('ctrl', 'a', 'c')

Exercise

Auto Painting

1. Get any picture from website
2. Import modules
3. Recognition color and cursor position
4. Painting on drawing program’s

import time,pyautogui

Answer

#! python
import time, pyautogui
time.sleep(5)

try :
    for i in range(30, 620, 4):
        for j in range(80, 650, 4):
            # Recognition color
            pixelColor = pyautogui.screenshot().getpixel((i, j))
            if pixelColor != (255, 255, 255):
                pyautogui.click(i+700, j) 
                pyautogui.dragTo(i+700+10, j+10) 
except KeyboardInterrupt:
    print('\nDone.')

Download full code here AutoDrawDog.py !!

def Day5End() :

     return 'Thank U ❤'