Simulating the Mouse and Key Strokes

There are a number of tools that allow you to simulate mouse moments and key stokes. Applications for this could include:

  • testing of a user interface
  • communicating with an app where there isn’t an API
  • “Visual Macros” where a sequence of keys and entries are repeated

For this blog I will mostly focus on Linux but there are also tools in the Microsoft Windows world.

Microsoft Windows – Sendkey

Sendkey is a command line program that can be used in a batch file or Powerscript to do visual key and mouse automation.

Some useful short cut keys include:

  • Alt+Tab: Switch apps.
  • Alt+F4: Close apps.
  • Win+left arrow or Win+right arrow: Snap windows.
  • Win+Tab: Open the Task view.
  • Tab and Shift+Tab: Move backward and forward through options.
  • Ctrl+Esc: Open the Start menu.

The VBA Application object also supports the Sendkey method so it is possible to write scripts in standalone VBA or in Word or Excel.

Linux Mouse and Key Stroke Simulation

In the Linux environment there a number of choices, such as xautomation and xdotools.

I thought that the xdotools package was feature rich with a number of special functions for desktop and windows functions. The xautomation package was a little simpler and it has some Python wrappers.

I found that it was very useful to install wmctrl, this allows you to easily see which windows are running and it can set the active window with a substring (you don’t need the full window name).

To install xautomation and wmctrl in Ubuntu:

sudo apt-get install xautomation wmctrl

Use wmctrl -l to see which windows are open.

A specific window can have the focus based on a substring.

An example bash script to open a file (a Python file in Idle) and then write some text would be:

# Open a file, set it to the focus and write some text
testfile="dummy.py"

idle $testfile &

sleep 2 #Give Idle time to come up

wmctrl -a $testfile  #set the focus to dummy.py

xte 'str #this is a line from script'

Test a Visual Interface

As a second example I wanted to test a GUI program. For the GUI app I put a bug in the program so when the second button was pressed 10 times the app would close.

# simtest.py - two button test program
#   exit program if the 2nd button is pressed 10 times

import tkinter as Tkinter

top = Tkinter.Tk()
btcount = 0

def CallBack2():
    global btcount
    btcount += 1
    if btcount == 10:
         quit()
    label.config(text = btcount)

label = Tkinter.Label(top, text = "0" )
label.pack()

B1 = Tkinter.Button(top, width=50,text ="Button 1")
B1.pack()
dummy
B2 = Tkinter.Button(top, width=50, text ="Button 2", command = CallBack2)
B2.pack()
# bind the return key to the 2nd button
top.bind('<Return>', lambda event=None: B2.invoke())
top.mainloop()

The next step is to make a bash script that hits the two button indefinitely and then checks to see if the program is still running it. Finally if the program crashes the script needs to show some diagnostic information.

There are a lot of options on how to write this script, below is just one example:

# simkey2 - push buttons on a program and check app status

testfile="simtest.py"

# run the test program
python3 $testfile &

sleep 2 #give the program time to come up

wmctrl -a $testfile  #set the focus to the test file

# Loop until the test app crashes
while true; do
  xte  'key Tab' 'key Return'
  sleep 2
  # see if the app is still running
  isRun=$(ps -eo pid,cmd | grep $testfile | grep -v grep)

  # Quit if the app is no longer running
  #echo ${#isRun} $isRun 
  if [  ${#isRun} -eq 0 ]
  then
     clear
     echo "-----------------------------"
     echo "| "  $testfile  " crashed"
     echo "-----------------------------"
     break
  fi
  
done

Final Comments

Using simulated keyboard and mouse movements is a nice tool to have in your back pocket when you are looking at “closed” or older systems.

However there are some inherent problems when you try to simulate mouse and key boards such as:

  • You take over the PC. It can be challenging to do other functions
  • Doing a TAB to select the next enter-able area works great for “RETURN” key inputs, but it may not work for buttons that expect a mouse click.
  • Your testing script can not “see” the results so some thinking is required to determine if the operations at working correctly.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s