Automate Windows Applications using Python

This article was published as a part of the Data Science Blogathon

Introduction

Performing repetitive tasks is very boring and hectic. But what if you can automate those stuff and let a script do that boring task. It is amazing, isn’t it? So, in this article, we are going to learn how to automate Windows applications. Because if you can automate such repetitive tasks you can save your precious time for other important things. Automation is also increasing day by day because there are many things that are repetitive in nature and if they get automated they save your efforts and in many cases huge manpower involved in those tasks.

Image 1

Automation offers great help to reduce workload and also there are the scripts working 24×7 without any human intervention. So, whenever a machine comes into pla

Author

y, work automatically becomes more precise, fast, and error-free.

Applications of Automation

Before moving forward let’s look at some very popular automation use cases:

1. Automated help desk support
   Various companies or banks support help desks where a person asks questions in email, SMS, Slack, etc. and they immediately get the response.
2. Form Autofill:
   This boring task of filling online forms and resetting forgotten passwords etc. can be easily handled by automated scripts.
3. Automatic mail triggers daily:
   This helps where you have to mail an attachment on daily basis. An automated script scheduled for daily will easily manage this thing.
4. Quick Weather updates
   You can check the weather easily and quickly too, but if it gets to you automatically then it’s more convenient. So, this is also an example of automation where you get notifications of daily weather update automatically.

These are just a few examples of automating boring things which ultimately makes your life a little comfortable. But in this article, we are going to automate windows applications.

So, to automate the windows application we are going to use Python. Python offers great readability and easy-to-learn syntax. In Python, we are going to use the Pywinauto module whose sole purpose is to automate windows applications. So, let’s begin.

You can refer to Pywinauto documentation here.

Command to install Pywinauto : pip install pywinauto

We will look at some examples in this article. Firstly, we will start with Notepad automation which is quite basic. After that, we will automate the excel application. In excel, we will try to read data from cells and writing data to cells. And we will also look at some more examples ahead. This we will be doing with Python and Pywinuato.

Once we have installed pywinauto, the first thing to do is to determine which accessibility backend of pywinauto to be used for our application. There are basically two supported backends in windows:

1. Win32 API:

It is the default backend used by pywinauto.

If you want to use it just specify as backend = “win32” in the Application() method of the pywinauto.

2. MS UI Automation:

It is a much-preferred backend and also when you want to access elements from the window it is quite easy with this backend.

To use this backend just specify backend as backend  = “uia” in the Application() method of the pywinauto.

Automating Notepad using Pywinuato

By looking at this example we will be learning the basics of automation using Pywinuato. We will be looking at some concepts like how to start an application, how to maximize it, how to write text in a notepad.

Start with importing the required files.

from pywinauto import application

app = application.Application()
app.start("Notepad.exe")

We need to extract the top window of our notepad app, so there are multiple ways to do it.

We can get a top window as:

app.windows()[0]: It returns the list of all windows and in this list first window is what we need, so just snap it from here.

This is the whole window we get by printing the app.windows() and on index 0 is the window we need.

app.top_window(): It will only return you the top window of all the opened apps and it is our notepad window that we need.

app[‘Untitled – Notepad’]: This is one more way to get the window we need. “Untitled – Notepad” is the name of the window which opens up on execution of app.start() method.

With this, our notepad will open up. Now let’s keep it simple and we will try to write something in the text area. So, for that simply use a type_keys() method. Whatever you want to type in the text area just pass it as a string inside type_keys() method.

text_message = "Hi Aliens! Text coming from pywinauto!"
dlg.type_keys(text_message, with_spaces=True)

Output:

 

Most of the time, the notepad window opens up as a small screen, so if we want to maximize it we can just simply use the is_maximized() method. It will return false if the window is not in a fully maximized state.

if dlg.is_maximized() == False:
    dlg.maximize()

Automating Excel Application

Now, our basics are now in place. So, let’s go to the next step and work on automating excel. This is going to be interesting because we will look at how you can read data from excel, how you can send data in an excel sheet. Let’s start with opening our excel file.

program_path = r"C:Program Files (x86)Microsoft OfficerootOffice16EXCEL.EXE"
file_path    = r"C:UsersDesktopsample.xlsx"

Firstly, start our application. In the case of excel it is quite different because we need to open our xlsx file through excel.exe, so we can do it like this:

app = application.Application(backend="uia").start(r'{} "{}"'.format(program_path, file_path))

This starts our application. Now, we need to select the window of sample.xlsx because we have to work with that, so as discussed above we will go like that.

print(app.windows())

Output:

[, , , , ]

So, we can either select 0 index window or we can get our window by directly giving the name.

dlg = app['sample.xlsx - Excel']

Now, let’s say we want to know what all attributes or control elements are there in the window of the opened app. Then we can use print_control_identifiers() method. All the possible “best_match” names are displayed as a Python list for every control in a tree form.

Pywinauto supports the most common GUI control elements, including:

• Button
• CheckBox
• RadioButton
• GroupBox
• ComboBox
• Edit
• Header
• ListBox
• ListView
• PopupMenu
• Static
• StatusBar
• TabControl
• Toolbar
• ToolTips
• TreeView
• UpDown

dlg.print_control_identifiers()

Output:

Control Identifiers:
Dialog - 'sample.xlsx - Excel'    (L-8, T-8, R1374, B647)
['Dialog', 'sample.xlsx - Excel', 'sample.xlsx - ExcelDialog']
child_window(title="sample.xlsx - Excel", control_type="Window")
   |
   | Pane - 'DropShadowTop'    (L0, T161, R1366, B167)
   | ['DropShadowTop', 'DropShadowTopPane', 'Pane', 'Pane0', 'Pane1']
   | child_window(title="DropShadowTop", control_type="Pane")
   |
   | Pane - ''    (L0, T617, R1366, B639)
   | ['Pane2']
   |    |
   |    | Toolbar - ''    (L0, T617, R1366, B639)
   |    | ['Toolbar', 'Toolbar0', 'Toolbar1']
   |    |    |
   |    |    | Pane - 'Status Bar'    (L0, T617, R1366, B639)
   |    |    | ['Status BarPane', 'Pane3', 'Status Bar', 'Status Bar0', 'Status Bar1']
   |    |    | child_window(title="Status Bar", control_type="Pane")
   |    |    |    |
   |    |    |    | Pane - ''    (L0, T617, R1366, B639)
   |    |    |    | ['Pane4']
   |    |    |    |    |
   |    |    |    |    | Pane - ''    (L0, T617, R1366, B639)
   |    |    |    |    | ['Pane5']
   |    |    |    |    |    |
   |    |    |    |    |    | StatusBar - 'Status Bar'    (L0, T617, R1366, B639)
   |    |    |    |    |    | ['StatusBar', 'Status Bar2', 'Status BarStatusBar']
   |    |    |    |    |    | child_window(title="Status Bar", control_type="StatusBar")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | Button - 'Cell Mode Ready'    (L0, T618, R47, B639)
   |    |    |    |    |    |    | ['Button', 'Cell Mode ReadyButton', 'Cell Mode Ready', 'Button0', 'Button1']
   |    |    |    |    |    |    | child_window(title="Cell Mode Ready", control_type="Button")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | TabItem - 'Normal'    (L1066, T618, R1106, B639)
   |    |    |    |    |    |    | ['TabItem', 'NormalTabItem', 'Normal', 'TabItem0', 'TabItem1']
   |    |    |    |    |    |    | child_window(title="Normal", control_type="TabItem")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | TabItem - 'Page Layout'    (L1106, T618, R1146, B639)
   |    |    |    |    |    |    | ['TabItem2', 'Page Layout', 'Page LayoutTabItem', 'Page Layout0', 'Page Layout1', 'Page LayoutTabItem0', 'Page LayoutTabItem1']
   |    |    |    |    |    |    | child_window(title="Page Layout", control_type="TabItem")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | TabItem - 'Page Break Preview'    (L1146, T618, R1186, B639)
   |    |    |    |    |    |    | ['TabItem3', 'Page Break Preview', 'Page Break PreviewTabItem']
   |    |    |    |    |    |    | child_window(title="Page Break Preview", control_type="TabItem")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | Button - 'Zoom Out'    (L1187, T618, R1205, B639)
   |    |    |    |    |    |    | ['Button2', 'Zoom Out', 'Zoom OutButton', 'Zoom Out0', 'Zoom Out1', 'Zoom OutButton0', 'Zoom OutButton1']
   |    |    |    |    |    |    | child_window(title="Zoom Out", control_type="Button")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | Slider - 'Zoom'    (L1205, T618, R1305, B639)
   |    |    |    |    |    |    | ['Slider', 'ShareSlider']
   |    |    |    |    |    |    | child_window(title="Zoom", control_type="Slider")
   |    |    |    |    |    |    |    |
   |    |    |    |    |    |    |    | Button - 'Zoom Out'    (L1205, T618, R1253, B639)
   |    |    |    |    |    |    |    | ['Button3', 'Zoom Out2', 'Zoom OutButton2']
   |    |    |    |    |    |    |    | child_window(title="Zoom Out", control_type="Button")
   |    |    |    |    |    |    |    |
   |    |    |    |    |    |    |    | Thumb - 'Position'    (L1253, T623, R1258, B634)
   |    |    |    |    |    |    |    | ['Thumb', 'PositionThumb', 'Position', 'Thumb0', 'Thumb1', 'PositionThumb0', 'PositionThumb1', 'Position0', 'Position1']
   |    |    |    |    |    |    |    | child_window(title="Position", control_type="Thumb")
   |    |    |    |    |    |    |    |
   |    |    |    |    |    |    |    | Button - 'Zoom In'    (L1258, T618, R1305, B639)
   |    |    |    |    |    |    |    | ['Button4', 'Zoom In', 'Zoom InButton', 'Zoom In0', 'Zoom In1', 'Zoom InButton0', 'Zoom InButton1']
   |    |    |    |    |    |    |    | child_window(title="Zoom In", control_type="Button")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | Button - 'Zoom In'    (L1305, T618, R1323, B639)
   |    |    |    |    |    |    | ['Button5', 'Zoom In2', 'Zoom InButton2']
   |    |    |    |    |    |    | child_window(title="Zoom In", control_type="Button")
   |    |    |    |    |    |    |
   |    |    |    |    |    |    | Static - 'Zoom 100%'    (L1323, T618, R1366, B639)
   |    |    |    |    |    |    | ['Static', 'Zoom 100%Static', 'Zoom 100%', 'Static0', 'Static1']
   |    |    |    |    |    |    | child_window(title="Zoom 100%", control_type="Text")

This tree is not complete here, it will go much ahead. You can run the above code and check by yourself.

Before moving further reading or writing data in an excel sheet. I want to give you a brief introduction to the UI Inspection tool. In this article, I will be covering Inspect.exe but there are other tools also like UISpy.exe, etc.

Inspect.exe

It is a great tool created by Windows. You can download it from the above hyperlink of inspect.exe. Once you open this tool switch to  MS UI automation mode which if what we need.

Inspect.exe is an application that will give the ids by which you can extract identifiers of any element from the UI. So, in the inspect.exe tool just looking for Automation ID and by this, we can get access to any element from the application window.

For example, If you want the automation ID of the A1 cell in the excel sheet just click your mouse on that cell and you will get the automation ID.

 

So, the automation ID of cell A1 is A1. Hence we can now use it. Now, we will use these automation ids to read and write data in an excel sheet.

Automating Reading Data from excel sheet

We will be reading this data from an excel file and print it as a sentence.

Since “We” is written in cell A1 and its automation id is also A1, so now it is quite obvious that the automation id of the B1 cell is B1, for C1 cell is C1, and so on.

So, to read data below is the code:

text = ""
for i in range(65, 65+5):
    e_id = str(chr(i))+str(1)
    dlg.child_window(auto_id = e_id).click_input()
    ei = dlg.child_window(auto_id=e_id)
    ei.click_input()
    text+=(ei.legacy_properties()['Value'])
    text+=" "
print("Text read from sample.xlsx is:",(text))

Output:

Text read from sample.xlsx is: We are reading this data.

Explanation

Inside for loop, e_id is the automation ID we are making for each cell. child_window() is used here to get a particular element where automation ID is passed as a parameter. Then we do a mouse click on in using click_input() to grab that element. Now, we want to read the value from the cell in excel so for that use legacy_properties()[‘Value’].

Now, let’s move forward and try to write in the next line of the above excel sheet.

Automating Writing data in an excel sheet

We will write data in the B3 cell of the excel sheet. Here also you can use the inspect.exe tool to grab its automation ID. Then use this code to write data.

write_data = "This is written using pywinauto. Amazing!"
window = dlg.child_window(auto_id="B3")
window.click_input()
window.type_keys(write_data, with_spaces=True)

Output:

Explanation:

write_data is the string which we need to write. In the second we again grab that element and click on it using click_input(). Now, to write data in that B3 cell type_keys() method is used. It takes 2 parameters, one is the string to be written and the other is to check for space i.e with_spaces if kept true spaces will also be typed while writing the string data.

Till now, we saw pywinauto use cases on notepad and excel applications. We will now see its use case on the calculator. But before moving ahead let me make you aware of some more concepts in the pywinauto.

Automating Keyboard and Mouse Inputs in Windows

Let’s see in detail how can we give keyboard inputs and mouse inputs like click, double click, right-click, left-click, etc.

To emulate keyboard inputs send_keys() module is used.

• to press enter: send_keys(‘{ENTER}’)  
• control key of keyboard (ctrl) is represented as: ‘^’
• alt key of keyboard is represented as: ‘%’
• shift key of keyboard is represented as: ‘+’

Some examples on how to use these:

• To press Ctrl + A: send_keys(‘^a’)
• To press ALT+F4: send_keys(‘%{F4}’)
• To press Ctrl+Alt+V: send_keys(‘^%V’)
• To press Ctrl+A i.e. select all and then Ctrl+C (copy to clipboard): send_keys(‘^a^c’)

For clicking with mouse you can use these methods in the pywinauto:

• click_input(): clicks at an element
• pywinauto.mouse.click(button=’left’, coords=(100,100)): single left click at the specified coordinates
• pywinauto.mouse.double_click(button=’left’, coords=(100,100)): double left click at the specified coordinates
• pywinauto.mouse.right_click(coords=(100,100)): right click at the specified coordinates
• pywinauto.mouse.scroll(coords=(100,100), wheel_dist=2): it performs mouse wheel.

Capturing image using pywinauto

Suppose you want to do debugging where an exception gets raised, in that case, you can capture the image of the window which is in front. This can be done by using the capture_as_image method. You can trigger this method and save the resulting PIL image into a folder. Let’s take a look at how you can do this but make sure to create an image_folder in the same directory where your script is running:

import datetime
dlg.set_focus()
debug_image = dlg.capture_as_image()
debug_image.save('image_folder/' + str(int(datetime.datetime.now().timestamp())) + '.jpg')

Note: This code brings the application to the foreground and then it captures the image based on the position. So, it is possible that captured image may contain some part of the other program or another window that was in the foreground.

Now, let’s take one more use case of the pywinauto i.e. calculator.

Automating Calculations on Calculator in Windows

Step 1: Do the required imports from Pywinauto and Desktop module.

from pywinauto import Desktop, Application

Step 2: As we have seen in notepad and excel, we need to start our application using the start method of application.

app = Application(backend = "uia").start('calc.exe')

Step 3: Now, if you want to print controls of the calculator as tree view:

dlg = Desktop(backend="uia").Calculator
dlg.print_control_identifiers()

Step 4: Giving input to the calculator as adding up two numbers 3 and 5

dlg.type_keys("3+5")
dlg.type_keys("=")

After this if you want to minimize your calculator window or open up it again, you can use this:

dlg.minimize()
Desktop(backend="uia").window(name='Calculator', visible=None).restore()

Apart from all these things Pywinauto is also widely used for GUI Testing of windows apps.

GUI Testing of Windows Apps

Graphical user interface testing is a very important part of quality assurance testing. It is important because it lets you check your application for any bugs or errors so that the user experience becomes smooth and easy. According to Mike Cohn’s book Succeeding with Agile, UI testing is at the top of the test automation pyramid.

                                           Image 2

This image shows that UI tests take less time and effort as compared to unit and service tests. But the major drawback is UI test can only be done at the latest stages of the development lifecycle. But since, the GUI testing of applications is from the user’s perspective so it is quite helpful and important.

In this article above we automated some of the apps like notepad, excel, calculator, those same concepts can be used for GUI automation testing also. Let me give you brief points here also for automation.

1. Run the application you want to test. Or you can also connect with the already running application using its process id.

2. Now define the main application window.

3. Find the necessary control elements like text-field, drop-downs, navigation bar, buttons, etc. from the application.

4. Perform the user action on the element. For example, click on a text box and add some text there.

5. Analyze the results.

You can see these steps are exactly the same as what we did above.

We studied pywinauto for windows app automation. There are many other modules in python for automating various kinds of automation tasks. Let’s have a look at them also.

Other Automation Modules in Python

There are different types of automation libraries in Python.

• Full-stack automation
• Graphical User Interface Automation
• Test Automation

Full-stack automation libraries:

1. Selenium:

• It is one of the most popular modules used in Python to perform testing and also lets you connect to different browsers.
• This module is mainly used to automate the web browser interaction by just using Python.
• You can execute the entire test suite with Selenium.
• It also gives you the flexibility to pause and resume a test case.

2. Requests:

With the help of the request module, you can send HTTP requests like GET, POST, PUT, DELETE, PUT. So it helps you when there are APIs involved in automation.

3. BeautifulSoup

• It is considered the best library for extracting data from HTML and XML files.
• It is a great tool for web scrapping.  It is very helpful if used on a simple static webpage where you have to find some information hidden deep inside HTML code.
• Overall, it’s a great library.

GUI automation:

This we have discussed in this article. It can be done with these modules like Pywinauto and Pyautogui.

1. Pyautogui:

• It allows our python scripts to control the mouse and keyboard so that interaction with other applications can happen.
• We can easily take screenshots, send keystrokes, etc.
• Pyautogui works on almost all operating systems.

Test automation libraries:

1. Pytest:

• It is a framework where you can write test codes using python language. Anything can be tested like API, the database is in some cases UI as well.
• Mainly it is used for writing tests for APIs.

2. Robot framework:

• This is a generic open-source framework for automation.
• It is widely used for test automation and robotic process automation.
• This framework is basically python-based , but can also be used with Jython i.e. Java, and IronPython i.e.. NET.

3. Pandas:

• It is an open-source library that gives a good range of tools for data analysis and manipulation.
• It is widely used to read data from files such as CSV, SQL database, JSON files, Excel.
• It has many inbuilt methods for combining data, filtering time-series functionality, and much more.
• So, the pandas library is by default used for data-related tasks but can also be used to create more powerful and focused data tools.

So,  this is all about automating windows applications using the Python library Pywinauto.

If you face any difficulties working on pywinauto, you can use this StackOverflow link to post your queries or read answers to already posted questions.

End Notes

Gaurav Sharma

Love Programming, Blog writing and Poetry