Sikuli Script: Troubleshooting Problems with Scripts

One of Sikuli's appealing features is its relatively straightforward concept; ironically, that can make it all the more puzzling when things aren't working quite the way they should-- you can see your button right there on the screen, so why can't Sikuli?  This post has a couple of tips for new users for troubleshooting common script problems.

Note: as of this writing, the latest version of Sikuli is version 930, available in Windows as a set of files to be unzipped over the previous version, version 905.  Version 930 includes some significant fixes; if you're experiencing issues and using version 905, I'd recommend upgrading to version 930 as a first step.

If Sikuli can't find a graphic pattern:

1) Be sure to check any FindFailed error messages carefully.  I know, I know-- that may sound obvious, but when you're knee-deep in a scripting session and you've already worked through half a dozen "FindFailed" errors in the last hour, you may see the seventh pop up in Sikuli's log and immediately assume it's the same as the others.  A case in point from my own personal experience: after troubleshooting for some time to figure out why Sikuli wasn't finding a match for my graphic pattern, a closer inspection of the FindFailed error revealed that Sikuli wasn't looking for a graphic pattern at all-- for some reason it was looking for text (treating my pattern file's name as a string).  Among other things, a FindFailed error can also indicate a problem finding a png file in its expected location on disk.

2) If you're looking for the pattern in a defined region (as opposed to a search on the entire screen), make sure the region covers the intended area and fully contains the pattern you're looking for; one way to check this is by calling the highlight() method on your region-- e.g., my_defined_region.highlight(2). You can remove the call later on once you've finished debugging.

3) Don't be stingy with wait times.  Sikuli can interact with an application much faster than a human, and that's not always a good thing.  In some cases it may not find a pattern simply because it hasn't had time to draw completely (in response to some previous action).  Use a simple wait() statement to give the screen, region, etc. more time to draw before proceeding to the next step.

If Sikuli incorrectly identifies a match:

1) A mis-click near a pattern (close but not centered on it) may stem from re-drawing and or re-sizing a window or components in a window. Consider a scenario where Sikuli is waiting for a button to appear onscreen; as soon as it appears it registers the match.  Subsequently, the window finishes drawing, during which the placement of the button is adjusted to make room for other components.  Sikuli doesn't recognize this change in the button's location-- as far as it's concerned, the button is still where its pattern was first detected, and that's where Sikuli will try to interact with it.  Possible solutions in this case include inserting a fixed (designated in seconds) wait() statement or using another pattern as a cue to Sikuli that the screen has finished drawing.

2) If Sikuli keeps matching a pattern to items other than the one you're actually trying to isolate, first make sure your pattern is "focused"-- you generally want to avoid any nondescript surrounding white space that doesn't help Sikuli differentiate it from similar regions.  You can also increase the similarity setting for the pattern to require a closer match.  With your script loaded in Sikuli, bring up the application screen where a match for your pattern should be found and click on the pattern in your script, launching the Pattern Settings dialog.  The Matching Preview tab shows the current screen, highlighting any regions that match your pattern-- if you adjust the Similarity slider, the preview automatically updates to reflect the new value.  Yet another solution: limiting your search to a specific region around your target, excluding other potential matches outside the region.


The Matching Preview tab of the Pattern Settings dialog
For scripting issues not necessarily related to finding pattern matches, the popup() method can be used as a poor man's break point for debugging.  For example, you can use it to pause your script and print out the value of a counter variable in a for loop, a value evaluated in an if statement, etc.  The method can be removed from your final script once you've finished troubleshooting.

Finally, when all else fails or you're seeing completely inexplicable issues, save your work, shut down Sikuli, and check for any orphaned processes (in Windows, these appear as javaw.exe instances) before restarting.  I've seen scenarios where a script didn't seem to shut down properly (particularly when execution was interrupted with Alt + Shift + C) and behavior becomes flaky and unpredictable when I attempt to re-run the script or run another script.  Clearing out these orphaned processes and re-starting Sikuli sometimes helps in these cases.

For Beginners: Java Learning Resources

I've published a few posts on using Groovy in soapUI; as much as I'd like to believe my directions and writing are so clear that no further knowledge is required, I know that's probably not the case-- at the very least, a basic understanding of objects, variables, and other Java programming concepts are necessary, which means that readers who are genuinely new to scripting may have difficulty following some of the material in the posts.  I do want this blog to be a resource for absolute beginners; I may tackle a Groovy tutorial in future posts, but for the time being I thought I'd recommend a few resources for learning Java that I've found helpful.

A logical first step, of course, are the Java Tutorials available online and for download in multiple formats (HTML, mobi, and epub) on Java's home site here.  For beginners or near beginners, the Getting Started, Learning the Java Language, and Essential Java Classes tutorials are recommended.

For third-party publications, I've looked at three notable books on Java, each of which uses a different approach to learning the language.  Learn to Program with Java by John Smiley (Osborne/McGraw-Hill, 1st Edition - 2002) has a very gentle learning curve and an interesting framing premise: the book purports to follow an introductory Java class (taught by the author himself) as they work through creating a practical working Java application.  Head First Java by Kathy Sierra and Bert Bates (O'Reilly Media, 2nd Edition - 2005) takes a more irreverent approach, trying to inject a bit of humor into the topic.  Ultimately, it covers more material than the Smiley book, albeit at a faster pace with a more challenging learning curve.  Finally, there's Bruce Heckel's Thinking in Java (Prentice-Hall, 4th Edition - 2006); coming in at a breezy 1150 pages, this book is pretty in-depth, going beyond the language and syntax to explain what goes on "under the hood" in a Java program-- memory management, the timing of the creation of objects, etc.  This book covers everything you probably wanted to know about Java-- not to mention a lot of stuff you didn't know you wanted to know.  Generously, Mr. Heckel makes the 3rd edition of the book available for free online at his company's (MindView, Inc.) website here.  As I said, all of these books take very different approaches; if possible, try looking at a few sample pages of each to see which best matches your personality or learning style.

If you prefer learning via video, I highly, highly recommend Stanford University's Programming Methodology lecture series available on iTunes U.  Comprised of video recordings of an introductory computer science course, this one is easy to overlook due to its relatively nondescript title.  In fact, while the course is primarily intended to be an introduction to programming concepts in general (hence the name), Java is the language chosen to illustrate those concepts, so it turns out to be a pretty decent introduction to Java specifically.  The lecturer, Professor Mehran Sahami, seems to have genuine affection for his job-- he injects his lessons with plenty of humor and explains things in engaging, easily understandable terms.  My primary issue with the series is the video quality; while it's not a major problem most of the time, text from presentation slides is occasionally hard to read.  The series is popular on iTunes; I hope Stanford might consider re-recording the class sessions in higher quality in the future, although I wouldn't want anything else to change about them.

Of course, this is the tip of the tip of the tip of the iceberg of what's out there, and Java's ubiquity means the list of materials (free and otherwise) will only grow.  Perhaps you have some favorites of your own; are there any Java learning resources you'd recommend to others?