Putting It All Together

By now, you have been introduced to the major concepts in Design Studio, you have taken a tour of the Design Studio user interface, you know the concepts of robots, types, step actions and data converters, and you have a feel of what patterns and expressions are.

Let us now put all this Design Studio-knowledge into use and make some robots. However, first we need an overview of how robots are normally built — that is, their structure.

Robots mimic human behavior — that is, they do (more or less) what you do when you are looking for content on the Internet using a browser: You start by searching for the content. Once found, you read and process it. Similarly, most robots can be roughly divided into two parts: A navigation part and an extraction part.

Navigation is concerned with "getting to where the content is." Navigation mainly includes loading pages and submitting forms. When navigating in Design Studio, you typically use the Click action to navigate to and between web pages.

Extraction is concerned with "getting the right content." Extraction mainly includes selecting, copying, and normalizing content from a web page that you have navigated to. When extracting in Design Studio, you typically use the Test Tag action to skip uninteresting ("noisy") content, the Extract action to copy content into variables, and the data converters for normalizing the content so that it gets the format you want, e.g. the right date and number format. Once extracted, you output the value with the "Store in Database" or "Return Value" action.

So, the typical robot starts with one or more steps, each containing a Load Page or Click action, in order navigate to the interesting content on some web site. It proceeds with one or more steps, each containing an Extract action, and ends with a step storing or returning the extracted value.

Note that in many robots the navigation and extraction parts overlap because the content to extract is located on several pages. Again, this is similar to when you look for content yourself; often, you have to visit several pages to get the content you want.

Most robots include other actions than the ones mentioned above, e.g. a For Each Tag action for loading several similar looking pages or extracting values from several similar looking table rows. Because robots have different tasks, they have different needs. For this reason, we have included a considerable number of step actions and data converters in Design Studio. Start with familiarizing yourself with the basic and most commonly used step actions and data converters, and then begin to explore. Experience shows that one can create most robots using only a handful of step actions and data converters. So, find your own favorite step actions and data converters and stick to them until you feel a need to explore others.