Learning Center Tutorials How to automate web form or API using Bash scripts
How to automate web form or API using Bash scripts

How to automate web form or API using Bash scripts

March 01 2023

Share

In this tutorial, I hope to explain where to start when automating web forms or HTTP-based REST APIs using Bash or other UNIX command-line scripting languages. This tutorial should work for bash, the default on most GNU / Linux systems, and zsh, the default for macOS systems.

Step 1: Gathering information about the web form or API

First, we need to know how to use the API. For many APIs, this is a URL, starting with HTTP, like https://api.github.com/repos/modulojs/modulo/

Questions to ask yourself:

  • Is it an API? Do you know the API? Like, what sort of requests it expects? It typically expects something called a "request" (for our students: relevant lessons include Backend 1.1 and Backend 1.2). Requests can be generated from bash as well, using a Bash program such as curl or wget. We'll learn how to use curl in this tutorial.

  • Is it a web form?

  • If you are trying to automate a web form, when you hit "submit", behind the scenes your browser will send the thing called a "request" to the server. We'll need to simulate this same request using the bash script.

  • You can actually even "freeze" requests sent in the browser into a curl bash command with the following steps:

  1. Bringing up dev tools (Ctrl+J or Command+J)
  2. Click on the network tab
  3. Identify the request that sends the info
  4. Select "Copy as Curl" to copy the info of the request as a curl command

Once you've located the URL, ensure it works by pasting it in your web browser. Do you get JSON or some other data format visible? If so, you probably found the correct API for the resource you want. For example, try clicking on our example API below:

Step 2: Start a new Bash script

  1. Create a new Bash script by opening an empty file in your preferred text editor, such as the free open VS Code by Microsoft, that should work on all platforms (Windows, Mac and Linux). For now, write echo Hello API world into it
  2. Save the Bash script using a filename, such as api_script.sh (avoid using spaces in filenames, consider all-lowercase)
  3. Navigate to where you saved the Bash script in a Terminal. You know you are in the right place when you see the filenmae (e.g. api_script.sh) when you run ls (i.e., this means you are in the same directory as the thing). (In Linux, you can right-click on a window and select "Open in Terminal". In other OS's, open your terminal, type cd followed by space, then drag over the directory)
  4. Run the bash script with the command bash api_script.sh

All together, it's like this:

$ cd /home/michaelb/projects/myapiproj/
$ ls
api_script.sh
$ bash api_script.sh
Hello API World

Step 3: Add in the cURL commands to get data from the API

The cURL command copied from the browser following the steps above, and then can be pasted into a bash script to repeat that exact request. For example:

echo Modulo info
curl https://api.github.com/repos/modulojs/modulo

echo Cheatsheet info
curl https://api.github.com/repos/kickstartcoding/cheatsheets

Step 4: Possibly add in form data as POST data or JSON APIs

The cURL command can do a lot, such as using POST and also including "post data", which would be the form data itself. If you use the "Copy as cURL" command then you get this automatically. Otherwise, you'll need to use form encoding, as such:

curl -d "param1=value1&param2=value2" -X POST https://api.some.api.io/some/path/etc

curl -d '{"param1":"value1", "param2":"value2"}' -H "Content-Type: application/json" -X POST https://api.some.api.io/some/path/etc