Expect command and how to automate shell scripts like magic

Today we are going to talk about a tool that does magic to our shell scripts; that tool is the Expect command or Expect scripting language.

Expect command or expect scripting language is a language that talks with your interactive programs or scripts that require user interaction.

Expect scripting language works by expecting input, then the Expect script will send the response without any user interaction.

You can say that this tool is your robot, which will automate your scripts.

If the expect command if not installed on your system, you can install it using the following command:

$ apt-get install expect

Or on Red Hat based systems like CentOS:

$ yum install expect

 

 

Expect Command

Before we talk about expect command, Let’s see some of the Expect command which used for interaction:

spawn Starts a script or a program.
expect Waits for program output.
send Sends a reply to your program.
interact Allows you to interact with your program.
  • The spawn command starts a script or a program like the shell, FTP, Telnet, SSH, SCP, and so on.
  • The send command sends a reply to a script or a program.
  • The Expect command waits for input.
  • The interact command allows you to define a predefined user interaction.

We are going to type a shell script that asks some questions, and we will make an Expect script that will answer those questions.

First, the shell script will look like this:

#!/bin/bash

echo "Hello, who are you?"

read $REPLY

echo "Can I ask you some questions?"

read $REPLY

echo "What is your favorite topic?"

read $REPLY

Now we will write the Expect scripts that will answer this automatically:

#!/usr/bin/expect -f

set timeout -1

spawn ./questions

expect "Hello, who are you?\r"

send -- "Im Adam\r"

expect "Can I ask you some questions?\r"

send -- "Sure\r"

expect "What is your favorite topic?\r"

send -- "Technology\r"

expect eof

The first line defines the expect command path which is

#!/usr/bin/expect

On the second line of code, we disable the timeout. Then start our script using spawn command.

We can use spawn to run any program we want or any other interactive script.

The remaining lines are the Expect script that interacts with our shell script.

The last line means the end of the interaction.

Now Showtime, let’s run our answer bot and make sure you make it executable.

$ chmod +x ./answerbot
$./answerbot

expect command

Cool!! The send command answered all questions.

If you get errors about the location of Expect command you can get the location using the which command:

$ which expect

We did not interact with our script at all; the Expect program does the job for us.

You can apply the above method to any interactive script or program. Although the above Expect script is very easy to write, maybe the Expect script little tricky for some people, well you have it.

 

Automate interactive commands

The -c option for expect allows you to provide expect commands directly from the command line rather than from a file.

Let’s say you want to spawn a shell, wait for the shell prompt, and then send the command whoami:

expect -c 'spawn /bin/bash; expect "$ "; send "whoami\n"'

First, spawn /bin/bash initiates a new shell instance.

Then, theexpect "$ " waits for the shell to present its typical prompt, which is often represented as $ for a regular user.

Once the script identifies this prompt, send "whoami\n" instructs the script to input the whoami command into the shell, followed by a newline (represented by \n), which is equivalent to pressing the “Enter” key.

The result is the same as if you manually typed the whoami command into a shell to find out the name of the currently logged-in user.

 

Using autoexpect

To build an expect script automatically, you can use the autoexpect command.

autoexpect works like expect command, but it builds the automation script for you. You pass the script you want to automate to autoexpect as a parameter, and you answer the questions, and autoexpect saves your answers in a file.

$ autoexpect ./questions

autoexpect command

The autoexpect generates a file called script.exp contains the same code as we did above with some additions that we will leave it for now.

autoexpect script

If you run the auto-generated file script.exp, you will see the same answers as expected:

autoexpect script execution

Awesome!! That super easy.

Many commands produce changeable output, like the case of FTP programs, the expect script may fail or stuck. To solve this problem, you can use wildcards for the changeable data to make your script more flexible.

 

Working with Variables

You can use the set command to define variables in expect scripts like this:

set MYVAR 5

To access the variable, precede it with $ like this $VAR1

To define command line arguments in expect scripts, we use the following syntax:

set MYVAR [lindex $argv 0]

Here we define a variable MYVAR, which equals the first passed argument.

You can get the first and the second arguments and store them in variables like this:

set my_name [lindex $argv 0]

set my_favorite [lindex $argv 1]

Let’s add variables to our script:

#!/usr/bin/expect -f

set my_name [lindex $argv 0]

set my_favorite [lindex $argv 1]

set timeout -1

spawn ./questions

expect "Hello, who are you?\r"

send -- "Im $my_name\r"

expect "Can I ask you some questions?\r"

send -- "Sure\r"

expect "What is your favorite topic?\r"

send -- "$my_favorite\r"

expect eof

Now try to run the Expect script with some parameters to see the output:

$ ./answerbot SomeName Programming

expect command variables

Awesome!! Now our automated Expect script is more dynamic.

 

Conditional Tests

You can write conditional tests using braces like this:

expect {

    "something" { send -- "send this\r" }

    "*another" { send -- "send another\r" }

}

We are going to change our script to return different conditions, and we will change our Expect script to handle those conditions.

We are going to emulate different expects with the following script:

#!/bin/bash

let number=$RANDOM

if [ $number -gt 25000 ]; then

	echo "What is your favorite topic?"

else

	echo "What is your favorite movie?"

fi

read $REPLY

When you run the script, it generates a random number every time, and based on that number; we put a condition to return different expects.

Let’s make out Expect script that will deal with that.

#!/usr/bin/expect -f

set timeout -1

spawn ./questions

expect {

    "*topic?" { send -- "Programming\r" }

    "*movie?" { send -- "Star wars\r" }

}

expect eof

expect command conditions

Very clear. If the script hits the topic output, the expect script will send programming, and if the script hits movie output, the expect script will send star wars. Isn’t cool?

 

If else Conditions

You can use if/else clauses in expect scripts like this:

#!/usr/bin/expect -f

set NUM 1

if { $NUM < 5 } {

    puts "\Smaller than 5\n"

} elseif { $NUM > 5 } {

    puts "\Bigger than 5\n"

} else {

    puts "\Equals 5\n"

}

if command

Note: The opening brace must be on the same line.

 

While Loops

While loops in expect language must use braces to contain the expression like this:

#!/usr/bin/expect -f

set NUM 0

while { $NUM <= 5 } {

    puts "\nNumber is $NUM"

    set NUM [ expr $NUM + 1 ]

}

puts ""

while loop

 

For Loops

Like any scripting or programming language, you can use for loops for repetitive tasks, three fields must be specified, like the following format:

#!/usr/bin/expect -f

for {set NUM 0} {$NUM <= 5} {incr NUM} {

    puts "\nNUM = $NUM"

}

puts ""

for loop

 

User-defined Functions

You can define a function using proc like this:

proc myfunc { TOTAL } {

    set TOTAL [expr $TOTAL + 1]

    return "$TOTAL"

}

And you can use them after that.

#!/usr/bin/expect -f

proc myfunc { TOTAL } {

    set TOTAL [expr $TOTAL + 1]

    return "$TOTAL"

}

set NUM 0

while {$NUM <= 5} {

    puts "\nNumber $NUM"

    set NUM [myfunc $NUM]

}

puts ""

user-defined functions

 

Interact Command

Sometimes your expect script contains some sensitive information that you don’t want to share with other users who use your expect scripts, like passwords or any other data. Hence, you want your script to take this password from you and continuing automation normally.

The interact command reverts the control to the keyboard.

When you use the interact command, expect will start reading from the keyboard.

This shell script will ask about the password as shown:

#!/bin/bash

echo "Hello, who are you?"

read $REPLY

echo "What is you password?"

read $REPLY

echo "What is your favorite topic?"

read $REPLY

Now we will write the Expect script that will prompt for the password:

#!/usr/bin/expect -f

set timeout -1

spawn ./questions

expect "Hello, who are you?\r"

send -- "Hi Im Adam\r"

expect "*password?\r"

interact ++ return

send "\r"

expect "*topic?\r"

send -- "Technology\r"

expect eof

interact command

After you type your password type ++ and the control will return from the keyboard to the script.

Expect language is ported to many languages like C#, Java, Perl, Python, Ruby, and Shell with almost the same concepts and syntax due to its simplicity and importance.

You can use the expect scripting language in quality assurance, network measurements such as echo response time, automate file transfers, updates, and many other uses.

I hope you now supercharged with some of the most important aspects of expect command, autoexpect command, and how to use it to automate your tasks in a smarter way.

Thank you.

55 thoughts on “Expect command and how to automate shell scripts like magic
  1. I need some suggestion
    How can i automate a password entry for a command and that command is not to login to another server its like a permission within the server.

    I need to copy a file from specific path but when i try to copy it will ask me to enter password for security reason , whether can i automate this one..
    Here’s the script i have used

    #!/bin/ksh
    copy_command()
    {
    cp /xxx/xxxxx/logs/SAMPLE.2016-11-29 /xxx/xxxxx/sample/folder2/ > /xxx/xxxxx/sample/log.txt 2>&1
    }
    error_check()
    {
    if [ -f /xxx/xxxxx/sample/log.txt ]
    then
    if [ -s /xxx/xxxxx/sample/log.txt ]
    then
    echo “File exists and not empty”
    if grep -q ‘Cannot find the requested security attribute’ /xxx/xxxxx/sample/log.txt; then
    echo “found”
    efskeymgr -o ksh
    expect “*EFS password*\r”
    send — “abcd@123\r”
    fi
    fi
    fi
    }
    main()
    {
    copy_command
    retval=$?
    error_check
    fi
    }
    #Calling Main Function
    main

  2. Thanks for the article; Would i be able to spawn a function within the same script?
    If so could you give me an example.
    Spawning a script abcd.ksh within an expect script (xyz.ksh) is working but i would like to make abcd.ksh as a function inside xyz.ksh and call the same.

    Could you give me an example please

    1. Yes, you can.
      To spawn a function from inside the same file, you can use the -c of the bash like this:
      spawn bash -c "myfunc"

  3. Is it possible to use a variable stored by bash? Or is there a way to get input without showing the input like displaying a password to the screen? I know you can do it in bash with read -s I am trying to learn expect because I would like to remotely change the passwords on windows 10, linux, and a couple different firewalls every 60 days. So having something like this would be beneficial to get a password one time and then send it to change the password on each OS.

    1. If your calling process exported an environmental variable, you can use it like this:
      $::env(myvar)
      But since you are using expect, you don’t need to use Bash, you can access variables like this:
      set myvar [lindex $argv 0]
      if {$myvar == "myval"}
      #Your code goes here

      Hope that helps.

  4. Your articles are very well written and a great help! Thanks for all your hard work and thank you for sharing!

  5. Below is the code I used but didn’t get output with autoexpect command..it is struck after asking first question..

    #!/usr/bin/expect -f

    set timeout -1
    spawn ./questions
    expect “Hello, How are you?\r”
    send — ” Hi Im Adam\r”
    expect “*password?\r”
    interact ++ return
    send “\r”
    expect “*topic?\r”
    send — “Technology\r”
    expect eof

  6. Here I’m trying to execute a simple command, I’m not getting any error however, its coming out without issuing the last command (df -gt)
    kindly let me know what is the error or mistake?

    #!/usr/bin/expect

    set username [lindex $argv 0]
    set hostname [lindex $argv 1]
    set password [lindex $argv 2]

    set username “inxxxxxxx”
    set hostname “xxxxxx111”
    set password “xxxxxxxxxx2299”

    spawn ssh $hostname
    expect “$username@$hostname\’s password: ” {send “$password\r”}

    #expect “password: ” {send “$password\r”}

    expect “\] ” {send “sudo su -\r”}

    expect “root-\] ” {send “su – db2v105\r”}

    expect “db2v105\> ” {send “df -gt\r”}

    1. Since there is no error as you said, then you should debug your expect script or run it step by step to check the output of every line.
      To debug your script, you can use -D option like this:
      $ expect -D 1 yourscript.file
      Also, you can run a line each time using -b option like this:
      $ expect -b
      Hope that helps!

  7. Ok, that didn’t work, however can you give simple program or explain program where we can issue command, I may learn from there.

    Thank you!
    Mokhtar

  8. i want to run some test programs one after another.
    by using while loop i ran it.
    first test program got failed and stuck i mean i am not getting response from CLI to run second test program.

    shell script to run Test programs
    n=1
    while (($n <= 500))
    do
    ./DOWNLOAD0350 -debug
    sleep 5
    ./DOWNLOAD0340 -debug
    sleep 5
    n=$(( n+1 ))
    done

    CLI Output:
    -* Invalid compile option : customize_code=74
    fail : <> (line#108) [FW Rev:40F82840]

    1. That’s a wired message.
      It’s a script, not a code to compile. How do you run your script?

  9. I think this is one of the best blogs for me because this is really helpful for me. Thanks for sharing this valuable information for free

  10. Hi all
    Do you know why I cant run “Expected” when I use a cron task ?

    the script works when I run it manually but does not works using a cron task
    #!/usr/bin/expect -f
    set client client

    spawn scp [lindex $argv 0] plclient@x.x.x.x:/dfcxact/workarea/Global
    send “\r”
    expect “password:”
    send “$client”
    send “\r”
    interact

    Thanks

  11. Hi all

    I got the fix…

    I just set

    “expect eof” istead of “interact”

    Regards

  12. hello,
    how to give control to keyboard entirely.
    Only after pressing lets say cntrl+c or any key control should come back to expect shell script.
    Thanks in advance!

    1. As mentioned in the tutorial, the interact command gives control to the keyword.
      There is an example explains that.

      Regards,

  13. Hi, Mokhtar.
    I found your post about the ‘expect’ command very interesting and useful.
    At the end of this post, you pinpointed a few possible application areas, like network measurements and file transfer automation.
    Can you give us a practical example for each of those?
    I guess you have already done things of this sort plenty of times, so maybe all you need is just take some of your already made maintenance scripts and adapt or simplify them, leaving out all blows and whistles or any unessential elements, which might confuse beginners.
    Thank you so much in advance.

    1. Hi Valerio,

      The remains the same. All you know about shell scripting can be automated with the same tools used here in this tutorial.
      I’ll plan to make new tutorials about expect and autoexpect in the near future.

      Regards,

    1. Why would you need to send an answer for nothing prompted?
      Sending something for an expected question or prompt.

  14. Hi mokthar,
    I’m trying to automate a shell script.wherein the question prompted is “do you want to continue?”. I want to send “yes” until the question is prompted.

    1. Hi Jay,

      You can send yes in a loop and inside the loop, you can expect the question and if it matches your question, you can do the rest.

  15. Hi Mokhtar,

    This is the code:
    expect {
    “Do you want to continue? (yes / no) * :”
    {
    send “yes \r”
    exp_continue
    }
    “Enter choice * :”
    {
    send “2 \r”
    }
    }

    But the loop is not happening.

  16. Hi Mokhtar,

    Is the code correct ?
    set Ques “Do you want to continue? (yes / no) * :”
    set Temp “Enter choice * :”
    while { $Ques == “Do you want to continue? (yes / no) * :” | $Temp == “Enter choice * :” }
    {
    expect {
    “Do you want to continue? (yes / no) * :”
    {
    send “yes \r”
    exp_continue
    }
    “Enter choice * :”
    {
    send “2 \r”
    }
    }
    }

      1. Hi Mokhtar,

        Receiving the following error.
        wrong # args: should be “while test command”
        while executing
        “while { $Ques == “Do you want to continue? (yes / no) * :” | $Temp == “Enter choice * :” }”
        (file “./dsr.exp” line 23)

        1. You need to debug your code and make sure maybe you miss a quotation or so.
          However, you can debug your code using -v and -x like this:
          $ bash -v myscript.sh
          Also, you can visually debug your code and put breakpoints on your code using the bash debug plugin which is a VSCode plugin.

          Regards,

  17. Hi
    #!/usr/bin/expect -f
    spawn ssh user@ipaddress
    expect “Password:*”
    send “passwordhere\r”
    expect “$ ”
    interact

    this works fine to ssh into remote vm
    after ssh, i want to execute powershell script with2 arguments ..
    so how can i achieve with expect?

    1. Hi,

      You can execute the commands you want using the spawn command as mentioned in the tutorial.

  18. Excellent article. Thank you so much!

    One question:

    If I send a message like this:
    expect “*something*”
    send “message”

    How do I hide what is sent (in this case “message”)? I mean, I do not want prompt to print out “message”

    1. Hi,
      expect by itself shows output on the screen, but as I understand from your question, you need to hide output from showing on the screen.
      However, you can use stty command to disable the echoing of input characters:

      echo -n "Enter Password: "
      stty -echo
      read password
      stty echo
      echo

      Here, the input won’t be displayed on the terminal.

  19. Hi Mokhtar Ebrahim,

    Thanks for the detailed information.
    My requirement is to read from a file, make a list and iterate through the list. So, the parameter value should be taken from the iteration instead of command line argument.

    If I go with bash, I’m able to do it. But I cannot use both bash & expect in the same script right. Can you please help me do this completely in expect itself?

    Thanks!

    1. Hi,

      If I got your point right, you want to read from a file, make a list, and then iterate through that list using expect.

      #!/usr/bin/expect -f
      
      # Read lines from a file into a list
      set fileId [open "yourfile.txt" r]
      set data [read $fileId]
      close $fileId
      
      # Convert data into a list (split on newlines)
      set list [split $data "\n"]
      
      # Iterate over the list
      foreach item $list {
          # Replace this with your logic.
          puts "Processing: $item"
      }
      

      Hope that helps!

  20. Hi Mokhtar!

    In the below code, I am fetching list of items from a file and iterating using foreach loop. So, this is in sequential manner. I want to iterate each item in a parallel manner, so tried with ‘&’ to run each item in the background, but it is not supported in expect script.

    #!/usr/bin/env expect

    if { $argc < 1 } {
    send_user "Syntax: bulk-clean \n”
    exit
    }

    set filename [lindex $argv 0];
    set list [exec cat $filename]
    set user
    set password
    set prompt “# ”

    foreach name $list {
    set hostname $name
    puts “\n”
    puts $hostname

    spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 22 $user@$hostname
    expect “assword:”
    send “$password\r”
    expect $prompt

    send
    expect $prompt

    send
    expect $prompt

    send “exit\r”
    }
    puts “\n”

    Please help me with this.

    Thank You!

    1. Hi,
      Expect language is designed for automating interactions with programs that require user input. It isn’t built to support parallel operations natively.
      However, you can utilize xargs to accomplish parallel processing:

      #!/bin/bash
      
      filename=$1
      user=
      password=
      prompt="# "
      
      cat $filename | xargs -n 1 -P 10 -I % ./expect_script.exp $user $password % $prompt

      The -n 1 tells xargs to use one line of input for each command line, -P 10 runs up to 10 processes at a time, and % is the placeholder for each line of input.
      The expect_script.exp will be like this:

      #!/usr/bin/env expect
      
      if { $argc < 4 } {
          send_user "Syntax: expect_script.exp user password hostname prompt \n"
          exit
      }
      
      set user [lindex $argv 0]
      set password [lindex $argv 1]
      set hostname [lindex $argv 2]
      set prompt [lindex $argv 3]
      
      puts "\n"
      puts $hostname
      
      spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 22 $user@$hostname
      expect "assword:"
      send "$password\r"
      expect $prompt
      
      send "exit\r"
      puts "\n"
  21. Only reason why am i asking about -c flag is because i wanna learn to use it so before spawn just kicks in i can define which command to use before any other in shell.

  22. Hi Mokhtar!
    How will i run multiple commands in a single loop.
    my requirement is First need to login host ip using spawn and then device ip and i want to execute all the commands in single loop on device.

    1. Hi,
      You can Use spawn to SSH into the host, then after successful login to the device, loop through a list of commands and execute them one-by-one.
      Here an example expect script:

      #!/usr/bin/expect -f
      set timeout 20
      set host_ip "YOUR_HOST_IP"
      set host_user "YOUR_HOST_USERNAME"
      set host_pass "YOUR_HOST_PASSWORD"
      set device_ip "YOUR_DEVICE_IP"
      set device_user "YOUR_DEVICE_USERNAME"
      set device_pass "YOUR_DEVICE_PASSWORD"
      
      # Commands to be executed on the device
      set commands { "command1" "command2" "command3" }
      
      spawn ssh $host_user@$host_ip
      
      expect {
          timeout { send_user "Connection timed out\n"; exit }
          "yes/no" { send "yes\r"; exp_continue }
          "password:" { send "$host_pass\r" }
      }
      
      expect {
          "$ " { send "ssh $device_user@$device_ip\r" }
      }
      
      expect {
          timeout { send_user "Connection to device timed out\n"; exit }
          "yes/no" { send "yes\r"; exp_continue }
          "password:" { send "$device_pass\r" }
      }
      
      # Execute each command in a loop
      foreach cmd $commands {
          expect {
              "$ " { send "$cmd\r" }
          }
      }
      

      Hope that helps!

  23. Hi Mokhtar,
    I am using expect to scp files from Linux to Windows and Windows to Linux Vice Versa.
    When I try to move bunch of files with Wild card at the end of the file name is not working when scp from Linux to Windows , but its working Windows to Linux.
    scp command on the terminal is working but in with SPWAN command .

    Example :
    Working :
    scp /Linux_path/file_name*.csv /Windows_path/
    Not Working
    spwan scp /Linux_path/file_name*.csv /Windows_path/

    Error :
    /Linux_path/file_name*.csv : No such file or directory

    1. Hi,

      As Expect runs commands in a non-interactive shell, wildcard expansion (globbing) might not work, because this is a feature of interactive shells.

      You could try to use glob in tcl (Expect scripting), or use a workaround like ls and for loop to parse this files one by one.

      Example with glob:

      set files [glob -nocomplain /Linux_path/file_name*.csv]
      spawn scp $files /Windows_path/

      Example with for loop:

      foreach file [glob -nocomplain /Linux_path/file_name*.csv] {
          spawn scp $file /Windows_path/
          expect "password:"
          send "YourPassword\r"
          expect eof
      }

      Hope that helps!

Leave a Reply

Your email address will not be published. Required fields are marked *