Expect command and how to automate shell scripts like magic
Today we are going to talk about a tool that does magic to our shell scripts; that tool is the Expect command or Expect scripting language.
Expect command or expect scripting language is a language that talks with your interactive programs or scripts that require user interaction.
Expect scripting language works by expecting input, then the Expect script will send the response without any user interaction.
You can say that this tool is your robot, which will automate your scripts.
If the expect command if not installed on your system, you can install it using the following command:
$ apt-get install expect
Or on Red Hat based systems like CentOS:
$ yum install expect
Expect Command
Before we talk about expect command, Let’s see some of the Expect command which used for interaction:
| spawn | Starts a script or a program. |
| expect | Waits for program output. |
| send | Sends a reply to your program. |
| interact | Allows you to interact with your program. |
- The spawn command starts a script or a program like the shell, FTP, Telnet, SSH, SCP, and so on.
- The send command sends a reply to a script or a program.
- The Expect command waits for input.
- The interact command allows you to define a predefined user interaction.
We are going to type a shell script that asks some questions, and we will make an Expect script that will answer those questions.
First, the shell script will look like this:
#!/bin/bash echo "Hello, who are you?" read $REPLY echo "Can I ask you some questions?" read $REPLY echo "What is your favorite topic?" read $REPLY
Now we will write the Expect scripts that will answer this automatically:
#!/usr/bin/expect -f set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Im Adam\r" expect "Can I ask you some questions?\r" send -- "Sure\r" expect "What is your favorite topic?\r" send -- "Technology\r" expect eof
The first line defines the expect command path which is
#!/usr/bin/expect
On the second line of code, we disable the timeout. Then start our script using spawn command.
We can use spawn to run any program we want or any other interactive script.
The remaining lines are the Expect script that interacts with our shell script.
The last line means the end of the interaction.
Now Showtime, let’s run our answer bot and make sure you make it executable.
$ chmod +x ./answerbot
$./answerbot
Cool!! The send command answered all questions.
If you get errors about the location of Expect command you can get the location using the which command:
$ which expect
We did not interact with our script at all; the Expect program does the job for us.
You can apply the above method to any interactive script or program. Although the above Expect script is very easy to write, maybe the Expect script little tricky for some people, well you have it.
Automate interactive commands
The -c option for expect allows you to provide expect commands directly from the command line rather than from a file.
Let’s say you want to spawn a shell, wait for the shell prompt, and then send the command whoami:
expect -c 'spawn /bin/bash; expect "$ "; send "whoami\n"'
First, spawn /bin/bash initiates a new shell instance.
Then, theexpect "$ " waits for the shell to present its typical prompt, which is often represented as $ for a regular user.
Once the script identifies this prompt, send "whoami\n" instructs the script to input the whoami command into the shell, followed by a newline (represented by \n), which is equivalent to pressing the “Enter” key.
The result is the same as if you manually typed the whoami command into a shell to find out the name of the currently logged-in user.
Using autoexpect
To build an expect script automatically, you can use the autoexpect command.
autoexpect works like expect command, but it builds the automation script for you. You pass the script you want to automate to autoexpect as a parameter, and you answer the questions, and autoexpect saves your answers in a file.
$ autoexpect ./questions
The autoexpect generates a file called script.exp contains the same code as we did above with some additions that we will leave it for now.
If you run the auto-generated file script.exp, you will see the same answers as expected:
Awesome!! That super easy.
Many commands produce changeable output, like the case of FTP programs, the expect script may fail or stuck. To solve this problem, you can use wildcards for the changeable data to make your script more flexible.
Working with Variables
You can use the set command to define variables in expect scripts like this:
set MYVAR 5
To access the variable, precede it with $ like this $VAR1
To define command line arguments in expect scripts, we use the following syntax:
set MYVAR [lindex $argv 0]
Here we define a variable MYVAR, which equals the first passed argument.
You can get the first and the second arguments and store them in variables like this:
set my_name [lindex $argv 0] set my_favorite [lindex $argv 1]
Let’s add variables to our script:
#!/usr/bin/expect -f set my_name [lindex $argv 0] set my_favorite [lindex $argv 1] set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Im $my_name\r" expect "Can I ask you some questions?\r" send -- "Sure\r" expect "What is your favorite topic?\r" send -- "$my_favorite\r" expect eof
Now try to run the Expect script with some parameters to see the output:
$ ./answerbot SomeName Programming
Awesome!! Now our automated Expect script is more dynamic.
Conditional Tests
You can write conditional tests using braces like this:
expect {
"something" { send -- "send this\r" }
"*another" { send -- "send another\r" }
}
We are going to change our script to return different conditions, and we will change our Expect script to handle those conditions.
We are going to emulate different expects with the following script:
#!/bin/bash let number=$RANDOM if [ $number -gt 25000 ]; then echo "What is your favorite topic?" else echo "What is your favorite movie?" fi read $REPLY
When you run the script, it generates a random number every time, and based on that number; we put a condition to return different expects.
Let’s make out Expect script that will deal with that.
#!/usr/bin/expect -f
set timeout -1
spawn ./questions
expect {
"*topic?" { send -- "Programming\r" }
"*movie?" { send -- "Star wars\r" }
}
expect eof
Very clear. If the script hits the topic output, the expect script will send programming, and if the script hits movie output, the expect script will send star wars. Isn’t cool?
If else Conditions
You can use if/else clauses in expect scripts like this:
#!/usr/bin/expect -f
set NUM 1
if { $NUM < 5 } {
puts "\Smaller than 5\n"
} elseif { $NUM > 5 } {
puts "\Bigger than 5\n"
} else {
puts "\Equals 5\n"
}
Note: The opening brace must be on the same line.
While Loops
While loops in expect language must use braces to contain the expression like this:
#!/usr/bin/expect -f
set NUM 0
while { $NUM <= 5 } {
puts "\nNumber is $NUM"
set NUM [ expr $NUM + 1 ]
}
puts ""
For Loops
Like any scripting or programming language, you can use for loops for repetitive tasks, three fields must be specified, like the following format:
#!/usr/bin/expect -f
for {set NUM 0} {$NUM <= 5} {incr NUM} {
puts "\nNUM = $NUM"
}
puts ""
User-defined Functions
You can define a function using proc like this:
proc myfunc { TOTAL } {
set TOTAL [expr $TOTAL + 1]
return "$TOTAL"
}
And you can use them after that.
#!/usr/bin/expect -f
proc myfunc { TOTAL } {
set TOTAL [expr $TOTAL + 1]
return "$TOTAL"
}
set NUM 0
while {$NUM <= 5} {
puts "\nNumber $NUM"
set NUM [myfunc $NUM]
}
puts ""
Interact Command
Sometimes your expect script contains some sensitive information that you don’t want to share with other users who use your expect scripts, like passwords or any other data. Hence, you want your script to take this password from you and continuing automation normally.
The interact command reverts the control to the keyboard.
When you use the interact command, expect will start reading from the keyboard.
This shell script will ask about the password as shown:
#!/bin/bash echo "Hello, who are you?" read $REPLY echo "What is you password?" read $REPLY echo "What is your favorite topic?" read $REPLY
Now we will write the Expect script that will prompt for the password:
#!/usr/bin/expect -f set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Hi Im Adam\r" expect "*password?\r" interact ++ return send "\r" expect "*topic?\r" send -- "Technology\r" expect eof
After you type your password type ++ and the control will return from the keyboard to the script.
Expect language is ported to many languages like C#, Java, Perl, Python, Ruby, and Shell with almost the same concepts and syntax due to its simplicity and importance.
You can use the expect scripting language in quality assurance, network measurements such as echo response time, automate file transfers, updates, and many other uses.
I hope you now supercharged with some of the most important aspects of expect command, autoexpect command, and how to use it to automate your tasks in a smarter way.
Thank you.
Mokhtar is the founder of LikeGeeks.com. He is a seasoned technologist and accomplished author, with expertise in Linux system administration and Python development. Since 2010, Mokhtar has built an impressive career, transitioning from system administration to Python development in 2015. His work spans large corporations to freelance clients around the globe. Alongside his technical work, Mokhtar has authored some insightful books in his field. Known for his innovative solutions, meticulous attention to detail, and high-quality work, Mokhtar continually seeks new challenges within the dynamic field of technology.
That a very well written article 🙂
Thank you very much!
I need some suggestion
How can i automate a password entry for a command and that command is not to login to another server its like a permission within the server.
I need to copy a file from specific path but when i try to copy it will ask me to enter password for security reason , whether can i automate this one..
Here’s the script i have used
#!/bin/ksh
copy_command()
{
cp /xxx/xxxxx/logs/SAMPLE.2016-11-29 /xxx/xxxxx/sample/folder2/ > /xxx/xxxxx/sample/log.txt 2>&1
}
error_check()
{
if [ -f /xxx/xxxxx/sample/log.txt ]
then
if [ -s /xxx/xxxxx/sample/log.txt ]
then
echo “File exists and not empty”
if grep -q ‘Cannot find the requested security attribute’ /xxx/xxxxx/sample/log.txt; then
echo “found”
efskeymgr -o ksh
expect “*EFS password*\r”
send — “abcd@123\r”
fi
fi
fi
}
main()
{
copy_command
retval=$?
error_check
fi
}
#Calling Main Function
main
Did you test this code?
If so, what is the error?
Thanks for the article; Would i be able to spawn a function within the same script?
If so could you give me an example.
Spawning a script abcd.ksh within an expect script (xyz.ksh) is working but i would like to make abcd.ksh as a function inside xyz.ksh and call the same.
Could you give me an example please
Yes, you can.
To spawn a function from inside the same file, you can use the -c of the bash like this:
spawn bash -c "myfunc"Is it possible to use a variable stored by bash? Or is there a way to get input without showing the input like displaying a password to the screen? I know you can do it in bash with read -s I am trying to learn expect because I would like to remotely change the passwords on windows 10, linux, and a couple different firewalls every 60 days. So having something like this would be beneficial to get a password one time and then send it to change the password on each OS.
If your calling process exported an environmental variable, you can use it like this:
$::env(myvar)But since you are using expect, you don’t need to use Bash, you can access variables like this:
set myvar [lindex $argv 0]if {$myvar == "myval"}
#Your code goes here
Hope that helps.
Your articles are very well written and a great help! Thanks for all your hard work and thank you for sharing!
Thank you very much for the kind words!
That drives me to do my best.
Below is the code I used but didn’t get output with autoexpect command..it is struck after asking first question..
#!/usr/bin/expect -f
set timeout -1
spawn ./questions
expect “Hello, How are you?\r”
send — ” Hi Im Adam\r”
expect “*password?\r”
interact ++ return
send “\r”
expect “*topic?\r”
send — “Technology\r”
expect eof
It depends on what is in your questions file.
awesome!
Thanks!
Here I’m trying to execute a simple command, I’m not getting any error however, its coming out without issuing the last command (df -gt)
kindly let me know what is the error or mistake?
#!/usr/bin/expect
set username [lindex $argv 0]
set hostname [lindex $argv 1]
set password [lindex $argv 2]
set username “inxxxxxxx”
set hostname “xxxxxx111”
set password “xxxxxxxxxx2299”
spawn ssh $hostname
expect “$username@$hostname\’s password: ” {send “$password\r”}
#expect “password: ” {send “$password\r”}
expect “\] ” {send “sudo su -\r”}
expect “root-\] ” {send “su – db2v105\r”}
expect “db2v105\> ” {send “df -gt\r”}
Since there is no error as you said, then you should debug your expect script or run it step by step to check the output of every line.
To debug your script, you can use -D option like this:
$ expect -D 1 yourscript.fileAlso, you can run a line each time using -b option like this:
$ expect -bHope that helps!
Ok, that didn’t work, however can you give simple program or explain program where we can issue command, I may learn from there.
Thank you!
Mokhtar
i want to run some test programs one after another.
by using while loop i ran it.
first test program got failed and stuck i mean i am not getting response from CLI to run second test program.
shell script to run Test programs
n=1
while (($n <= 500))
do
./DOWNLOAD0350 -debug
sleep 5
./DOWNLOAD0340 -debug
sleep 5
n=$(( n+1 ))
done
CLI Output:
-* Invalid compile option : customize_code=74
fail : <> (line#108) [FW Rev:40F82840]
That’s a wired message.
It’s a script, not a code to compile. How do you run your script?
I think this is one of the best blogs for me because this is really helpful for me. Thanks for sharing this valuable information for free
You’re welcome! Thank you very much!
I’ll do my best always.
Hi all
Do you know why I cant run “Expected” when I use a cron task ?
the script works when I run it manually but does not works using a cron task
#!/usr/bin/expect -f
set client client
spawn scp [lindex $argv 0] plclient@x.x.x.x:/dfcxact/workarea/Global
send “\r”
expect “password:”
send “$client”
send “\r”
interact
Thanks
Hi all
I got the fix…
I just set
“expect eof” istead of “interact”
Regards
hello,
how to give control to keyboard entirely.
Only after pressing lets say cntrl+c or any key control should come back to expect shell script.
Thanks in advance!
As mentioned in the tutorial, the interact command gives control to the keyword.
There is an example explains that.
Regards,
Hi, Mokhtar.
I found your post about the ‘expect’ command very interesting and useful.
At the end of this post, you pinpointed a few possible application areas, like network measurements and file transfer automation.
Can you give us a practical example for each of those?
I guess you have already done things of this sort plenty of times, so maybe all you need is just take some of your already made maintenance scripts and adapt or simplify them, leaving out all blows and whistles or any unessential elements, which might confuse beginners.
Thank you so much in advance.
Hi Valerio,
The remains the same. All you know about shell scripting can be automated with the same tools used here in this tutorial.
I’ll plan to make new tutorials about expect and autoexpect in the near future.
Regards,
How to keep sending an answer (say yes) until the question is prompted?
Why would you need to send an answer for nothing prompted?
Sending something for an expected question or prompt.
Hi mokthar,
I’m trying to automate a shell script.wherein the question prompted is “do you want to continue?”. I want to send “yes” until the question is prompted.
Hi Jay,
You can send yes in a loop and inside the loop, you can expect the question and if it matches your question, you can do the rest.
Hi Mokhtar,
This is the code:
expect {
“Do you want to continue? (yes / no) * :”
{
send “yes \r”
exp_continue
}
“Enter choice * :”
{
send “2 \r”
}
}
But the loop is not happening.
Hi Jay,
You need to wrap your code inside a while loop
Hi Mokhtar,
Is the code correct ?
set Ques “Do you want to continue? (yes / no) * :”
set Temp “Enter choice * :”
while { $Ques == “Do you want to continue? (yes / no) * :” | $Temp == “Enter choice * :” }
{
expect {
“Do you want to continue? (yes / no) * :”
{
send “yes \r”
exp_continue
}
“Enter choice * :”
{
send “2 \r”
}
}
}
Looks fine.
Test it with the script you want to automate.
Hi Mokhtar,
Receiving the following error.
wrong # args: should be “while test command”
while executing
“while { $Ques == “Do you want to continue? (yes / no) * :” | $Temp == “Enter choice * :” }”
(file “./dsr.exp” line 23)
You need to debug your code and make sure maybe you miss a quotation or so.
However, you can debug your code using -v and -x like this:
$ bash -v myscript.sh
Also, you can visually debug your code and put breakpoints on your code using the bash debug plugin which is a VSCode plugin.
Regards,
Hi
#!/usr/bin/expect -f
spawn ssh user@ipaddress
expect “Password:*”
send “passwordhere\r”
expect “$ ”
interact
this works fine to ssh into remote vm
after ssh, i want to execute powershell script with2 arguments ..
so how can i achieve with expect?
Hi,
You can execute the commands you want using the spawn command as mentioned in the tutorial.
Excellent article. Thank you so much!
One question:
If I send a message like this:
expect “*something*”
send “message”
How do I hide what is sent (in this case “message”)? I mean, I do not want prompt to print out “message”
Hi,
expect by itself shows output on the screen, but as I understand from your question, you need to hide output from showing on the screen.
However, you can use stty command to disable the echoing of input characters:
Here, the input won’t be displayed on the terminal.
Hi Mokhtar Ebrahim,
Thanks for the detailed information.
My requirement is to read from a file, make a list and iterate through the list. So, the parameter value should be taken from the iteration instead of command line argument.
If I go with bash, I’m able to do it. But I cannot use both bash & expect in the same script right. Can you please help me do this completely in expect itself?
Thanks!
Hi,
If I got your point right, you want to read from a file, make a list, and then iterate through that list using expect.
#!/usr/bin/expect -f # Read lines from a file into a list set fileId [open "yourfile.txt" r] set data [read $fileId] close $fileId # Convert data into a list (split on newlines) set list [split $data "\n"] # Iterate over the list foreach item $list { # Replace this with your logic. puts "Processing: $item" }Hope that helps!
Thanks for your help Mokhtar!
You’re welcome!
Hi Mokhtar!
In the below code, I am fetching list of items from a file and iterating using foreach loop. So, this is in sequential manner. I want to iterate each item in a parallel manner, so tried with ‘&’ to run each item in the background, but it is not supported in expect script.
#!/usr/bin/env expect
if { $argc < 1 } {
send_user "Syntax: bulk-clean \n”
exit
}
set filename [lindex $argv 0];
set list [exec cat $filename]
set user
set password
set prompt “# ”
foreach name $list {
set hostname $name
puts “\n”
puts $hostname
spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 22 $user@$hostname
expect “assword:”
send “$password\r”
expect $prompt
send
expect $prompt
send
expect $prompt
send “exit\r”
}
puts “\n”
Please help me with this.
Thank You!
Hi,
Expect language is designed for automating interactions with programs that require user input. It isn’t built to support parallel operations natively.
However, you can utilize xargs to accomplish parallel processing:
The -n 1 tells xargs to use one line of input for each command line, -P 10 runs up to 10 processes at a time, and % is the placeholder for each line of input.
The expect_script.exp will be like this:
#!/usr/bin/env expect if { $argc < 4 } { send_user "Syntax: expect_script.exp user password hostname prompt \n" exit } set user [lindex $argv 0] set password [lindex $argv 1] set hostname [lindex $argv 2] set prompt [lindex $argv 3] puts "\n" puts $hostname spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 22 $user@$hostname expect "assword:" send "$password\r" expect $prompt send "exit\r" puts "\n"Thanks for your help here Mokhtar!
You’re welcome!
Only reason why am i asking about -c flag is because i wanna learn to use it so before spawn just kicks in i can define which command to use before any other in shell.
Hi,
The -c flag explanation is added to the article.
Best Wishes!
Hi Mokhtar!
How will i run multiple commands in a single loop.
my requirement is First need to login host ip using spawn and then device ip and i want to execute all the commands in single loop on device.
Hi,
You can Use spawn to SSH into the host, then after successful login to the device, loop through a list of commands and execute them one-by-one.
Here an example expect script:
#!/usr/bin/expect -f set timeout 20 set host_ip "YOUR_HOST_IP" set host_user "YOUR_HOST_USERNAME" set host_pass "YOUR_HOST_PASSWORD" set device_ip "YOUR_DEVICE_IP" set device_user "YOUR_DEVICE_USERNAME" set device_pass "YOUR_DEVICE_PASSWORD" # Commands to be executed on the device set commands { "command1" "command2" "command3" } spawn ssh $host_user@$host_ip expect { timeout { send_user "Connection timed out\n"; exit } "yes/no" { send "yes\r"; exp_continue } "password:" { send "$host_pass\r" } } expect { "$ " { send "ssh $device_user@$device_ip\r" } } expect { timeout { send_user "Connection to device timed out\n"; exit } "yes/no" { send "yes\r"; exp_continue } "password:" { send "$device_pass\r" } } # Execute each command in a loop foreach cmd $commands { expect { "$ " { send "$cmd\r" } } }Hope that helps!
Hi Mokhtar,
I am using expect to scp files from Linux to Windows and Windows to Linux Vice Versa.
When I try to move bunch of files with Wild card at the end of the file name is not working when scp from Linux to Windows , but its working Windows to Linux.
scp command on the terminal is working but in with SPWAN command .
Example :
Working :
scp /Linux_path/file_name*.csv /Windows_path/
Not Working
spwan scp /Linux_path/file_name*.csv /Windows_path/
Error :
/Linux_path/file_name*.csv : No such file or directory
Hi,
As Expect runs commands in a non-interactive shell, wildcard expansion (globbing) might not work, because this is a feature of interactive shells.
You could try to use glob in tcl (Expect scripting), or use a workaround like ls and for loop to parse this files one by one.
Example with glob:
Example with for loop:
foreach file [glob -nocomplain /Linux_path/file_name*.csv] { spawn scp $file /Windows_path/ expect "password:" send "YourPassword\r" expect eof }Hope that helps!