Screen

Overview

Screen is essentially `nohup` on steroids. Screen is a userland application that allows one to start interactive login sessions, detach them, and later reattach them. Screen is extremely helpful in the following three manners:

1. Safety. It allows for sensitive interactive work through sshd on unreliable network connections. Imagine if you were running a dump or restore of a filesystem, or running a newfs on a filesystem, or some other sensitive action that, if interrupted, would not only halt the action itself, but could jeopordize the targets of these actions by leaving them in a half-done state that cannot be resumed - all you could do is log back on, kill the process, and try to salvage what was left and start over. However, if you were running that process in a screen window, if your connection died you could simply reattach to that screen window - as if you had never lost connection at all.
2. Location Portability. If you are logged into a system from point A, and later you go to point B, you cannot regain the exact same session in point B - any new login to the system would be just that - a new login. However, if your session is "screened", you can detach it, and re-attach at the new location, and present yourself with the exact same login session as you started with.
3. Organization. Because screen has multiple windows, and those windows can be nested, you can arrange a large number (no more than 100, practically speaking) of login sessions all in one ssh session. So you need only keep one single terminal window (or putty or secureCRT in Windows) open at any given time.

You start screen simply by logging into a system and running:

screen

with no arguments. After you bypass a short informational message, you will simply be at a new login prompt - just like the one you left behind when you ran screen. The difference is, you are in a screened login session that can be detached (dropping you back to your original login prompt, unscreened) and reattached from within a new login session.

Screen is manipulated with a "command sequence", which, by default, is

ctrl-a

You can create a new window by typing:

ctrl-a, then c

You can navigate between windows by typing:

ctrl-a, then X

(where X is a number between 0 and 9)

You can detach the screened session by typing:

ctrl-a, then d

After detaching screen, you can login at a later date and reattach by running:

screen -r

That is the essence of screen - starting it, creating new windows, and detaching it. There are, however, some important caveats. First, it is difficult to use more than 10 windows. You can type:

ctrl-a, then c

all day, and have 50 or more windows, but there is no way to address a window other than by the numbers between 0 and 9. Therefore, if you create an 11th window, you have to change to window 9, and then type:

ctrl-a, then n

(which moves you to the next window). So, if you wanted to move to window #20, you would have to go to window 9, and then type:

ctrl-a, then n

ten times. For this reason, opening more than 10 screen windows is impractical. It is also unnecessary, which we will see later when we get to nesting.

Another problem is bad detaches. Even though screen is essentially built on the concept of being able to reattach the session no matter how it was detached, there can indeed be bad detaches that can leave the screen in a zombie state that cannot be reattached no matter what you do. There are two ways to avoid this:

1. Do not run more than one screen session on the same host at the same time (update- we can do this if we run with –m). If you log in from one place and start a screen, and then login from another place and start a second screen, theoretically you can reattach to them selectively with their process IDs. However, if one or both of them is detached ungracefully (through loss of connection, rather than running ctrl-a, d, to detach) in practice one or both of them will be zombie-fied and will have to be killed from the command line - and all windows open in each screen will be killed off as well. So don't start more than one screen session on one host at the same time.
2. Always force a detach of a screen before reattaching it. Generally, we are logged in and using a screen, and then we detach the screen with:

ctrl-a, then d

then we log on later, perhaps from somewhere else, and reattach with:

screen -r

and that is all that needs to be done. However, in practice, it is always best to run:

screen -d
screen -r

to reattach - even if the detach was graceful. screen -d does a force detach of the screen that is running on the system, which is useful in the case of ungraceful detaches, because the screen will think it is still attached for some unspecified amount of time. Since force detaching does no harm if the screen was gracefully detached, it make sense to simply do this every time you reattach, just in case.

PLEASE NOTE: when you log into the screen host server, your home directory will contain a script named "dr" which contains nothing but:

screen -d
screen -r

Therefore, upon logging in, you simply run:

dr

and you will reatach yourself - and it will be safe no matter how ungracefully you detached. Presently, upon logging in dr will automaticallt run. To disable edit /home/user/.cshrc and comment out dr

If you are using screen and you suddenly find yourself at a blank screen, you probably hit the "-" key instead of "0" as you were navigating between windows. Hitting:

ctrl-a, then -

brings you to a special blank screen - sort of a "supervisor key", or security screen. You can leave this screen just by moving to some other window.

Scrollback is achieved by hitting:

ctrl-a, then <esc>

you will then have access to the same navigation keystrokes that you have in `vi` - you can page up with ctrl-b, and back down with ctrl-f, or move line by line with h-j-k-l ... and you can exit the scrollback buffer by simply hitting escape.

The default scrollback buffer is rather small though, so when you start your screen for the very first time (from scratch) you can start it with:

screen -h 10000

which gives you 10000 lines of scrollback.

The command sequence ctrl-a is very commonly used in the shell and in pine, etc., to move the cursor to the beginning of the line ... but since screen interprets that as a command character, how can we type that ? Simply hit:

ctrl-a then a

and that will be equivalent to hitting ctrl-a in a non-screened terminal.

Nesting screens

Ten windows is nice, but it is not nearly enough for what we do at JohnCompanies. Therefore, we need to nest screens. This means we run the screen command inside of one or more of our screen windows. There are two problems with this:

The first problem is that, as we mentioned above, we should not start more than one screen on the same system ... this is true, again, because in the event of a ungraceful detach, it is almost impossible to reattach properly to each one. However, this can be worked around by starting the new screens on a second system - and making sure that second system is so stably connected to the first system that there will not be ungraceful detaches - that is, there will be no ungraceful detaches between the main system and the second system. You may still have ungraceful detaches from the main screen session - that is ok.

The second problem is that you need to define a new command key in the nested screens, since ctrl-a is used in the main screen. Luckily screen allows you to set an alternate command key sequence when you start screen, so you can use one sequence for the main screen, and another to navigate between sub-screens.

So, here is how it is done. First, log into a system 1 where no screens are running, and start a new screen with:

screen -h 10000

You are now in screen with one window. Type ctrl-a, then c, to create a second window. You can move to the second window with:

ctrl-a, then 1

and back to the first with:

ctrl-a, then 0

Now, in the first window, ssh to system 2. Once logged into system 2, start screen again, on system 2 this time, with:

screen -e^Pp -h 10000

The -e flag allows us to set ctrl-p as the command character, and we once again set a 1000 line scrollback. So at this point, we have not gained anything - we still have one window open in 0, and one window open in 1. But now, in window 0, hit:

ctrl-p, then c

You now have two windows in window 0 ... and you can move between them with:

ctrl-p then 0

and

ctrl-p then 1

You can create a third window with:

ctrl-p, then c

You now have three windows in the first window. So now let's go back to ctrl-a, and move to window 1 of the original screen by hitting:

ctrl-a, then 1

Now, in window 1, ssh to system 2 ... the same system you just ssh'd to from window 0 ... and once logged in, run this again:

screen -e^Pp -h 10000

Now, earlier we warned against starting multiple screens on the same system ... but in this case, we are doing just that. The reason it is safe here is because it is presumed that the connection between system 1 and system 2 is a local LAN connection that will not be interrupted. Because we can assume that there will never be a ungraceful detach between the main screen and the nested screens, this is safe. Remember, you can still detach ungracefully from the main screen - since it is the only screen running on system 1. The main screen knows nothing about the nested screens in each of its windows - all it knows is you ran an ssh command in each of its windows.

Using this nesting method, we can effectively run 100 windows in a single screen session, navigating through windows 0-9 in the main screen, and nested screens 0-9 in each window of the main screen.

Finally, if you ever become ungracefully detached and cannot reattach (unlikely, since the screen you attach and detach, the main screen, is the only one running on that system) you should just kill the screen processes, and then log onto system 2 and kill every single screen process there, and just start from scratch.

Again, by using the "dr" script, which forces a detach (in case the running screen still thinks it is attached) before reattaching, this is extremely unlikely to happen.

It is, however, very common to see the first incarnation of "dr" to fail ... it may complain that there is no running screen to attach to ... just run "dr" a second time and it will attach. This is common when you ungracefully detach and very quickly log back on to reattach. You should only be concerned about having to start over from scratch if you run "dr" several times and still do not attach.

Nested screens on the same host:

start screen

start subscreens inside of a screen with the -m option:

screen -m -e^Pp -h 10000 -S mailboxes

reattach properly with:

screen -d -RR

(which grabs the first screen, by PID)

So ... dr should now read:

screen -d -RR

But it doesn’t and yet it still works. Whatever.

Screen History

A very useful aspect of screen is the ability to look back at previous screen output. This is easily done as follows:

ctrl-p, then <escape>

you’re now in the vi editor and you can scroll up down, etc. To exit history mode, press <escape> once.

Another note about this- you can also ctrl-a, then <escape> to get at screen history. This is usually only useful as a means to get at history which was lost in a “p” window. But note that this history will be all over the map as it includes history from any of the sub-p screens which the a screen contains.

Screen Organization

At JohnCompanies, we currently run a main screen with 10 windows, each window having between 2 and 10 (or more) nested windows inside of it.

The main johncompanies server which hosts all the screens is simply nat.johncompanies.com (or 69.55.233.195). Simply ssh as your user to nat.johncompanies.com, and your home directory will contain the "dr" (or 'hr') script for reattaching.

The window organization is roughly as follows (the actual way you organize your screens is up to you, this is the organization for the user 'user'):

Window 0:
        0 - info@johncompanies.com pine inbox
        1 - support@johncompanies.com pine inbox
        2 - payments@johncompanies.com (or sales@) pine inbox
        3 - linux@johncompanies.com pine inbox

Window 1:
        0 - johncompanies.com root shell, pwd = /usr/local/www/jc_pub/data
        1 - johncompanies.com root shell, mysql
        2 - johncompanies.com root shell, running tail -f /usr/local/www/signup/ipn.log
        3 - johncompanies.com root shell, pwd = /usr/local/www/scripts

         - dev windows used by dave -
        4 - devweb.johncompanies.com root shell, pwd = /usr/www
        5 - devweb.johncompanies.com root shell, mysql
        6 - devweb.johncompanies.com root shell, root shell
        7 - devweb.johncompanies.com root shell, root shell

Window 2:
        0 - ns1c.johncompanies.com root shell, pwd = /etc/namedb/s
        1 - ns2c.johncompanies.com root shell, pwd = /etc/namedb/s
        2 - firewall root shell [69.55.230.1]
        3 - backup1 root shell
        4 - backup2 root shell
        5 - backup1 root shell
        6 - backup1 root shell, mysql

Window 3:
        1 - jail1 root shell
        2 - jail2 root shell
        7 - jail7 root shell
        8 - jail8 root shell
        9 - jail9 root shell

Window 4:
        0 - mx1 root shell
        1 - jail11 root shell
        3 - mx2 root shell
        6 - jail16 root shell
        7 - jail17 root shell
        8 - jail18 root shell
        9 - jail19 root shell

Window 5:
        0 - quar1 root shell
        9 - virt9 root shell

Window 6:
        1 - virt11 root shell
        2 - virt12 root shell
        3 - virt13 root shell
        5 - virt15 root shell
        6 - virt16 root shell
        7 - virt17 root shell
        9 - virt19 root shell

Window 7: (log in as user console)

(note- order of "O" windows may not always be accurate. Simply cycle through them till you find the server you want with the sequence: ctrl-o n)
P0:
O0 - 3750
O1 - switch-p1a
O2 - switch-p1b
O3 - switch-mx1
O4 - switch-mx2
O5 - switch-3550
O6 - switch-p1
O7 - switch-p3
O8 - switch-p4
O9 - switch-p13
O10 - switch-p16
O11 - switch-p17

P1:
O0 - nat
O1 - mail
O2 - bwdb
O3 - backup1
O4 - backup2
O5 - firewall (live firewall)
O6 - gate (backup firewall)

P2: 
O0 - jail1 serial console
O1 - jail2 serial console
O2 - jail3 serial console
O3 - jail7 serial console
O4 - jail8 serial console
O5 - jail9 serial console

P3: 
O0 - mx1 serial console 
O1 - mx2 serial console 
O2 - jail11 serial console
O2 - jail17 serial console
O3 - jail18 serial console
O4 - jail19 serial console

P4:
O0 - quar1 serial console
O1 - virt9 serial console
O2 - virt11 serial console
O3 - virt12 serial console
O4 - virt13 serial console
O5 - virt15 serial console
O6 - virt16 serial console
O7 - virt17 serial console
O8 - virt19 serial console

P5: (i2b)
O0 - switch-p20
O1 - switch-p21 
O2 - switch-p22
O3 - switch-p23
O4 - switch-p24
O5 - switch-p25
O6 - switch-p26
O7 - switch-p27
O8 - nat2
O9 - firewall2
O10 - backup3
O11 - bwdb2

P6: (spare @ castle)

Window 9:
        1 - backup2 root shell - spare window for exec quad/safe
        2 - backup2 root shell - spare window for exec quad/safe
        3 - backup2 root shell - spare window for exec quad/safe
        4 - backup2 root shell - spare window for exec quad/safe

Window 0 is usually split into 2 panes to allow easy viewing of the 2 support queues. To move between them, ctrl-p, then the tab key. To un-split the panes, ctrl-p then Q. To resplit the screen, ctrl-p , then S

Some of the windows have a nested window of zero with nothing in it ... this is because they are numbered based on machine name ... window 1 is jail1, and so on, but of course you need to have a window 0 ... so it just sits empty.

On the console screen server (sun), we've also fully automated console screen creation on the console server. Login as console and execute /home/screen/startconsoles NOTE: *** only execute startconsoles if there are currently no consoles running *** if you run startconsoles while other consoles are running you'll end up with a big mess. TODO: update this script

To disconnect from a serial console session, hit CR to get to a blank line (regardless of screen echo) and hit ~~. (2 tilde’s and a period, CR)

Sharing screen

CTRL-A
:multiuser on
CTRL-A
:acladd heath