“Large” Memory Sun Host

We recent acquired the host formerly known as “Sunblast”, a v890 with 16 UltraSPARC IV 1.3GHz CPUs and 64GB of RAM.  We added this to the cluster in order to facilitate the execution of large memory, multi-threaded simulations using HFSS, ANSYS, and some other FEM and Multi-Physics applications.  Currently, we’re testing with a few users over HFSS.  The system had been under-utilized before, accepting logins and running XDMCP sessions for various users.  Now that it is part of the grid and is properly managed by GridEngine, we can put this hardware to work doing what it was meant to.  Hopefully, we’ll be able to add to or replace this system with some newer multi-core, multi-socket systems with even more RAM (on the order of 128GB or more).  A general announement of availability should hit the Circe and Sunblast mailing lists within the next couple of days.

MSL Cluster

Dr. Ivan Oleynik of the Physics department just added 36 Dual-Quad Core Opteron 2384s with 16GB of RAM on InfiniBand with the help of a DoD grant.  After ironing out some issues with the IB Verbs library and OpenMPI,  we recompiled our OpenMPI libraries to support PSM (QLogic’s user-space IB driver interface) along with OpenIB and released the system for use today.  Within a few weeks, those systems should be part of a low-priority queue where extra cycles can be utilized by other researchers.

Storage

After several months of testing and design, the new storage system has been brought online, providing 12TB of space for /home directories.  The storage is clustered between two NFS servers which attach to two x4500s acting as backing stores.  The x4500s are mirrored using ZFS to provide data redundancy and high-availability.

Networking

Our Force10 S50 switches have all been updated to the latest OS version, FTOS 7.8.1.0.  This OS is based on FreeBSD and provides greater stability, performance, and features than the previous OS, SFTOS.  We’ve also made necessary changes to our network configuration to allow for multiple uplinks to USF’s network for redundancy.  The OS upgrade alone has helped to reduce issues with dropped packets and false positives that have been plaguing our monitoring system for quite some time.

Software

We’re planning updates to supported applications such as Matlab, Maple, Vasp, and others as well as the decomissioning of several older versions of these and other applications and libraries.  These will be announced in the coming weeks.

Our latest and greatest project was the construction of a 120 node beowulf cluster paid for by a National Science Foundation grant.  Each node, a SunFire x4150, contains two Quad-Core Xeon X5460s, 16GB of RAM, and is connected via InfiniBand.  Full specifications are provided below.  The project involved a number of preliminary steps not the least of which was finding a suitable data center. Our 12′x40′x10′ cube was no longer sufficient for running new hardware (and is, arguably, insufficient for what’s currently running in it!). Luckily, the Central Florida Regional Data Center in the SVC building was the perfect place for this new system. In the process of renovation and having raised floors, this would be the perfect place to jack in some more power and feed in some water for cooling.

To keep costs down for our HVAC needs, we went with InRow APC units. Since USF sports a central chilled water facility, it was simply a matter of running some pipes. The units themselves book-end the cluster and there is one between each rack.

Here are some preliminary photos of the work.

Installing Chilled Water Pipes

APC InRow RC Units

Electrical Panel Install

This is one completed rack

More to come soon!

Brian and I thought- cool. Let’s script these operations through SSH then, and make our lives easier. If we can automate these tasks, it would be a lot easier to manage maintenance of 120 new machines.

More, after the jump.

Usually, when you wish to remotely execute a command with SSH, you just tack whatever you want on the end of the ssh command, and it will execute the command on the remote host and return the results to you. However, not all systems support this. Notably, ILOM interfaces that you find on switches and some Sun hardware, do not support this kind of thing:

ssh root@ilom1 ls
Password:
Invalid operation


Invalid operation? But why, you ask? You can ssh into the host and execute commands there, but you can’t push a command out to be executed? What gives?

Simple. SSH daemons can operate in two modes- interactive and non-interactive. Interactive mode is when you log into the host and type commands there. Non-interactive is when you pass commands like the failed example above. The problem is, some ssh daemons, mostly on switch hardware and other network appliances, do not support the non-interactive mode. Which means, that if you want to admin these devices, you have to login to each of them and type in the commands, one by one.

Wrong.

Brian had a really cool idea. What if we wrote a program that initiated an interactive ssh session and we passed commands to it for it to execute? In short, the program would connect like a human would with his ssh client.

Enter Net::SSH::Perl, which is a module that allows you to initiate ssh sessions, and has all the features your standard ssh client has. We could just start an interactive session with it, and execute a whole bunch of commands through the script. One problem though- interactive sessions expect the user to type in the password into the TTY. That’s lame. How can we avoid that? It isn’t like we have sshkeys built into the host, and even if they could, they won’t when we first use them. I wouldn’t want to set that up for 120 machines by hand.

Basic UNIX knowledge tells us that every program has STDIN(program input), STDOUT (program output), and STDERR(out-of-band diagnostic messages/errors). So, in our case, what we need is to start a separate process and push our password and our commands through STDIN, and read our output through STDOUT/STDERR.

Perl saves the day again, with a module called IPC::Open3. It allows us to create those handles and attach them to a separate process for us to manipulate. In our case, we just want to fork. The parent process will be responsible for sending commands and aggregating output, and the child will be responsible for actually initiating the connection and invoking those commands and reading the output. We will also use IO::Select to manage reading from STDOUT/STDERR.

Final outcome: we can execute commands and process it’s output. We wrote a script just to print the mac address of the ILOM:

7:0 devel # ./ssh_test
MAC Address: 00:14:4F:20:D6:90


Here’s some code for you to play with: ssh_test Edit to your heart’s content.

A WARNING: This piece of perl does stuff that the designers of SSH did not intend. This script involves storing a password, and if you do not give this file proper permissions, you are guaranteeing a way to get rooted. You’ve been warned. (Might I suggest chmod 700 with chown root:root?)

With that said, Happy Hacking!

• Monitor Gigabit and Infiniband switch interfaces for throughput, types of traffic, errors, etc.
• Monitor UPS’s for Voltage, Battery Capacity and Temperature.
• Tie it together with SGE to see network performance on a job-by-job basis.
• Link it to our current Ganglia installation to see status of individual nodes on a job-by-job basis.
• Have all that information accessible in the same place and somehow have it look pretty.

Well, Brian didn’t have the time to implement it himself, so he bequeathed the daunting task to me. Let’s see how that turned out, after the jump.

The first step was to figure out how to harvest all of this information. Lucky for us, our Force10 Gigabit and Cisco Infiniband switches talk SNMP. I put two and two together, and I realized that MRTG was perfect for this task. The only problem was, most MRTG installations (and the config file generator) produces one giant config file. When Apache serves you the data in your browser, you have a ton of graphs to sift though. What a pain. So, my solution to this problem was this-

• Create a single MRTG config file for each service per interface per host. We have quite a few switches and UPS devices; meaning- lots of files. Oh, configure them to use rrdtool.
• Write templates for each service you want to monitor.
• Hand-hack a perl script per type of device to generate all the config files using the templates and regular expression replacement. Each perl script reads a simple newline-delimited flat file containing all the hosts you want to monitor for a particular type of device. Each output file should be named using the hostname, interface, and service.
• Generate a global MRTG config file that reads in all the small config files and generates .rrd files. Configure crontab to run MRTG against the global file every 5 minutes.
• Tie it all together using the 14all.cgi script by making symlinks to that script with the naming convention.

Don’t worry. I’ll give you an simple example.

1. Generate the config files

You want to monitor all the incoming and outgoing bits on each active interface on a switch. Here’s a template that will do this: bitstemplate. I would take a look at this, if I were you

See HOST and INTERFACE in that file? We are going to replace those for each host we specify and for each active interface the host has. Here’s a perl script that will generate the files for you: genswitchconf. It’s pretty short, and you can add more templates to load for a more comprehensive solution.

One more thing- make a file called switchhosts and add one hostname per line. When you run the script, the perl script will determine what interfaces are active for each host, and generate a config file for each one.

2. Tell mrtg to use the config files

Copy all the config files to a location, perhaps /etc/mrtg. Next, create a file called /etc/mrtg.conf using this wicked cool one-liner:
for i in `ls /etc/mrtg/*.cfg`; do echo "Include: $i">> /etc/mrtg.cfg; done

Then put this in crontab using crontab -e:
*/5 * * * * env LANG=C /path/to/mrtg /etc/mrtg.cfg --logging /var/log/mrtg.log

Your .rrd files should be automagically be generated. Check the logfile for errors.

3. Use 14all.cgi to make it easily web-accessible

This perl script is cool. Stick it in the /cgi-bin/ of your webspace, and configure it to use symlinks to load the desired metric. This ain’t an Apache or MRTG/14all.cgi tutorial, so the logistics of this is outside the scope of this article. Next, use this even more awesome one-liner to create your symlinks:
for i in `ls /etc/mrtg/*.cfg`; do ln -s /usr/lib/mrtg/cgi-bin/14all.cgi /usr/lib/mrtg/cgi-bin/`basename $i .cfg`.cgi &> /dev/null; done

4. Write a perl script to generate html to browse the device

I have a sample for the current example: genswitchhtml. When you run it, you should get a single html file per switch, containing a table which links to the 14all.cgi with the desired host and interface. Copy these to your webspace. I’d write an index for all these, if I were you.

5. Enjoy.

It’s a basic solution, but it gets the job done. I generated config files for 8 switches, a ton of UPSs, and a few Infiniband switches, making a one-stop shop for looking at performance metrics for all these devices. In the next article, I’ll show you how to use SGE to your advantage so you can see metrics on a per-job basis. Have Fun!

-Amin

If there was only a way to speed up the process..

It would be great to just ask the computer:

“Hey, You got function foobar installed?”

And it would be even greater if the computer replied:

“Yea, I got it, and here’s the library that has it!”

Guess what? In the nineties a professor at USF named David Rabson implemented such a solution called Librarian. It was written in C, used ndbm as it’s database backend, and did the job. You gave it the symbol/function name, and it gave you a list of possible candidates to link against. Good times.

But then Brian and I looked at it. We both agreed, it did the job. But there were some key features that we wanted that were missing, primarily being able to query using the all-powerful regular expression. Since ndbm didn’t support regexes, and after some research I found that MySQL did.. one thing led to another and..

Enter Librarian-NG. I ended up reimplementing the whole sucker in Perl. Even edited the man pages.

Example:


[aastaneh@host ~]$ lbnlookup malloc
Symbol Name Location
---------------------------------------
malloc: /usr/lib/libc.a
malloc: /usr/lib/syslinux/com32/libcom32.a
malloc: /usr/lib64/libc.a
malloc: /opt/priv/openmpi-1.2.4/pgi-7.0-7/lib/libopen-pal.a
malloc: /opt/priv/openmpi-1.2.3-pgi-7.0-i386/lib/libopen-pal.a


Download: librarian-ng-1.0.0.tar.gz

Until today.

Using the powers of ZFS, rsync, and Bash, I have devised a solution which makes accessing and restoring backups so simple, even normal users can do it!

Let’s say I have an account on a multi-user system, and I accidentally deleted a file in my home directory:

[aastaneh@login0 ~]$ rm fftw-3.1.2.tar.gz

It’s a really important file, and I need it back right now. I call my trusty sysadmin to restore the file for me. He can either enter in a bunch of hard-to-remember esoteric commands, or do this:

[root@backups ~]# zfs_restore aastaneh
Now in ZFS File Restore Environment
Restricted Command Set: dates setdate restore [file], ls, cd [dir], exit
backupsh snapshot>

The admin is now given a custom command shell which has a simple set of commands. We use ‘dates’ first to give the listing of all the snapshots present:

backupsh snapshot> dates
20071206000201 20071209000143 20071212000103 20071216000135 20071218132440
20071207000228 20071210000134 20071213000210 20071217000054 20071219000106
20071208000251 20071211000107 20071214010339 20071218000233

The ‘20071219000106′ looks the most recent. We choose it with ’setdate’:

backupsh snapshot> setdate 20071219000106

Now we are in my home directory from yesterday. The admin has the ability to dig around in my home directory using ‘cd’ and ‘ls’.

backupsh 20071219000106> ls
. .ssh gmxtest.tgz
.. .viminfo gromacs-3.3.2
.Xauthority abinit-5.4.4 gromacs-3.3.2.tar.gz
.bash_history abinit-5.4.4.tar.gz gromacs-clean
.bash_logout abinitconfigure gromacs-configure.patch
.bash_profile configure-working grompp.out
.bashrc discrep.ieee id_rsa
.elinks discrep.reg id_rsa.pub
.forward discrep.reg-1030 mail
.lesshst fftw-3.1.2 maillog
.matlab fftw-3.1.2.tar.gz scratch
.mbox gmxtest-3.3.2.tgz

We see the lost fftw-3.1.2.tar.gz, so we restore the file with ‘restore’(which just calls rsync on the file with the right options):

backupsh 20071219000106> restore fftw-3.1.2.tar.gz
building file list ...
1 file to consider
fftw-3.1.2.tar.gz
2736360 100% 92.08MB/s 0:00:00 (xfer#1, to-check=0/1) sent 2736798 bytes received 42 bytes 5473680.00 bytes/sec
total size is 2736360 speedup is 1.0
backupsh 20071219000106> exit
(root@backups) Thu Dec 20 13:33:37

Voila! My day is saved. My file is now restored to my home directory, timestamped so that the restore process does not clobber files.

[aastaneh@login0 ~]$ ls fftw*.gz*
fftw-3.1.2.tar.gz-20071219000106

Requirements: The script must be running on a system with ZFS snapshots, probably a Solaris system. This script was written assuming a NIS install somewhere, but I can imagine working around that if NIS isn’t used on the network. Rsync was also used, but I also think another option can be used as well (scp?)

Caveats:

• If the script does not work successfully the first time, first check with the ZFS utilities to see if a mountpoint has been defined already for the user you are trying to restore. You will have to manually perform the unmount before the script will work properly.
• The restored file gets copied to ‘~user’, not to it’s respective subdirectory where the file was restored from.
• This script needs some better sanity checks to ensure that future users don’t try to traverse up the directory structure to / or something silly

   

   

Code: zfs_restore source

Lucky for us, most new server hardware comes with some on-board hardware that provides an IPMI service. IPMI refers to Intelligent Platform Management Interface and it provides various mechanisms for chassis power control, system event logging, hardware monitoring, and even serial console access. On Dell systems, an integrated BMC or Board Management Controller provides the necessary hardware interface to provide the IPMI service while piggy-backed to one of the system’s NICs to provide remote accessibility. On the Sun x4500, the ILOM or Integrated Lights-Out Management unit provides the IPMI services through an attached service processor with its own NIC and on-board operating system (that happens to be Linux).

The really cool thing is that vendors who implement IPMI like Sun and Dell tend to do a pretty complete job populating all of the potential data points for the IPMI SDR (Sensor Data Record) and they also build-in factory specified thresholds for determining the failure of any particular component. All temperatures, fan speeds, status indicators, etc. have established nominal operating ranges and IPMI has a very easy way to determine if a device is operating within those established parameters.

The first thing that we’ll have to do to set up IPMI monitoring through Nagios is, of course, to make sure that you have a ready and running copy of Nagios running somewhere and that you are familiar with adding plugins to Nagios’ command list. If you don’t get started reading here: http://www.nagios.org/docs

The next thing you’ll need is a current installation of OpenIPMI and OpenIPMI tools (only necessary for Linux, in this case). It will be useful to have this installed on each Linux host with an IPMI adapter such as the Dell PowerEdge server. With CentOS, you need only issue the following yum commands to get this software:

[root@host ~]# yum install OpenIPMI OpenIPMI-tools

This will provide the necessary command line tools such as ‘ipmitool’ and the init script we’ll use to load the appropriate kernel modules. Once you’ve installed these packages, you can edit and run the following script to establish a base configuration for your IPMI adapter:

#!/bin/bash
# Amin Astaneh, Research Computing, USF
# impiscript: configures ipmi for network access on a node.
CHANNEL=1
#
# First, ensure that all the modules are loaded
service ipmi restart
#
# Use NIS to determine what IP address to use. This merely adds 200 to the third octet.
HOST=`hostname`
#IPADDR=`ypcat hosts.byaddr | awk "/$HOST/ { str=\\$1; split(str,ip,\".\"); printf \"%s.%s.%s.%s\", ip[1], ip[2], ip[3] + 200, ip[4] }"`
#
# In this case, just hard-code whichever address you want for the adapter
#
IPADDR=x.x.x.x
NETMASK=y.y.y.y
GATEWAY=z.z.z.z
#
ROOT_PW='XXXXXXXXXX'
NAGIOS_PW='XXXXXXXXXX'
#
# The package name is OpenIPMI-tools
ipmitool lan set $CHANNEL ipaddr $IPADDR
ipmitool lan set $CHANNEL ipsrc static
ipmitool lan set $CHANNEL netmask $NETMASK
ipmitool lan set $CHANNEL defgw ipaddr $GATEWAY
ipmitool user set password 2 "$ROOT_PW"
ipmitool lan set $CHANNEL access on
ipmitool user set name 3 nagios
ipmitool user set password 3 "$NAGIOS_PW"
ipmitool channel setaccess 1 3 privilege=3 ipmi=on
ipmitool mc reset cold
#

I’ve seen that in many cases, you’ll need to completely power-cycle the box for the IPMI adapter to actually work. This can mean either a full reboot or a complete unplugging of the system power.

Before we get down to playing with this device, what about configuring the ILOM on the x4500? We’ll, you’ll first want to read through the Sun Lights-Out Manager (ILOM) Administration Guide provided here http://docs.sun.com/app/docs/coll/x4500-rels-ilom. You’ll want to set up a basic configuration where you can log in via ssh as administrator to the device. This means we’ll only be preoccupied with setting up a user account. Go ahead and log into the device and issue the following commands to create the ‘nagios’ user:

-> cd /SP/users
-> create nagios
-> cd nagios
-> set password='XXXXXXXXX'
-> set role=Operator
-> exit

You’ll now be able to use the ‘nagios’ users on both the Dell BMCs and the Sun ILOM to access IPMI SDR information for monitoring the devices. Lets have a look at some sample output from ‘ipmitool’ (see ipmitool(1) in the man pages for information on command line syntax).

Here is an example using ‘ipmitool’ on a Dell PowerEdge server:

[user@host ~]$ ipmitool -H x.x.x.x -U nagios -P 'XXXXXXXXX' -L OPERATOR -I lan sdr list all
Temp | disabled | ns
Temp | disabled | ns
Ambient Temp | 25 degrees C | ok
CMOS Battery | 0x00 | ok
VCORE | 0x01 | ok
VDDIO | 0x01 | ok
VDDA | 0x01 | ok
VTT | 0x01 | ok
VCORE | 0x01 | ok
VDDIO | 0x01 | ok
...

The nice thing about ’sdr list full’ is that all available metrics are read and processed against the built-in threshold values. This makes for very easy parsing. The fields with ‘ns’ in the 3rd column are obviously unavailable so we can grep them out pretty easily. This standard is also followed for the x4500. The only difference is the LAN interface used (see option ‘-I lan’ in the above command line). For the x4500, we’ll be using the LANplus interface. Here’s an example:

[user@host ~]$ ipmitool -H x.x.x.x -U nagios -P 'XXXXXXXXX' -L OPERATOR -I lanplus sdr list all
proc.p0.t_core | 51 degrees C | ok
proc.p1.t_core | 49 degrees C | ok
dbp.t_amb | 25 degrees C | ok
io.front.t_amb | 39 degrees C | ok
io.rear.t_amb | 40 degrees C | ok
proc.front.t_amb | 29 degrees C | ok
proc.rear.t_amb | 34 degrees C | ok
ft0.prsnt | 0x02 | ok
ft0.f0.speed | 7700 RPM | ok
ft0.f1.speed | 7800 RPM | ok
...

Well anyway, Nagios plugins in bash are incredibly easy to write and since we have all of the output we’ll ever need for monitoring purposes, lets just make the ‘ipmitool’ commands above the basis for our monitoring scripts. Here’s one for the Dell’s:

#!/bin/bash
###################################
# check_x4500
#
# Checks status of machine via ipmi
#
# Amin Astaneh, aastaneh@rc.usf.edu
# 10-01-2008
###################################
# We can add 200 to the third octet for our Dell hosts so that we don't have to create a separate nagios host for the ipmi adapter
HOST=`echo $1 | awk -F'.' '{ printf "%s.%s.%s.%s", $1, $2, $3 + 200, $4 }'`
LOGS=/usr/share/nagios/logs
IPMI="ipmitool -I lan -H $HOST -U nagios -P rc_ipmi_info -L OPERATOR"
RESULTS=$($IPMI sdr list all | egrep -v '^.*\|.*\|.*(ok|ns)$' | awk -F'|' '{ print $1":"$2":"$NF}');
#
if [ -n "$RESULTS" ]; then
echo "######### Status Display ###########" > $LOGS/$HOST.log
echo $RESULTS >> $LOGS/$HOST.log
echo "######### System Event Log ###########" >> $LOGS/$HOST.log
$IPMI sel list >> $LOGS/$HOST.log
echo "WARNING: <a href=\"https://nagios.rc.usf.edu/logs/$HOST.log\">See Logfile</a>";
exit 1;
else
echo "OK: All Components Online."
exit 0;
fi

In this script, we simply look for lines in the output of sdr list full that contain something other than ‘ok’ or ‘ns’ in the 3rd column. If a result is returned, we know something is awry and we print that line containing a URL to a log file along with WARNING or CRITICAL (which Nagios uses to determine if there is a problem). The log file contains the spurious outputs from the command as well as a listing of the system event log.

With a couple small modifications, this same script can be used with the x4500 ILOM as we see here:

#!/bin/bash
###################################
# check_x4500
#
# Checks status of machine via ipmi
#
# Amin Astaneh, aastaneh@rc.usf.edu
# 10-01-2008
###################################
#
HOST=$1
LOGS=/usr/share/nagios/logs
IPMI="ipmitool -I lanplus -H $HOST -U nagios -P rc_ipmi_info -L OPERATOR"
RESULTS=$($IPMI sdr list all | egrep -v '^.*\|.*\|.*(ok|ns)$' | \
awk -F'|' '{ print $1":"$2":"$NF}');
#
if [ -n "$RESULTS" ]; then
echo "######### Status Display ###########" > $LOGS/$HOST.log
echo $RESULTS >> $LOGS/$HOST.log
echo "######### System Event Log ###########" >> $LOGS/$HOST.log
$IPMI sel list >> $LOGS/$HOST.log
echo "CRITICAL: <a href=\"https://nagios.rc.usf.edu/logs/$HOST.log\">See Logfile</a>";
exit 1;
else
echo "OK: All Components Online."
exit 0;
fi

Here’s the same thing only the ILOMs are defined as their own host in Nagios (so we don’t need to do any address translation) and we use the LANplus interface to communicate with the device.

Now, all thats left to do is to add this plugin to Nagios and assign it as a service to the appropriate hosts. Now you have complete monitoring of all the hardware’s vital statistics while utilizing the factory-specified ranges for determining thresholds with a very minimal amount of scripting. A big thanks to Amin for getting these scripts written.

Enabling ssh connections to an automated kickstart configuration is fairly painless. To start, you’ll need a few things:

1. A copy of /usr/sbin/sshd that is accessible. If you are installing over NFS, this can be copied into your distribution directory. If you are installing over HTTP or FTP, you’ll need to use curl to download the binary from some location
2. A basic sshd_config file, ssh_host_key and ssh_host_key.pub
3. An ssh public key that can be used to authenticate a root login during the installation

A really nice thing about the base system that is contained in an Anaconda install is that it contains all of the necessary libraries to fire up an sshd demon during the installation. This made our work incredibly easy. At this point, we assume that you have a working ks.cfg that you push out to your remote hosts for automated installations. What you’ll need to do is add a %pre script to the file that will configure sshd and start the daemon during the very begging of the installer. The following is part of an example ks.cfg file. Pay attention to the section that says “Lets enable sshd”:

%pre
echo "root:x:0:0:root:/root:/bin/sh" > /etc/passwd
echo 'root:<your_hash_here>:13732:0:99999:7:::' > /etc/shadow
echo "sshd:x:74:74::/var/empty/sshd:/sbin/nologin" >> /etc/passwd

# Make necessary directories and files for logins
mkdir -p /var/empty/sshd
chown root /var/empty/sshd
chmod 700 /var/empty/sshd
mkdir -p /var/log
touch /var/log/btmp
chmod 600 /var/log/btmp
chmod 400 /etc/shadow

# Add public key support
mkdir -p /root/.ssh
chmod 700 /root/.ssh
cp /mnt/source/ssh/authorized_keys2 /root/.ssh

# Start sshd
/mnt/source/ssh/sshd -f /mnt/source/ssh/sshd_config

For this script to work, you’ll need a bunch of things that this document wont cover. We’ll focus strictly on the sshd part. In this script, we require a directory, $KSROOT/ssh where $KSROOT is the directory where your CentOS/RedHat/Fedora installation media is located. The following files will need to be in this directory

• authorized_keys2: Put a public key in here for authentication purposes
• sshd_config: A basic sshd configuration. An example is provided below
• ssh_host_rsa_key: You can copy this from an existing installation that you want to mirror or generate a new one with ssh-keygen
• sshd: The sshd binary from an install of the OS you are installing with Anaconda

The following sshd_config should be sufficient for most cases:

Port 22
Protocol 2

# Logging
SyslogFacility AUTH
LogLevel ERROR

# Authentication:
PermitRootLogin yes

HostKey /mnt/source/ssh/ssh_host_rsa_key

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes

# Change to no to disable s/key passwords
ChallengeResponseAuthentication no

UsePAM no

# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL

Obviously, you should not expect these configuration files to be drop-in-ready and you will need to do a little bit of tweaking to get them to play nicely with your environment. The nice thing is, now, you can remotely login to your host as it is installing (or, in this case, if the install fails) to get a list of the hardware, check the kernel ring buffer with dmesg, and run some basic diagnostics. Even though VNC allows you to see the installer, you cannot use a shell. This is, to my knowledge, the best way to get a remote shell during an Anaconda Kickstart install.