Install OES Components and Configure eDirectory:

• Start the OES install:
  • YaST: Software, Add-on Product

• Select Local Directory or NFS (depending on where your ISO images are)
• If using NFS, specify the source server’s fully qualified name address
• Check the ISO Image box and set the path to /install/OES2-SP1-x86_64-CD1.iso

iManager/ConsoleOne Tasks:

• Add an eDirectory replica to new server:

• • In iManager, go to Partition and Replica Management, Replica View
  • Enter SITE.XXXX, click OK
  • Click Add Replica. Select the new server and click OK.

Configure NSS (not needed if only re-installing the OS for a failed server – following these steps will delete existing NSS pool/volumes):

• Run nssmu from the command shell
• Go to Devices, select the logical drive for the NSS pool, and hit F3 to Initialize it (be careful, make sure you have selected the correct drive!)
• Create new pool named DATA using all of the logical drive
• Create new volumes on DATA pool

Turn off oplocks:

• Edit /etc/opt/novell/nspserv.conf and make sure it has these two lines:
  • OPLOCK_SUPPORT_LEVEL 0
  • CROSS_PROTOCOL_LOCKS 1
• This change requires a restart of NCP server (reboot)

Register and Patch the server:

• Register the server with Novell:
  • At the command prompt:
    suse_register -a email=xxx@xxxxx -a regcode-sles=XXXXXXXXXXXX -a regcode-oes=XXXXXXXXXXXX

• Update the server using YaST, Software, Online Update (this works in text mode with screen)

This document describes the steps necessary to deploy a new web application via a reverse HTTPS proxy on the Blue Coat Proxy SG. It involves setting up a virtual IP address on the Blue Coat that is servicing the web servers, and using the proxy to terminate an SSL connection from the client to that address. It will then forward the request to the servers themselves, in a load balanced manner. In order to provide full acceleration from other locations, we will need to set up a SSL intercept on each remote proxy, causing it to intercept the connection and forward it across the ADN to the master that is actually servicing the (virtual) website.

Set up Virtual IP (VIP) Address on Servicing Blue Coat Proxy:

• Use the Infoblox Grid Manager to choose an unused IP address in the app server subnet, and add a host record with that address and the corresponding URL for your virtual host, such as virtualserver.example.com
  • (I will add a link here to the Infoblox instructions when I post them.)
• Configuration -> Network -> Advanced -> VIPs
• Click New, enter the virtual IP address
• Click OK and Apply to save the configuration
• You should now be able to ping the virtual IP address by DNS name (ping virtualserver.example.com)

Set up Forwarding Hosts:

• Configuration -> Forwarding -> Forwarding Hosts -> New
• Enter the following values:
  • Alias: a simple descriptive name, such as the host name (webserver1)
  • Host: the fully qualified host name (webserver1.example.com)
  • Ports: check HTTP and enter the host server port (in this case, we use 7778 since the web site is hosted on Oracle HTTP server)
• Click OK
• Repeat the above steps for each host that will service this virtual host
• Click Apply to save

Set up Forwarding Group:

• Configuration -> Forwarding -> Forwarding Groups -> New
• Enter a descriptive group name
• Select each host server that will service the website and click Add
• Click OK and Apply to save

Set up SSL Access Policy in VPM:

• Configuration -> Policy -> Visual Policy Manager -> Launch
• Click on SSL Access Layer tab. If this did not already exist, you would add one by:
  • Click on Policy -> Add SSL Access Layer
  • Give it a descriptive name (SSL Access Layer, for example)
  • Click OK
• Click Add Rule
• If necessary, use the Move Up or Move Down buttons to change the order of this rule
• Right-click on the word Any under the Destination column in the rule you just added and select Set
• Click New, select Server URL
• Set up your matching parameters and click Add, Close, OK
  • In our current example, we’d use Simple Match, and enter the virtual host URL, virtualserver.example.com
  • Note: You could also create a new Combined Destination Object to create a list of URLs
• Right-click on the word None under the Action column in the rule you just added and select Set
• Select Allow and click OK to save

Set up Forwarding Policy in VPM:

• If not already in the VPM, go to Configuration -> Policy -> Visual Policy Manager -> Launch
• Click on the Forwarding Layer tab. If this did not already exist, you would add one by:
  • Click on Policy -> Add Forwarding Layer
  • Give it a descriptive name (Forwarding Layer, for example)
  • Click OK
• Click Add Rule
• If necessary, use the Move Up or Move Down buttons to change the order of this rule
• Right-click on the word Any under the Destination column in the rule you just added and select Set
• Select the same Destination URL that you added under the SSL Access Policy and click OK
• Right-click on the word Send Direct under the Action column in the rule you just added and select Set
• Click New, select Select Forwarding
• Enter a descriptive name (no spaces)
• Under Forward to:, select the forwarding group you created earlier and click Add, OK, OK
• Click Install Policy, then close the VPM

SSL Notes:

You have two options to create the SSL private key and your certificate signing request (CSR). The Blue Coat interface is easier to understand, but openSSL is sometimes more compatible. (NOTE: iManager would not issue a wild card certificate from a CSR generated on the Blue Coat, requiring me to use openSSL.)

Create New Keyring and Generate Private Key:

These instructions assume you are creating the key with the Blue Coat management interface.

• Start by creating a new keyring: Configuration -> SSL -> Keyrings -> Create
• Enter a keyring name.
  • For all of our internal web apps, we are using a wildcard certificate for (*.example.com), so we use something like wildcard.example.com-key as the keyring name.
• If you want to back up the private key or use it on any other device, select Show key pair or Show key pair to director. For greatest security, leave it set to Do not show key pair.
• For all internal purposes, we use 2048 bits.
  • 1024 bits is virtually unbreakable without a supercomputer or large cluster of computers, so it is not insecure, we just use the same number of bits as the default SSH keys on SuSE.
  • If you will be requesting a certificate from a recognized certificate authority, the number of bits should correspond to the certificate product you will order.
• Click OK, then Apply to save the new keyring.

Generate Private Key with openSSL:

• (I will insert a link when I post these instructions.)
• Create a new keyring as described above, but instead of creating a new key, select Import
• Copy and paste in your private key generated with openSSL

Generate Certificate Signing Request (CSR):

These instructions assume you are creating the CSR with the Blue Coat management interface.

• Click on the new keyring and click Edit
• Under Certificate Signing Request, click Create
• Fill out the fields similar to the following:
  • State/Province: Nevada
  • Country Code: US
  • City/Locality: Las Vegas
    
  • Organization Name: Examples-R-Us, Inc.
  • Unit: IT Department
    
  • Common Name: virtualserver.example.com
  • Challenge: (Blue Coat requires an entry, openSSL does not, since this is optional)
  • Email Address: mail@example.com
  • Company: Examples-R-Us
    
• Click OK, Close and then Apply. This will save the new keyring and generate the CSR.
• Click on the keyring and click Edit.
• Copy the contents of Certificate Signing Request box and save to a file (virtualserver.example.com, for example).

Generate CSR with openSSL:

• (I will insert a link when I post these instructions.)
• It is possible to import your CSR into the keyring if desired. This would be handy if you wanted to use the same CSR later to generate another certificate (perhaps with a new expiration date).

Issue Certificate from iManager and Import:

• (I will insert a link when I post these instructions.)

Set Up SSL Intercept Service:

• Configuration -> Services -> Proxy Services -> New Service
• Enter a Name for your service, such as Example SSL Proxy
• Under Service Group, select the group (this is just for organizational purposes, it’s not a critical step)
• Under Proxy settings, Proxy, select HTTPS Reverse Proxy from the drop-down
• Under the Keyring drop-down, select the keyring you created above
• At the bottom, click New to create a new listener
• Choose Destination host or subnet
• Enter the virtual IP address (VIP) you created in the beginning
• Click OK, OK, Apply to save

If you have done everything correctly, you should now be able to open a web browser and type in https://virtualserver.example.com/ and get your website!

Optional – Set Up URL Rewriting:

The Blue Coat Proxy allows you to re-write the request URL. So for example, you can allow the user to enter a “pretty” URL, and hide the actual request string that is sent to to the host server.

• Install the re-write policy by going to Policy -> Policy Files -> Install Local File From -> Text Editor-> Install
• Sample rewrite policy:
• define action virtualserver_example_com_portal
  rewrite(url, “^https://virtualserver.example.com/$”,”https://virtualserver.example.com/prod/”)
  end
  
  define action force_uncompressed
  delete(request.header.Accept-Encoding)
  end

  <Proxy>
  url.host=virtualserver.example.com action.virtualserver_example_com_portal(yes)

  <Cache>
  action.force_uncompressed(yes)

• Storage setup for a standard file and print server:
  • Need two logical drives, RAID 1, 5 or 10
  • First logical drive is for /boot, swap and / partitions (recommend 50GB)
  • Second logical drive is for NSS pool (size accordingly)
• Installation options:
  • Local DVD install: Boot SLES 10 SP1 DVD and start a new install
  • Network install: Boot SLES 10 SP1 CD1 and use the following options:
    • hostip=xx.xx.xx.xx/24
    • gateway=xx.xx.xx.xx
    • nameserver=xx.xx.xx.xx
      
    • nfs://servername/install/SLES-10-SP2-DVD-x86_64-GM-DVD1.iso
  • To perform install remotely with VNC (DVD or network), add these options:
    • vnc=1
    • vncpassword=password
• Select Language, Accept License, select New Installation
• Set Clock and Time Zone info:
  • Set Hardware Clock Set To UTC
  • Region: USA, Time Zone: Pacific
  • Set Time and Date to current at your location (look at your watch)
• Click Partitioning:
  • Create Custom Partition Setup, Custom Partitioning (for experts)
  • You should have two logical drives available, sda and sdb:
    • Create 1GB primary partition on sda, format as ext2, mount as /boot
    • Create 2GB primary partition on sda, format as swap
    • Create a primary partition on sda, use remaining space, format as ext3, mount as /
    • sdb will hold the NSS pool, which will be configured after OES installation
• You can leave the defaults for software selection.
• Click Accept to start installation – Server will format partitions, install software, and reboot.

Staging Step 2 – Final Setup After Reboot:

• Enter password for root
• Enter hostname and domain (use lower case)
  • Uncheck Change Hostname via DHCP
• Network Configuration:
  • Click to Enable Firewall for SSH
    
  • Make sure VNC Remote Administration is enabled
  • Click Network Interfaces, Edit:
    • Add Name Servers
    • Add search domains
• Don’t check for Internet connectivity
• Accept defaults on CA Management and click Next
• Leave Authentication Method on Local, click Next
• Don’t create a local user, just click Next, and Yes that it’s OK
• Next on Release Notes
• Next on Hardware Configuration
• Uncheck Clone This System for Autoyast
• Base install is finished at this point. If you’re doing a VNC install, your installation session will die here. You can reconnect to the new server at this point if needed.

Staging Step 3 – Other Settings:

• If it is a normal file server, copy SLES and OES source ISO’s to server:
  • Create /install
  • Copy the following ISO’s to /install:
    • SLES-10-SP2-DVD-x86_64-GM-DVD1.iso
    • OES2-SP1-x86_64-CD1.iso
• Delete CD/DVD or network install source and add local source for SLES:
  • YaST: Software, Installation Source

• YaST: Network Services, NTP Configuration:
  • Change Automatically Start NTP Daemon to start During Boot
  • Go to Advanced Configuration
  • Add server ntp.doc.nv.gov and Enable for initial sync
• Configure and start SNMP:
  • Copy /etc/snmp/snmpd.conf from another server
  • Enter chkconfig snmpd on to set daemon to auto-start
  • Enter rcsnmpd start to start the daemon

• Enable Systat:

• Move on to http://danobarrjr.net/2010/03/oes-2-installation-and-setup.html
  

This week we finally got our new server ready to hold our SNMP monitoring system (we use What’s Up Gold, and I’ll have to do another post about this sometime). We use an off-box database to hold all the data for this system, rather than the default which just installs Microsoft SQL “light” (MSDE) on the same box. In theory, moving the engine to a new server is a very simple matter–just shut down the engine on the old box, install it on the new box, and point it at the database. I’ve done this a half dozen times, so it should’ve been very simple. One of my system admins ran through a default install on the new server, which installed a local database, then he grabbed me to show him how to connect the engine to the database server. We tried changing the ODBC connection string, and couldn’t get it to connect (don’t remember the error message). We concluded that we might as well just un-install, and re-install without the local database.

The first lesson in all of this is read warning messages VERY carefully. The un-install asked if we wanted to remove our data and settings. We answered yes, thinking it would just remove the local database. Oops. Despite the errors we encountered while trying to connect, we had entered the information to connect to the live database. So the un-install promptly “dropped” the live database on the SQL server. I mean it was completely gone, all traces of it. My heart sunk for just a second, but then I figured, well, we’ll just roll back to last night’s backup. That’s where the fun started.

We are using Novanet Backup on the database product (I’ll have to give a review on another post, but needless to say I like it), and I remembered creating and manually testing the backup job a month ago. However, I neglected to actually schedule the job to run. Oops, again. Now, it’s a whole other discussion as to why this was not noticed, but in a nutshell it’s because it was a new system, and it was just “IT division stuff.” In any case, that first manual backup was our salvation. We have made some significant changes to the network over the last month, and of course we lost all of September’s statistics with regards to uptime, latency, etc. But we had 90% of everything, and with about an hour’s worth of work the system was usable again. We’ll still need to spend a few more hours reviewing the documentation and adding and changing devices to match what’s out there now.

Even though it ended up costing us a half day’s work, our collective butt was saved because we had at least the one backup from a month ago. Had we not had that, it would’ve taken weeks to re-enter everything from our documentation into the monitoring system. Not only that, but we would’ve lost all historical data as to the performance of our entire infrastructure for the last year or so.

From this experience, I have vowed to work with my guys to ensure that EVERYTHING, including all data we IT folks need to do our jobs (not just our “customers’” data), is backed up, and that those backups are being religiously monitored and tested. It’s funny how even a seasoned pro can let things slide occasionally. I’ll have to do another post about why I hate tape and why don’t use it for anything. Yes, you read that right, we have reliable backups, including off-site, cold storage, etc. Why everyone seems to be so stuck in the stone age is beyond me, but as I said, that’s a story for another day.

Palm III

After using a Franklin planner for years, I made the jump to the Palm as my first “real” PDA. At the time, I was quite impressed with what it allowed me to do. I missed my old Franklin a little, and I thought the handwriting recognition sucked, but battery life was great, and this was a solid machine. Unfortunately it also belonged to my employer, so I had to give it up when I left them (March, 2001).

Palm IIIc

I bought the color version of the device I had gotten used to. It was a little larger than my old III, and battery life wasn’t quite as good, but it was still acceptable (I could go a full week between charges if I didn’t use it a lot). This still stands out as one of the best, most solid devices I’ve ever used.

Sharp Zaurus

Being a Linux lover, I fell in love with the Zaurus before I ever got my hands on it. My wife got this for me as a Birthday present, selling my IIIc (with my approval) to buy it. I still have it kicking around, my kids play with it occasionally. The Zaurus was and is an incredible, full function Linux computer. Unfortunately, it was huge, and had terrible battery life. I would’ve loved a larger screen (greater resolution). I am still amazed at what I could do with it (full root access!). Even though it was uncomfortably large and heavy for a pocket, I could deal with it. But its poor battery life was the nail in the coffin. Under moderate to heavy usage, I couldn’t get through a full day without risking a dead battery. It was just impractical for that reason only. I miss it in a lot of ways, even the basic PDA functions were actually very nice, better, in my opinion, than almost anything else I’ve used.

Dell Axim X30

I got a good deal on the Axim, and went over to the dark side . To its credit, the Axim was a solid device, performed well, and had battery life only slightly worse than my IIIc. For a while there was a swell of support for Microsoft devices, and I was able to find a lot of useful apps. I missed the keyboard on my Zaurus, but the on-screen entry wasn’t too bad. I think I still have this one sitting on my workbench, I suppose I should go re-charge the battery.

Verizon XV6700

I bit the bullet and went for a smartphone. At the time, the Windows Mobile devices were fairly popular, and this phone got good reviews. It did everything my Axim did, same resolution, etc. I was able to get a Cisco VPN client, and was thrilled to be able to connect to work and SSH in to my servers. Battery life was acceptable, I could get a couple of days out of it without charging, and I enjoyed the Internet access, though the low screen resolution (320×240) made it difficult to use. The worst thing about the XV was shoddy hardware. I went through three devices, and the third one conked out within a month of switching to my G1. It just didn’t stand up to everyday use.

T-Mobile G1

All this brings me to where we are today, my G1! My first impressions are all positive! Battery life looks great, at least as good as the XV6700. The devices feels solidly constructed, but I’m worried about the keyboard and the whole slider mechanism. The interface looks good, better than the Windows Mobile for sure. It’s a little difficult to use with my big fingers, and I’m trying to get used to not having a stylus, but it’s not too bad. I haven’t checked out the apps too much yet, but I can’t find a Cisco VPN client, which is going to be a big bummer if one is not available. T-Mobile coverage isn’t as good as Verizon, but for where I need it most, it’s not too bad.

I am still trying to figure out the best way to access my email, since I do not use a GMail address, I’ve been using my own domain for years. I’ve been using Yahoo Pro as my email client for a while, but in order to get the most out of my phone, I guess I’m gonna have to make the switch to the Google world. I suppose if my data is out in the cloud, nothing’s really private, but Yahoo, especially with a paid account, at least gives the appearance of privacy. Anyway, once I get it all figured out, I’ll post another update.

Just a few things in case I haven’t mentioned them lately…

…thanks for taking the time out to teach me things. I am very interested to learn all this new stuff, and feel very fortunate to be able to learn from you. I like the way you explain things and you never make me feel like an idiot. Don’t let me talk over you or go off on one of my tangents when you’re telling me things…I don’t want to miss out on that ONE little vital piece of information.

…

Compared to the tone and content of most messages I receive, getting an email like this just makes my day and helps me deal with all the stress of the job.

As I said, when I like a product, I’ll say so. It’s kind of like dealing with employees–if they screw up, I tell them so, but if they do something right, I try to compliment them too. So I’ve told everyone I know how pleased I am with Blue Coat’s products, and Blue Coat seems to like how I say it. I consider it a great honor that they value my humble opinion enough to fly me out to New York to speak to journalists, investors, analysts, etc. I joked with their VP, Steve Schick, and asked him if he’d be upset if I said anything derogatory. That probably wasn’t nice, after all, the poor guy was working hard to pull of a successful event and probably didn’t need any extra stress (even if said in jest).

I was nervous to be seated on the stand in front of people whose names I recognize from various blogs and articles. The Nevada DOC seems like small potatoes next to some of Blue Coat’s other customers. But I realized I just needed to be myself and talk about what I know, so that’s what I did and made it through OK. As I responded to the questions and later talked one-on-one with several of these folks, I came to better understand why Blue Coat wanted me to attend. My organization may not be huge (25 locations, less than 3,000 employees), but we face many of the same challenges any large company would face. And we are on the cutting edge with regards to some of the technologies and methods we are using to manage our networks and applications.

I have often found it strange that so many people in 3rd world countries have cell phones. No one had ever been able to justify the infrastructure investment for “land-lines.” Cell phone infrastructure is far cheaper, and so these folks have actually benefited in some ways by being late-comers to the game. It’s the same thing with DOC’s network. Since the Department built its network recently, with appropriate funds to do so, we were able to take advantage of newer technology (Blue Coat, Infoblox, etc.). In any case, I have come to realize that while my network is not “huge,” it is not “tiny,” and we manage the infrastructure in a modern, forward-thinking way.

After talking to Blue Coat and others in attendance, I came to the conclusion that if I were looking for job, maybe my ideal title would be “Application Delivery Architect.” Such a position requires skill with servers, networks, applications (programming) and more. Basically it’s a job for someone who gets the “big picture,” and knows how to use the appropriate tools (Blue Coat, etc.) to make things run smoothly. It requires project management and IT management skills, but with possibly more of a technical focus than many IT Manager positions.

All in all, it was a great trip, but I’m happy to be back home. We got stuck in Manhattan for a third night due to a snowstorm. Really, 3 inches of snow is enough to shut down an airport? Things really are different East of the Mississippi…

One of the most important things I did was to send out a clear agenda to all involved parties more than a week ago. This doesn’t mean we followed it perfectly or in order. But it gave us the bullet points we needed to discuss, and the reasons why we were meeting. This allowed people to decide if they really needed to attend, and for managers to choose the right people to send. As a result, I had the all right people in the room, and no one who didn’t need to be there.

Another thing that is important is to control the “tangent factor” and keep the meeting on topic, or keep people from going on and on and on. This requires that whoever is “in charge” have a strong personality (not afraid to politely cut someone off or redirect them, if necessary) and be on top of the subject (be prepared) so they can keep things focussed on what needs to be discussed. Fortunately, this was not much of an issue this time. I was well prepared, and due to the agenda and other reasons I’ve already mentioned, everyone was ready.

I mentioned that we came up with a solution to the issue at hand. In a nutshell, our agency obtains certain electronic documents from another agency. However, it’s difficult to get at those documents, and they’re not always ready in a timely manner. Because we had all the right people in the room, we were able to figure out what it is that everyone needs, and then come up with a more efficient way to obtain the documents. If we had not come up with a solution, the meeting would’ve been a waste. Of course, that also means it is critical to document the solution so that it can be followed through to implementation.

Anyway, I’m no meeting expert, I just know I’ve sat through many torture sessions, so I do my best to make sure any meeting I’m in charge of is useful and productive. Of course, I do tend to ramble, so maybe they’re still boring…