http://gurkulindia.com/main/2012/07/deployment-and-understanding-of-ldoms-logical-domains/#more-6944
Deployment and understanding of LDOMS (Logical Domains):
Ldom
technology allows us to allocate a system’s various resources such as
memory, CPUs and devices (I/O,Networl interfaces etc) into logical
groupings and create multiple, discrete systems each having their own
operating system, resources, and identity within a single computer
system.
Ex:
In General Server Techonoly, an OS is directly interacting with
hardware through defined system procedure calls and uses complete
hardawre resources. But in LDOM the concept is bit different, a new
layer between Hardware and OS has been introduced which is called
“HYPERVISOR”. It interacts as a medium between Hardware & OS and
hence present the hardware components as defined by users to OS. In this
case the resource allocation/representation is be done by HYPERVISOR
and hence individual OS can have well defined hardware resources. Also
remember in LDOM the allocated devices/resources are called VDEV’s
(Virtual Devices).
Note: Ldom techology only supported on T-series servers, sparc-4v architecture with Solaris-10 only.
Role of Hypervisor:
===============
The
hypervisor, with its stable sun4v interface, is the centerpiece to
creating logical domains. Important points to remember are:
• The hypervisor is the layer between the operating system and hardware.
•
The hypervisor implements a stable sun4v interface. The operating
system makes calls to the hypervisor, and therefore, does not need to
know intimate details about the hardware, even if the platform changes.
•
The hypervisor is very thin; it exists only to support the operating
system for hardware-specific functions, making it small and simple,
which assists in
stability.
•
The hypervisor creates a virtual machine allowing the system to be
partitioned by exposing some of the resources to a specific partition
and hiding others.
•
The hypervisor creates communication channels, logical domain channels
(LDCs), between domains to provide a conduit for services such as
networks and shared devices.
Domain Types:
============
There
are several different roles for logical domains, and these are mainly
defined by context, their usage defines them. A domain may have one or
more of these roles, such as combining the functions of an I/O and
service domain:
• Control domain – Creates and manages other logical domains and services by communicating with the hypervisor.
• Service domain – Provides services, such as a virtual network switch or a virtual disk service, to other logical domains.
•
I/O domain – Has direct ownership of and direct access to physical
input/output devices, such as a PCI Express card or a network device.
Can optionally share those devices to other domains by providing
services.
•
Guest domain – Presents a virtual machine that subscribes to services
provided by service domains, and is managed by the control domain.
LDOM Daemons:
==============
There are two LDOM daemons ruuning on the system which are.
• LDMD (Logical Domain Daemon)
• VNTSD (Virtual Network Terminal Server Daemon)
Note:
Some common terms used in LDOM are Virtual Machine Description (ldm
list-spconfig), Virtual Devices (CPU’s, Memory, I/O Devices), Networking
(Virtual network (vnet) device, Virtual network switch (vsw)), Storage
(Virtual disk client (vdc) driver, Virtual disk server (vds) driver),
Console, Cryptographic Devices etc. I will suggest to refer man ldm for
all these terms to have much clear understanding.
In this section, I will expalin and create a Controler Domain (Host Domain or Primary Domain).
Pre-requisites:
============
1.) Solaris OS release level:
=====================
The Solaris OS release level should be
at-least Solaris 10 11/06 OS. If not then release upgrade should be
required to have LDOM software on your system.
# cat /etc/release
Solaris 10 11/06 s10s_u3wos_10 SPARC
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 14 November 2006
2.) Recommneded Patches:
======================
Its always recommneded to have latest
Bundle patch installed on your system. Otherwise make sure below patches
should be there in the system. If not present kindly install them
before proceeding further:
•118833-36, Kernel update patch
•124921-02, which contains updates to the Logical Domains 1.0 drivers and utilities
•125043-01, which contains updates to the qcn (console) drivers.
Note: Kindly go through the readme file for these patches as they required reconfig reboots.
3.) Firmware Versions:
==================
Its always recommended to have latest
firmware level installed on your system. As this ensures the hardware
and hypervisor can communicate correctly, and all of the features of the
your servers can operate.
We can check our current firmware level at ALOM/OS level as follow:
sc> showhost
Sun-Fire-T2000 System Firmware 6.7.12 2011/07/06 20:03
Host flash versions:
OBP 4.30.4.d 2011/07/06 14:29
Hypervisor 1.7.3.c 2010/07/09 15:14
POST 4.30.4.b 2010/07/09 14:24
# prtconf -V
OBP 4.30.4.d 2011/07/06 14:29
Note: I have carried out my testing on T-2000 server and patched the box to latest firmware level.
4.) Installation of Logical Domains Manager:
===================================
The LDOM package is not a default part
of Solaris OS. We have to download it from Oracle site and have to apply
that on our system as superuser.
SUNWldm.v – Logical Domains Manager package
SUNWjass – Solaris Security Toolkit packages
I have installed LDOM 1.2 version on my machine as shown below:
# pkginfo -l SUNWldm
PKGINST: SUNWldm
NAME: Logical Domains Manager
CATEGORY: application
ARCH: sparc.sun4v
VERSION: 1.2,REV=2009.06.25.09.48
BASEDIR: /
VENDOR: Sun Microsystems, Inc.
DESC: LDoms Manager – UltraSPARC CMT virtualization
PSTAMP: svlpen-on10-020090625094807
INSTDATE: Jun 21 2012 11:44
HOTLINE: Please contact your local service provider
STATUS: completely installed
FILES: 66 installed pathnames
11 shared pathnames
20 directories
18 executables
6564 blocks used (approx)
# ldm -V
Logical Domain Manager (v 1.2)
Hypervisor control protocol v 1.3
Using Hypervisor MD v 1.1
System PROM:
Hypervisor v. 1.7.3. @(#)Hypervisor 1.7.3.c 2010/07/09 15:1415
OpenBoot v. 4.30.4. @(#)OBP 4.30.4.d 2011/07/06 14:29
Note: This package is installed as a script and not as a package.
Creation of Controller Domain:
=========================
1.) Test the ldom software is working and is communicating with the HYPERVISOR:
# ldm list
——————————————————————————
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
——————————————————————————
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -t-c– SP 32 32640M 0.1% 5d 4h 54m
Note: a.) Path for ldm comamnd is /opt/SUNWldm/bin/
b.) The “t” in FLAGS shows the domain is not yet up and “n” instead of “t” indicates that the server is up.
2.) Creation of default services, which
should always be there in Controller Domains. The services are Disk
Services, Console Services and Network services. Below are the steps to
create them:
# /opt/SUNWldm/bin/ldm add-vds primary-vds0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm add-vcc port-range=5000-5100 primary-vcc0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
3.) List the services which we have just created:
# /opt/SUNWldm/bin/ldm list-services primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
—————————————————————————-
…
Vds: primary-vds0
Vcc: primary-vcc0
port-range=5000-5100
Vsw: primary-vsw0
mac-addr=0:14:4f:f9:68:d0
net-dev=e1000g0
mode=prog,promisc
4.) Now our next step would be to
provide the Resources to controller Domain. We will specify an amount of
CPU and memory that should be considered a good starting point. Our
Primary Domain resources would be as:
• 1 x MAU (cryptographic) unit – these are bound on a per-core basis, and need
to be set up prior to assigning VCPUs
• 4 x virtual CPUs (1 core on an Ultra SPARC T1 system)
• 1024 Mbytes memory (as we will not be using ZFS to deliver disk services, we
do not need the minimum of 4Gbytes memory.)
• Configuration will be saved as “initial”
# /opt/SUNWldm/bin/ldm set-mau 1 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm set-vcpu 4 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm set-memory 1024m primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
5.) Creation and use of newly created configuration:
Now that the control domain is
configured the way we want it, we need to store it. The hypervisor will
then use this configuration after the next power cycle. First, we will
list available configurations and then create a new one called initial,
and finally confirm its creation:
# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]
# /opt/SUNWldm/bin/ldm add-spconfig initial
# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]
initial [next]
Note: We can see the new configuration
initial has been created and listed as the configuration to use at the
next reboot. If a configuration with the name initial already existed,
we would receive an error, and would need to use the ldm remove-spconfig
command to remove the existing configuration first.
6.) Now we need to reboot our box for
the previous changes which we have made to take effects releasing
resources for other logical domains.
# shutdown -i6 -g0 -y
Note: Do not use reboot comamnd. Always use init or shutdown command so that no rc phase gets bypassed.
7.) Enable Daemons, This is the point where we have to enable our vntsd.
# svcadm enable vntsd
# svcs -a | grep -i ldom
# svcs -a | grep -i ldom
disabled Jun_27 svc:/ldoms/agents:default
online Jun_27 svc:/ldoms/ldmd:default
online Jun_27 svc:/ldoms/vntsd:default
8.) Check the services running on the controller domain:
# /opt/SUNWldm/bin/ldm list-bindings primary———————————————————————
Name: primary
State: active
Flags: transition,control,vio service
OS:
Util: 12%
Uptime: 11m
Vcpu: 4
vid pid util strand
0 0 18% 100%
1 1 13% 100%
2 2 9.8% 100%
3 3 5.4% 100%
Mau: 1
Memory: 124m
real-addr phys-addr size
0×4000000 0×4000000 124m
Vars: reboot-command=boot
IO:
pci@780 (bus_a)
pci@7c0 (bus_b)
………………….
# ldm list
——————————————————————————
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
——————————————————————————
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-c– SP 4 1024M 0.1% 5d 1h 10m
Our first controller Domain is ready for
use. I will explain and create our Guest Domains in my next post and
will cover the complete LDOM stuff in that. Moreover I will try to cover
the release upgrade and Ldom patchings for Guest and Controller domains
with backout plans.
Reference: Oracle Documentation & support.oracle.com
and the landmark is below. There is a parameter on LOM/ALOM (for Sparc
arch.) and ILOM (for intel arch.) which you wont find in help. The
default value for that parameter is FALSE which wont allow your OS
instance to boot up and hence it just stuck and wait for manual
intervention at LOM level only.
