Eflags Registers

This article talks about intel IA32 eflags register & some interesting things that i found out while studying more about these flags.  This article would be using gnu debugger(gdb) to show the status of eflags register.

First the theory about eflags registers:

eflags register in an IA32 processor stores various flags corresponding to the result of last instruction executed.

Not all instructions use eflags register like mov, bswap, xchg, but instructions like “inc” (increment), add (addition), mul, div instructions use eflags register.

First before we go further in to eflags, there are few points to remember.

• We cannot examine the whole eflags register
• There is no instruction that can be used to modify this register directly.
• There are some instructions that can be used to modify certain bits of the register,  but they are beyond the scope of this article.

We will be looking at some of the flags of the register using simple examples:

1. Carry Flag

• Keeps the status of the final carry-out while computing the result of the last instruction set.
• While adding 2-numbers the carry flag contains the carry-out of the most significant bit.
• Example :
• Adding 253 & 4,  For this example, we will use “al” register , which is lower 8-bits of EAX register
• 

  General Purpose Registers

  I choose this example specifically to view the Carry flag.   Since our number is less than 255 we will use lower 8 bits of eax register which is  al and will be adding 4 to 253 . Below is the sample code

Assembly Language Program in AT&T style

We assemble the above code using Gnu Assembler and loader.

add1.s

we will use Gnu Debugger(gdb) to view the contents of the registers.

Gnu Debugger

We will set the break point to line 4 and run the program , Type “n” to execute the line 4

Set break point and run the program

Type “info registers”  at the gdb prompt to view the current value in registers

info registers

as we can see from the above figure, Register al is actually storing -3, instead of  253, this is because the range of numbers that can be stored in al is not from 0 to 255  but instead -128 to 127.

Type “n” or next to execute the line 5 of the program which adds 4 to register al.

CF is seen in gdb

when we do addition of 4 to -3 , the result is +1 , so the final value of register al is 0x01 which sets the Carry Flag (CF).  We can see from the above figure that eflags shows CF to be set as expected.

To check eflags register only we could type “info reg eflags” on gdb prompt.

2. Zero Flag

• Zero flag is set to 1 if the result of the last flag-modifying instruction is 0

Examples:

adding negative & positive number

In the above code we set 0xfd, which is -3 value set in register al, and then we add +3 to it. So when processor executes line 5 , the resultant value is 0.  So processor sets ZF in the eflags register. We can view this when we run the above program through gdb.

I will cover the rest of the eflags in next article.

New fuzzer in Fedora

Tags

fedora, fuzzing, radamsa

Do you fuzz ? If you do,  Fedora now has a fuzzer called radamsa. More information about radamsa can be found here.  Radamsa is now available in F20, F21 and F22.

Happy fuzzing!

Using gdb layout when debugging Assembly Language Programs

Tags

asm, assembly, code asm, gaming, gdb, gnu assembler, programming, software

In my quest to learn programming, I have started my initial steps with assembly Language programming (ALP). I have been on this endeavour from quite some time.

This post is not about ALP though but about an important option of gdb called layout. layout helps all the newbies who are learning ALP a lot. Before i explain about this option , Consider the below program:

Figure-1

The above program calculates sum of two values (17 and 15) using sum function in alp. The output is saved in ebx register. Let us first assemble and link the program. We are assembling our source code using GNU Assembler with option -gstabs to debug the assembly code through gdb.

From man as:
–gstabs:Generate stabs debugging information for each assembler line. This may help debugging assembler code, if the debugger can handle it.

Figure-2

From the Figure-2 we could see that program “mysum” runs successfully. Let us Run the program through gdb. What we want to know is the following:

i) Values of registers while the code is being executed at each step
ii) Most importantly we want to see the code and registers at the same time.

To accomplish the above goals gdb provides a Text User interface using curses library to show source file. This feature is not limited to source file only but also shows assembly code(asm), registers(regs). In our case we would require to view assembly code and registers and that too at the same time. To view the registers as our asm code executes.

Figure-3

We first start with invoking the program mysum with gdb and setting the break point at the start function:

Figure-4

After setting the break point run the program by typing “run” (or just “r”) at the gdb prompt which will stop at the first break point , which in our case is the _start function. At this point lets invoke “asm” layout by typing “layout asm” at the gdb prompt:

asm layout would look like this:

Figure-5

In Figure-5, we could see our asm code with more details, like address where the particular instruction resides, and the instruction. Our current program counter starts at 0x8048054 which is the start of our program. From here we will keep stepping through our code and view the register values.

To load the register layout type “layout regs” at the gdb prompt and gdb would automatically split the TUI to show both asm code and registers as shown below:

Figure-6

In Figure-6, Instruction to be executed is highlighted and also we could see the values of registers. We will step through the code by typing “step” (or in short “s”) command at gdb prompt which will execute the earlier instruction and the code to be executed next is highlighted:

Figure-7

In Figure-7,  we could see now our Program counter points to next instruction at address 0x8048056 which is to push 0xf (15 to stack). And also our register layout shows EIP is pointing to code to be executed and current ESP register value.

As we keep stepping through our code (use “s” at gdb prompt) and when our code enters sum function we should be able to see our base pointer  register value  (EBP) and if it’s saved with value of ESP register:

Figure-8

Our code has passed values 17 and 15 to the stack and in sum function we are copying these values to General puspose registers ecx and ebx. Figure-9 shows that ecx and edx registers have been loaded with 17 and 15 as mentioned in the code.

Figure-9

Keep stepping through the code and once code reaches to the end of sum function where we exit the function by popping the stack we could see the Register values at register layout Restore the stack pointer and returning back to _start):

Figure-10

Once we are back to _start function we see that the sum of 17 and 15 is stored in ebx register, load “1” to eax register and send the interrupt to call the exit system call. The output of the program i.e sum of values 17 and 15 can be viewed by check the status of exit system call which is value in ebx register.

Figure-11

I hope the above information would be useful for newbies while debugging assembly language code.

Note: “layout regs” doesn’t yet work on gdb version “gdb-7.2-51.el6.i686” on RHEL6 . It crashes gdb. Fedora 15 and latest rawhide has the fix . Hope later versions of the gdb on RHEL6 might have the fix.

Authenticating using polkit to access libvirt in Fedora 18

Tags

fedora, fedora-18, libvirt, linux, polkit, SELinux

From Fedora-18 there has been some noticeable changes to polkit. Policy kit helps access to certain privileged process to unprivileged applications or users in this case. I generally use systems with SELinux Enabled and also confine my users. Since most of my job requires testing various applications , I keep creating a lot of vm’s (RHEL5,RHEL6). For this virt-manager is my preferred application.

Recently i have been assigned with a new Intel Hardware which has hardware Virtualization enabled with 1TB Hard disk. So installed Fedora-18 to create VM’s. My requirement is i should be able to install vm’s using Non-root user and that too with user who’s confined.

• Create a user

      $ useradd test  

• Map this user to staff_u selinux user

      $ semanage login -a -s staff_u test
  
      Login Name           SELinux User         MLS/MCS Range        Service
  
      __default__          user_u               s0                   *
      ceres                sysadm_u             s0-s0:c0.c1023       *
      juno                 staff_u              s0                   *
      root                 root                 s0-s0:c0.c1023       *
      system_u             system_u             s0-s0:c0.c1023       *
      test                 staff_u              s0-s0:c0.c1023       *
  

• login as test user and connect to libvirt socket using virsh

      [mniranja@mniranja mar20]$ ssh test@10.65.201.167
      test@10.65.201.167's password: 
      Last login: Wed Mar 20 00:20:13 2013 from localhost
      [test@dhcp201-167 ~]$ id -Z
      staff_u:staff_r:staff_t:s0-s0:c0.c1023
  

• Connect to libvirt socket

      [test@dhcp201-167 ~]$ virsh -c qemu:///system
      error: authentication failed: Authorization requires authentication but no agent is available.
  
      error: failed to connect to the hypervisor
  

As you can see above it doesn’t allow to connect , In earlier versions of Fedora, you could use policy kit to create a authorization rule to connect to libvirt socket. Refer Libvirt documentation. This method is also called Policy Kit LocalAuthority. So on Fedora-16 system i had the following rule

        [root@reserved 50-local.d]# cat 50-org.example-libvirt-remote-access.pkla 
        [Remote libvirt SSH access]
        Identity=unix-group:virt
        Action=org.libvirt.unix.manage;org.libvirt.unix.monitor
        ResultAny=yes
        ResultInactive=yes
        ResultActive=yes

The above would allow users of group “virt” to access libvirt and manage libvirt through policy kit action “org.libvirt.unix.manage” . The above rules are placed in file 50-org.example-libvirt-remote-access.pkla under directory “/etc/polkit-1/localauthority/50-local.d”.
I hoped the same would work on Fedora-18 but it doesn’t as Policy kit localAuthority has been removed totally, instead all the custom policy kit rules should be placed under /etc/polkit-1/rules.d/ directory. Syntax of writing rules has been changed and Java Script syntax need to be used. Refer DavidZ blog for more information regarding the change.

On Fedora-18 i managed to do the same by adding the following rule file 10.virt.rules created under /etc/polkit-1/rules.d directory

        [root@dhcp201-167 rules.d]# cat 10.virt.rules 
        polkit.addRule(function(action, subject) {
        polkit.log("action=" + action);
        polkit.log("subject=" + subject);
        var now = new Date();
        polkit.log("now=" + now)
        if ((action.id == "org.libvirt.unix.manage" || action.id == "org.libvirt.unix.monitor") && subject.isInGroup("virt")) {
        return polkit.Result.YES;
        }
        return null;
        });

Thanks To Gilbert , As you can see the above allows polkit action “libvirt.unix.manage” || “org.libvirt.unix.monitor” to all the users of group “virt”

• Restart polkit service

      $ systemctl restart polkit.service
  

• Add the user test to group virt

      $ usermod -aG virt test
  

• login as test user and connect to libvirt using virsh

      [test@dhcp201-167 ~]$ id -Z
      staff_u:staff_r:staff_t:s0-s0:c0.c1023
  
      [test@dhcp201-167 ~]$ id
      uid=1002(test) gid=1003(test) groups=1003(test),1001(virt) context=staff_u:staff_r:staff_t:s0-s0:c0.c1023
  
      [test@dhcp201-167 ~]$ virsh -c qemu:///system
      Welcome to virsh, the virtualization interactive terminal.
  
      Type:  'help' for help with commands
         'quit' to quit
  

• Check the logs using journalctl

      [root@dhcp201-167 ~]# journalctl -xn
      -- Logs begin at Tue 2013-03-19 22:54:05 EDT, end at Wed 2013-03-20 00:43:25 EDT. --
      Mar 20 00:43:02 dhcp201-167.englab.pnq.redhat.com kernel: usb 1-1.3: Product: USB Optical Mouse
      Mar 20 00:43:02 dhcp201-167.englab.pnq.redhat.com kernel: usb 1-1.3: Manufacturer: PixArt
      Mar 20 00:43:02 dhcp201-167.englab.pnq.redhat.com kernel: input: PixArt USB Optical Mouse as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3:1.0/input/in
      Mar 20 00:43:02 dhcp201-167.englab.pnq.redhat.com kernel: hid-generic 0003:0461:4E22.006D: input,hidraw0: USB HID v1.11 Mouse [PixArt USB Optical Mouse] on usb
      Mar 20 00:43:18 dhcp201-167.englab.pnq.redhat.com sshd[3722]: Accepted password for test from 10.3.235.177 port 53789 ssh2
      Mar 20 00:43:18 dhcp201-167.englab.pnq.redhat.com systemd-logind[596]: New session 18 of user test.
      -- Subject: A new session 18 has been created for user test
      -- Defined-By: systemd
      -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
      -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
      -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/8d45620c1a4348dbb17410da57c60c66
      -- 
      -- A new session with the ID 18 has been created for the user test.
      -- 
      -- The leading process of the session is 3722.
      Mar 20 00:43:18 dhcp201-167.englab.pnq.redhat.com sshd[3722]: pam_unix(sshd:session): session opened for user test by (uid=0)
      Mar 20 00:43:25 dhcp201-167.englab.pnq.redhat.com polkitd[1688]: /etc/polkit-1/rules.d/10.virt.rules:2: action=[Action id='org.libvirt.unix.manage']
      Mar 20 00:43:25 dhcp201-167.englab.pnq.redhat.com polkitd[1688]: /etc/polkit-1/rules.d/10.virt.rules:3: subject=[Subject pid=3791 user='test' groups=test,virt,
      Mar 20 00:43:25 dhcp201-167.englab.pnq.redhat.com polkitd[1688]: /etc/polkit-1/rules.d/10.virt.rules:5: now=Wed Mar 20 2013 00:43:25 GMT-0400 (EDT)
  

Introduction to CTDB Cluster

Tags

cluster, ctdb, samba

Why CTDB ?

• Traditionally Clustering involves a SAN connected to n nodes. The storage can be  accessed only by the nodes participating in the cluster and as the the need for more storage and users grow , space tends to be small and clustering becomes small

• So we need a file system that can be accessed by arbitrary number of clients and not restricted to the systems participating in the cluster.  One of the answers to this problem is Distributed File system.

• We need to distribute the existing shared storage using network protocols like NFS and cifs. With samba and CTDB we can achieve this goal of distributing the shared File  system using CIFS Protocol

• CTDB is originally developed specifically as cluster Enhancement software and contains high availability, load balancing features which makes file services like samba, NFS and FTP cluster-able.

Basic Infrastructure of CTDB

• Storage is attached to nodes participating in the cluster through FC or iscsi
• Shared File system which supports POSIX-fcntl locks
• IBM General Parallel File system (GPFS)
• Global File system (GFS)
• GNU Cluster File system (Gluster)
• Sun’s Lustre
• OCFS2

Basics of CIFS File system

• CIFS (Common Internet File system is a standard remote file system access protocol for use over network, enabling groups of users to connect and share documents
• CIFS is open, cross-platform based on SMB (Server Message Block) Protocol, which is native file-sharing Protocol in the Windows Operating system. On RHEL this is implemented using samba
• CIFS runs over TCP/IP

Basics of Samba

• Samba provides File and Print services for all the clients using SMB/CIFS protocols
• Apart from File and Print services, it also does Authentication and authorization, Name
• Resolution and Service Announcement
• File and Print services are provided by the smbd daemon
• Name resolution and browsing is provided by nmbd daemon
• Configuration file is /etc/samba/smb.conf

TDB (Trivial Database)

• Samba keeps track of all the information needed to serve clients in a series of *.tdb files
• located in /var/lib/samba or /var/cache/samba
• Some of the TDB files are persistent
• TDB files are very small like Berkely database files
• Allow multiple simultaneous writes

Example TDB Files:

• account_policy.tdb    NT account policy settings such as pw expiration
• brlock.tdb  Byte-range locks
• connections.tdb Share connections (used to enforce max connections, etc…)
• Messages.tdb Samba messaging system

What Does CTDB do ?

• CTDB (Clustered Trivial Data Base) is a very thin and fast database that is developed for samba to make clusterize samba.

• What CTDB does is to make it possible for Samba to run and serve the same data from several different hosts in network at the same time

• which means samba becomes clustered service and and are active and exports the samba shares, read-write operations at the same time making it high-available.

• To do above we require a method of communication (IPC) for samba daemons running between nodes and share some persistent data (TDB files). Some of the information that should be shared are:

• User information
• For samba acting as a member server of a Domain, The Domain SID should be shared
• The user mapping tables Mapping of Unix UID’s and GID’s to Windows Users and Groups
• The active SMB-sessions and connections are shared
• locking information like byte-range locks granted exclusively to users to access a particular file have to be shared between all the nodes These locks are Windows Locks i.e when Multiple windows/samba clients access files these locks are given by smbd daemon so it makes sense to share these locks between smb daemons on different nodes

Sample Diagram on CTDB Messages are shared between 2 CTDB Clusters:

Below are the list of TDB files that are to be shared between CTDB Clusters:

• SMB Sessions (sessionid.tdb)
• share connections (connections.tdb)
• share modes (locking.tdb)
• byte range locks (brlock.tdb)
• user database (passdb.tdb)
• domain Join Information (secrets.tdb)
• id mapping tables (winbind_idmap.tdb)
• registry (registry.tdb)

Requirements to configure CTDB cluster on RHEL6

• GFS Packages
• HA Packages
• ctdb, samba
• ctdb-tools

Configuring samba to use CTDB

• We require 2 separate networks, one internal network through which CTDB daemons communicate and one public network through which it offers cluster services like samba, NFS etc.

• Install samba and CTDB Packages

$ yum install samba ctdb tdb-tools

• Configure /etc/samba/smb.conf to make samba cluster aware, Add the below lines in “global” section of smb.conf

clustering = yes
idmap backend = tdb2

• CTDB Cluster configuration

/etc/sysconfig/ctdb: is the primary configuration file and it contains startup parameters for ctdb. The important parameters are:

CTDB_NODES

CTDB_NODES=/etc/ctdb/nodes
This parameter specifies the file that needs to be created and should contain list of
Private IP address that CTDB daemons will use in the cluster. It should be a private
non-routable subnet which is only used for cluster traffic. This file must be same on all
nodes in the cluster.
contents of /etc/ctdb/nodes :
192.168.122.7
192.168.122.8
192.168.122.9
192.168.122.10

CTDB_RECOVERY_LOCK

This parameter specifies the lock file that the CTDB daemons use to arbitrate which
node is acting as a recovery master. This file must be held on shared storage so that all
CTDB daemons in the cluster will access/lock the same file.

CTDB_RECOVERY_LOCK=”/ctdb/cifs/lockfile”

CTDB_PUBLIC_ADDRESS
This parameter specifies the name of the file which contains the list of public addresses that particular node can host. While running, the CTDB cluster will assign each public address that exists in the entire cluster to one node that will host that public address. These are the addresses that the SMBD daemons and other services will bind to and which clients will use to connect to the cluster

Example 3 node cluster:
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses

Content of /etc/ctdb/public_addresses:

10.65.208.142/22 eth0
10.65.208.143/22 eth0
10.65.208.144/22 eth0

Configure it as one DNS A record (==name) with multiple IP addresses and let round- robin DNS distribute the clients across the nodes of the cluster

The CTDB cluster utilizes IP takeover techniques to ensure that as long as at least one node in the cluster is available, all the public IP addresses will always be available to clients.

/etc/ctdb/events.d This is a collection of scripts that is called to by CTDB when certain events occur to allow for site specific tasks to be performed

• Start CTDB daemon and let ctdb call the smbd daemon, samba daemon should not be started by init process

#chkconfig ctdb on
#chkconfig smb off
#chkconfig nmb off

• Start the ctdb daemon

# service ctdb start

Example Diagram of a 3 Node CTDB Cluster:

How does CTDB work

• On each node CTDB daemon “ctdbd” is running, samba instead of writing    directly to TDB databases it talks via local “ctdbd”

• “ctdbd” negotiates the metadata for the TDB’s over the n/w

• For actual read and write operations local copies are maintained on fast local storage

• We have 2 kinds of TDB files Persistent &  Normal

• Persistent TDB files should always be up-to-date and each node always has a updated copy. These TDB files are kept locally (LTDB) on the local storage and not on the shared storage. So the read and write operations are faster

• The node when wants to write to Persistent TDB, it locks the whole database , perform read and write operations and the transaction commit operations is finally distributed to all nodes and also written locally

• Normal TDB files are maintained temporarily . The idea is that each node doesn’t have to know all the records of a database. It’s sufficient to know the records which affect it’s own client  connections, so when the node goes down it is acceptable to lose those records

• Each node carry certain roles
• DMASTER (data master)
• Current, authoritative copy of a record
• Moves around as nodes write to a record
• LMASTER (Location Master)
• knows the location of the DMASTER
• Knows where the record is stored
• Only one node has the current authoritative copy of a record, i.e data master
• Step-1: Get a lock on a record in TDB
• Step-2: Check if we are on Data master
• if we are DMASTER for this record
• then operate on the record and unlock it when finished
• Step-3: if we are not DMASTER for this record unlock the record
• Step-4: Send a request to the local CTDB daemon to request the record to be migrated on to this node
• Step-5 once we get a reply from local “ctdb” daemon that the record is now   locally available, go to step-1

Failover

• CTDB assigns IP address from the pool (CTDB_PUBLIC_ADDRESS) to the healthy node
• When the node goes down IP is moved to other node
• Client reconnects to the new node using tickle ACKs if the below conditions are met :

• • Node goes down

  • Client doesn’t know yet that ip has moved

  • New node sends TCP ACK with seq 0 to the client

  • client sends correct ACK to the client

  • New node resets the connection using RST

  • client re-establishes connection to new node

• recovery master – performs recovery , collects most recent copy from all nodes and recovery master becomes data master.
• Recovery master is determined by the election process, the RECOVERY_LOCK file acts as arbitrator and nodes compete to get a lock (POSIX fcntl byte-range) on that file.
• If the Recovery master node is gone, we need to assign the role to a new node.

Commands to manage CTDB

$ctdb status: status command provides basic information about the cluster and the status of the nodes.

$ctdb ping: This command tries to ping each of the CTDB daemons in the cluster

$ctdb ip: This command prints the current status of the public ip addresses and which physical node is currently serving that ip

$onnode: onnode is used to run commands on ctdb nodes

examples:
$onnode all pidof ctdbd
$onnode all netstat -tn | grep 4379

CTDB Status Messages:

“ctdb status” specifies the node status , There are 5 possible states:

• ok This node is fully functional
• DISCONNECTED This node could not be connected through the network and is currently not participating in the cluster.
• UNHEALTHY ctdbd daemon is running but the service provided by ctdbd has failed
• BANNED Too many failed too many recovery attempts and is banned from participating in cluster for a period of “RecoveryBanPeriod” seconds
• STOPPED A node that is stopped does not host any Public IP address and is not part of the cluster

Troubleshooting:

• ctdb log file /var/log/log.ctdb

• Output of “ctdb status” and “onnode” commands output would be helpful

• /var/log/samba contains logs related to smbd daemon

• If needed tcpdump on port 4379 can be taken, wireshark is capable of identifying CTDB protocol and can display various CTDB status

• testparm output to check if clustering is enabled ?

Documentation

Man Pages:

man ctdb
man ctdbd
man onnode

Links:

• http://ctdb.samba.org/configuring.html

• http://wiki.samba.org/index.php/CTDB_Setup

• http://samba.org/~obnox/presentations/sambaXP-2009/sambaxp-2009-tutorial-samba-ctdb-slides-paper.pdf

• http://samba.org/~obnox/presentations/sambaXP-2009/samba-and-ctdb.pdf

FAQ ?

Q) When CTDB itself a cluster why do we require HA Packages to be installed like cman ?

CTDB will not work without Red Hat Cluster Suite. As CTDB requires gfs and gfs in-turn
requires cman to start dlm_controld and gfs_controld. So CMAN is pre requisite for CTDB

Q)How does CTDB solve the split-brain problem

Well this problem doesn’t arise in the first place as CTDB is all-active node and not active/passive setup where passive nodes suddenly become active.

Q)How to identify which node is actually serving or which node is Data master

The ip to which client is connected is the data master, i.e from the pool of public addresses , to which ever ip the client connects and the node to which the ip is assigned becomes Data  master (DMASTER)

Q) How to identify which node is recovery master (RMASTER)

The node which holds the lock file . The lock file is the file which is saved in the shared file
system. (CTDB_RECOVERY_LOCK)