Playing with Oracle ASM, multipath disks, TrueNas and VirtualBox — Part 2

4 min readApr 30, 2022

Welcome back to the second part. In this part, we are going to configure the iSCSI, the multipath and the ASM disks.

Oracle Environment

For the sake of simplicity, I will use a VM previously configured with Oracle Restart, and one database already used in other tests. I only added 2 new network interfaces to the VM in order to use the multipath.

I’ve added the truenas’ VM ips to the /etc/hosts

root@has03:/root $> cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#
10.1.1.1  truenas1
10.1.2.1  truenas2

Configuring the new network interfaces

You need to configure the new internal networks with IPs in the same network as the truenas portal (10.1.1.1/10.1.2.1).

Configuring the iSCSI

--discover the iscsi target
root #> iscsiadm -m discovery -t sendtargets -p truenas1
10.1.2.1:3260,1 iqn.2005-10.org.freenas.ctl:has03
10.1.1.1:3260,1 iqn.2005-10.org.freenas.ctl:has03--enable automatic login during startup
root #> iscsiadm -m node --op update -n node.startup -v automatic--login to both network interfaces
root #> iscsiadm -m node -p truenas1 --login
root #> iscsiadm -m node -p truenas2 --login

Configuring the multipath

--enable and start the service
root@has03:/ $> systemctl enable multipathd
root@has03:/ $> systemctl start multipathd--create the multipath.conf
cat <<EOF>/etc/multipath.conf
defaults {
    find_multipaths yes
    user_friendly_names yes
    failback immediate
}blacklist {
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z][[0-9]*]"
     devnode "^sda[[0-9]*]"
     devnode "^cciss!c[0-9]d[0-9]*"
}
EOFroot@has03:/ $> ll /etc/multipath.conf
-rw-r--r--. 1 root root 849 Apr 30 10:24 /etc/multipath.conf--reload the service
root@has03:/ $> systemctl reload multipathd--check if the disks exist
root@has03:/ $> ll /dev/mapper
total 0
crw-------. 1 root root 10, 236 Apr 30 09:34 control
lrwxrwxrwx. 1 root root       7 Apr 30 10:24 mpathd -> ../dm-4
lrwxrwxrwx. 1 root root       7 Apr 30 10:24 mpathe -> ../dm-2
lrwxrwxrwx. 1 root root       7 Apr 30 10:24 mpathf -> ../dm-3
lrwxrwxrwx. 1 root root       7 Apr 30 09:34 ol-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Apr 30 09:34 ol-swap -> ../dm-1

Using the multipath command, you can check if the disks are using both networks.

root@has03:/ $> multipath -ll
mpathe (36589cfc000000580ddee277e9cda411e) dm-2 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:2 sdf 8:80  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:2 sdi 8:128 active ready running
mpathd (36589cfc00000025b4072a1536ade2f9d) dm-4 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:1 sde 8:64  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:1 sdh 8:112 active ready running
mpathf (36589cfc000000f5feabc996aa1a12876) dm-3 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:3 sdg 8:96  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:3 sdj 8:144 active ready running
root@has03:/ $>

Let’s make our lives a little easier adding aliases to the disks. I’ve updated the multipath.conf file and reloaded the service.

root@has03:/ $> cat /etc/multipath.conf
defaults {
    find_multipaths yes
    user_friendly_names yes
    failback immediate
}blacklist {
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z][[0-9]*]"
     devnode "^sda[[0-9]*]"
     devnode "^cciss!c[0-9]d[0-9]*"
}multipaths {
    multipath {
        wwid    36589cfc000000580ddee277e9cda411e
        alias   mpdata1
    }
    multipath {
        wwid    36589cfc00000025b4072a1536ade2f9d
        alias   mpdata2
    }
    multipath {
        wwid    36589cfc000000f5feabc996aa1a12876
        alias   mpfra1
    }
}root@has03:/ $> systemctl reload multipathd

Now we have better names (aliases).

root@has03:/ $> multipath -ll
mpfra1 (36589cfc000000f5feabc996aa1a12876) dm-3 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:3 sdg 8:96  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:3 sdj 8:144 active ready running
mpdata2 (36589cfc00000025b4072a1536ade2f9d) dm-4 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:1 sde 8:64  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:1 sdh 8:112 active ready running
mpdata1 (36589cfc000000580ddee277e9cda411e) dm-2 TrueNAS,iSCSI Disk
size=5.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 7:0:0:2 sdf 8:80  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:2 sdi 8:128 active ready running

Nice, it worked. Now it’s time to use those disks in our ASM. The VM has03 already has an Oracle Restart installation, so I already have 3 ASM local disks.

Let’s add our multipath disks to the UDEV.

Now let’s swap our local disks with the multipath disks on the ASM diskgroups.

Now we wait until the end of the rebalance, and validate the new disks.

Voilà, everything working like a charm!

That’s the end of the second part. In the next and last part, we are going to run some tests on our new multipath disks. À la prochaine.

PART1, PART2, PART3

Playing with Oracle ASM, multipath disks, TrueNas and VirtualBox — Part 2

Oracle Environment

Written by Rodrigo Lima