Archive for the ‘volume-managers’ Category

Restoring a wordpress site by scraping/crawling google

Sunday, January 13th, 2013

I love challenges, but once in a while the tend to be way tooo big! During my christmasholidays I accidently wiped my home server. I wanted to do some LVM stuff online, remotely, without console access, through the ESXi console. … and really thought that nothing could go wrong ;-). First assumption being wrong.

To make a long history short. I shot myself in the foot and was without a server for a 3-4 days. When I got home again, I thought a simple reboot and some LVM magic would make everything all right. Second assumption being wrong.

So in the very end I had to reinstall my server from scratch. Luckily I backup my stuff using and so should you! It will save your butt some time!

It turned out that, for some bizaro reason, my database had not been dumped to csv files. So in the end I came to these conclusions:

  • I lost my database.
  • I thus lost my wordpress blog.

🙁

But loving challenges I refused to let that be the end. I thought about using archive.org, but they did not really have a new crawl of my site.

I decided to crawl google! Not as easy as it might sound for a couple of reasons:

  • Google does not like being crawled …. at all. If googles infinite number of computers discover that you are crawling them, then your IP will be blocked from seeing their cached content.
  • When you enter keywords into google you normally get thousans of links to follow. I needed one. The correct one! The one that was a cached version of my site.

So I fired up my editor and utilized the great WWW::Mechanize. I ended up with this script, which do all the hard work of scraping google. It will take some time to complete — hours and days even! It will get there though. If you try to speed things up it will take longer as you will be blocked by google when they detect you are scraping them. Be warned. Been there. Tried that. Got blocked.

Having retrieved all of my old site through google I had to parse these pages and import them into wordpress. So again I fired up my editor and wrote this little script. For this to work, you have to have

    • a clean wordpress installation with a hello world post
    • XMLRPC writing enabled in WordPress, as the script uses WordPress::XMLRPC.
    • the following in wp-config.php
      [code]
      define( ‘AUTOSAVE_INTERVAL’,    3600 );     // autosave 1x per hour
      define( ‘WP_POST_REVISIONS’,    false );    // no revisions
      define( ‘DISABLE_WP_CRON’,      true );
      define( ‘EMPTY_TRASH_DAYS’,     7 );        // one week
      [/code]

So in the end what did I loose and what did I learn? I lost my comments on my site. Or more preciesely: I have them,but I will postpone putting them back in until I get the time to fool around with coding again. And I learnt a lot about tripplechecking my backup for all their pieces before doing storage related work remotely without a proper console!

About growing logical volumes, md devices, partitions and filesystems in Ubuntu 10.10

Monday, November 15th, 2010

Recently I found myself in a situation, where I was running out of space in my server. Luckily for me, the disks was not 100% utilized. On the positive side, that meant, that I had the possibility to extend my system without buying new hardware. On the negative side it meant that I had to fool around with my precious data. Normally I do this for a living, but on my job it is just customer data. In this case we are talking about pictures of the kids and wife….. Much more important stuff for sure :-)

So my setup

  • / mounted as an ext3 fs on top of an logical volume (LVM2) called lvroot in vg000
  • /boot mount as an ext3 fs directly on /dev/md0
  • vg00 has one physical volume called /dev/md1
  • /dev/md0 is a RAID1 device on /dev/sda1 and /dev/sdb1
  • /dev/md1 is a RAID1 device on /dev/sda2 and /dev/sdb2

Here is the business case: both /dev/sda2 and /dev/sdb2 was not fully utilized. /dev/sdb2 had 67GB unused and /dev/sda2 had around 250GB unused. That meant that I could resize / with 67GB up front, for free!

I startet out with this situation

root@edison:~# df -k /
Filesystem             1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg000-root 240149008 232617156   2652288  99% /

So how do you do this, in an easy, controlled way, without taking time consuming backups of hundres of GB (I do make backups, so it is very easy for me to write this ;-) )? You can do it like I did below.

<DISCLAIMER>This is best-effort, free of charge information.  If you end up breaking your system into several pieces, it is your responsibility. I can and will not be held liable. You get to keep the pieces yourself! That will hopefully teach you not to trust ‘expert advice’ from some random internet place run by a random danish guy.</DISCLAIMER>

That said. Lets do some storage work

# First I fail /dev/sdb2
mdadm --fail /dev/md1 /dev/sdb2

# Then I remove /dev/sdb2 from the running config
mdadm --remove /dev/md1 /dev/sdb2
# Then we delete and recreate /dev/sdb2 with a larger size.
# Do a print beforehand to see the "before" values
fdisk /dev/sdb
Command (m for help): p

Disk /dev/sdb: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa286eb78

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          26      208813+  fd  Linux raid autodetect
/dev/sdb2              27       30400   243979155   fd  Linux raid autodetect
# delete it
Command (m for help): d
Partition number (1-4): 2
# recreate with larger size and same type
Command (m for help): n
Command action
 e   extended
 p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (27-38913, default 27):
Using default value 27
Last cylinder, +cylinders or +size{K,M,G} (27-38913, default 38913):
Using default value 38913
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)
# Write it
Command (m for help): w

If you are so lucky that you are not using the device for anything else (ie booting), the kernel can reread the modified partition table on the live system. If the system do use the device for something, you have to reboot the system. Which I had to do. Which I then did… By issuing the command:

reboot

When the server came back up, I assembled the /dev/md1 device again and waited for a resync. I speeded up the resync quite significantly by issuing the echo commands, which basically just tell md that it has to work as hard as it possibly can.

# Assemble again
mdadm --add /dev/md1 /dev/sdb2
# Speed up rebuild process
 echo 900000 > /proc/sys/dev/raid/speed_limit_min
 echo 900000 > /proc/sys/dev/raid/speed_limit_max
# watch status, eat pizze, drink a beer
watch cat /proc/mdstat

When the resync was done, I did it all over, replacing /dev/sdb2 with /dev/sda2.  After that,  I had grown the underlying partitions on my two drives, but both md and lvm had not noticed anything.

I made sure everything worked fine  and went onto the  next step – growing an md device on the fly.

# See current md device stats
root@edison:~# mdadm -D /dev/md1
/dev/md1:
 Version : 00.90
 Creation Time : Sat Jul 31 16:45:08 2010
 Raid Level : raid1
 Array Size : 243979072 (232.68 GiB 249.83 GB)
 Used Dev Size : 243979072 (232.68 GiB 249.83 GB)
 Raid Devices : 2
 Total Devices : 2
Preferred Minor : 1
 Persistence : Superblock is persistent

 Update Time : Mon Nov 15 20:16:27 2010
 State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

 UUID : f510bad0:59d990c1:19dfcc76:62eb3300
 Events : 0.204498

 Number   Major   Minor   RaidDevice State
 0       8        2        0      active sync   /dev/sda2
 1       8       18        1      active sync   /dev/sdb2

# Grow the device to the max - took about 1-2 seconds, but see below
root@edison:~#  mdadm --grow --size=max /dev/md1
# Recheck the device stat.
root@edison:~# mdadm -D /dev/md1
/dev/md1:
 Version : 00.90
 Creation Time : Sat Jul 31 16:45:08 2010
 Raid Level : raid1
 Array Size : 312359744 (297.89 GiB 319.86 GB)
 Used Dev Size : 312359744 (297.89 GiB 319.86 GB)
 Raid Devices : 2
 Total Devices : 2
Preferred Minor : 1
 Persistence : Superblock is persistent

 Update Time : Mon Nov 15 20:17:21 2010
 State : active, resyncing
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

 Rebuild Status : 78% complete

 UUID : f510bad0:59d990c1:19dfcc76:62eb3300
 Events : 0.204503

 Number   Major   Minor   RaidDevice State
 0       8        2        0      active sync   /dev/sda2
 1       8       18        1      active sync   /dev/sdb2

Notice how the drive went into sync state. Looking at mdstat confirms that. You can use the device before the mirroring is done. … or you can play it safe and wait. I decided to go ahead and use it before the sync was done. Living on the edge … this is not work. Only something your wife will never forgive you for ;-)

root@edison:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1]
[raid6] [raid5] [raid4] [raid10]

md1 : active raid1 sda2[0] sdb2[1]
 312359744 blocks [2/2] [UU]
 [===============>.....]  resync = 78.6% (245745600/312359744)
finish=21.2min speed=52265K/sec

md0 : active raid1 sdb1[1] sda1[0]
 208704 blocks [2/2] [UU]
unused devices: <none>

Next step. Making lvm aware of the change. Quite easy actually:

# Before
root@edison:~# pvdisplay /dev/md
 --- Physical volume ---
 PV Name               /dev/md1
 VG Name               vg000
 PV Size               232,68 GiB / not usable 832,00 KiB
 Allocatable           yes (but full)
 PE Size               4,00 MiB
 Total PE              59565
 Free PE               0
 Allocated PE          59565
 PV UUID               saiFOT-Dw2O-OnVv-rA4b-wZG3-OQHN-fFD1w

You then issue an pvresize on the pv  like this

root@edison:~# pvresize /dev/md1
 Physical volume "/dev/md1" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

And you end up with

root@edison:~# pvdisplay /dev/md1
 --- Physical volume ---
 PV Name               /dev/md1
 VG Name               vg000
 PV Size               297,89 GiB / not usable 2,62 MiB
 Allocatable           yes
 PE Size               4,00 MiB
 Total PE              76259
 Free PE               16694
 Allocated PE          59565
 PV UUID               saiFOT-Dw2O-OnVv-rA4b-wZG3-OQHN-fFD1wI

We are getting there? Now onto extending the volume. First peek at the current sizes/free space

root@edison:~# vgdisplay vg000
 --- Volume group ---
 VG Name               vg000
 System ID
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  4
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                1
 Open LV               1
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               297,89 GiB
 PE Size               4,00 MiB
 Total PE              76259
 Alloc PE / Size       59565 / 232,68 GiB
 Free  PE / Size       16694 / 65,21 GiB
 VG UUID               rL7lfw-H2FC-4ala-KVV3-bLIA-MbA9-co2n1O
root@edison:~# lvdisplay /dev/vg000/root
 --- Logical volume ---
 LV Name                /dev/vg000/root
 VG Name                vg000
 LV UUID                uDPn6b-AvyO-xH2n-zpXm-n0Xt-5g5n-aSdu4C
 LV Write Access        read/write
 LV Status              available
 # open                 1
 LV Size                232,68 GiB
 Current LE             59565
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           251:0

Then extend the volume

root@edison:~# lvextend -l 76259 /dev/vg000/root
Extending logical volume root to 297,89 GiB
Logical volume root successfully resized

One step left – resize the live filesystem

root@edison:~# resize2fs /dev/vg000/root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg000/root is mounted on /; on-line resizing required
old desc_blocks = 15, new_desc_blocks = 19
Performing an on-line resize of /dev/vg000/root to 78089216 (4k) blocks.
The filesystem on /dev/vg000/root is now 78089216 blocks long.

How much did I gain:

root@edison:~# df -k /
Filesystem              1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg000-root  307454080 232621080  68588172  78% /

Almost a walk in the park. I did actully run into a problem during this procedure. When I removed /dev/sdb2, grub would not reboot. Instead it gave me a grub rescue prompt.  I overcame this booting a rescue CD, starting up the system using that. I then added /dev/sdb2 into /dev/md1 from the rescue cd, waited for the mirroring and and rebooted again. When I did the same for /dev/sda2, grub was happy all along?! I have not bothered to figure out why grub misbehaved, I just put grub down as being like a teenager: unreliable and causing problems for no obvious reason ;-)

Saved by ZFS – a disk is about to die

Wednesday, May 19th, 2010

I have setup an NFS share under Solaris 10. It utilizes ZFS, which in turn uses  an IBM DS400 for backend storage. On top of that I have nagios running to monitor it. I got an alarm about the zfs pool being in a degraded state. I logged into the system and found this in the dmesg:

May 17 03:20:20 files DESC: The number of checksum errors associated with a ZFS device
May 17 03:20:20 files exceeded acceptable levels.  Refer to http://sun.com/msg/ZFS-8000-GH for more information.

To see more information I ran

-bash-3.00# zpool status
 pool: rz2pool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
 attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
 using 'zpool clear' or replace the device with 'zpool replace'.
 see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: scrub completed after 5h21m with 0 errors on Wed May 19 08:41:49 2010
config:

 NAME                        STATE     READ WRITE CKSUM
 rz2pool                     DEGRADED     0     0     0
   raidz2                    ONLINE       0     0     0
     c3t21000000D12643DEd0   ONLINE       0     0     0
     c3t21000000D12643DEd1   ONLINE       0     0     0
     c3t21000000D12643DEd2   ONLINE       0     0     0
     c3t21000000D12643DEd3   ONLINE       0     0     0
     c3t21000000D12643DEd4   ONLINE       0     0     0
     c3t21000000D12643DEd5   ONLINE       0     0     0
     c3t21000000D12643DEd6   ONLINE       0     0     0
   raidz2                    DEGRADED     0     0     0
     c3t21000000D12643DEd7   ONLINE       0     0     0
     c3t21000000D12643DEd8   ONLINE       0     0     0
     c3t21000000D12643DEd9   ONLINE       0     0     0
     c3t21000000D12643DEd10  ONLINE       0     0     0
     c3t21000000D12643DEd11  ONLINE       0     0     0
     c3t21000000D12643DEd12  DEGRADED     0     0   234  too many errors
     c3t21000000D12643DEd13  ONLINE       0     0     0

errors: No known data errors

And this is where ZFS is awsome. It may not be the fastest volume manager on the planet, or the smartest. But I trust the integrity of it (having read whitepapers on it).

What is really cool here:

  1. It has detected that the underlying LUN is misbehaving.
  2. It has marked the LUN as degraded
  3. It has saved my data from silent corruption.

There are not many volume managers out there, which does that. I have not lost data, the dataintegrity is still in place and I know what disk is about to fail. Kudos and thanks to the ZFS dev team!

Unstartable volume in vxvm

Wednesday, February 3rd, 2010

I recently had an case, where I got

ERROR V-5-1-1198 Volume misc2prd_redo2vol has no CLEAN or non-volatile ACTIVE plexes

The plex associated with the volume was in RECOVER state. This can happend “if the plex content is out of-date with respect to the volume. This can happen if a disk containing one or more of the plex’s subdisks has been replaced or reattached”.  In my case it was caused by a failing disk, that was brought online again later.  To recover I did:

# Force the plex into offline state
sudo vxmend -g misc2prd_dg2 -o force off misc2prd_redo2vol-01
# Put the plex into stale state
sudo vxmend -g misc2prd_dg2 on misc2prd_redo2vol-01
# Put the plex into clean state
sudo vxmend -g misc2prd_dg2 fix clean misc2prd_redo2vol-01

# Start the volume
sudo vxvol -g misc2prd_dg2 startall