Getting iDRAC6 working outside of IE with Java 8

This is a brief post describing how to get iDRAC6’s Virtual Console on 11th Generation Dell servers working with Java 8. This is a dead simple process but may not be apparent to someone unfamiliar with Java.

First thing you’ll need to do is configure the IP address of the iDRAC6. Hit Ctrl + E while booting in the BIOS at the appropriate time.

idrac-revised

After the iDRAC is configured for network access, log on to the web interface with the IP you configured. You’ll then need to set the virtual console plug-in type to Java from Native, the latter which uses ActiveX. This is done under Console/Media > Configuration.

idrac6-plugin-type

 

Then you’ll need to configure Java by going to Start >> Control Panel >> Programs >> Java. From the Java Control Panel go to the Security tab and then add the iDRAC’s IP to the Exception Site List.

2015-01-03 18_40_30-Java Control Panel-mod

Add the iDRAC’s IP with https.

2015-01-03 18_40_43-Exception Site List-mod

Now back in the web interface you can Launch  the console.

2015-01-03 18_51_23-unit8esxi - iDRAC6 - System Summary

Once pressed, a .jnlp file will prompt to download. Depending on the browser the file itself will download differently. Chrome seems to append parameters to the file extension which makes the next steps required each time. I use Firefox and only need to do the following of setting the default file association once. You’ll need to associate the .jnlp file with the Java Web Start application, which is located at C:\Program Files\Java\jre1.8.0_25\bin\javaws.exe.

2015-01-03 18_53_17-Open with-mod

After you set the association you be prompted with two security warnings and after about 20 seconds the virtual console will start.

2015-01-03 18_55_57-mod

The iDRAC virtual console works out of the box for Ubuntu and FreeBSD with IcedTea and Firefox installed. With OSX, simply open the .jnlp file with the Java Web Start.app application at /System/Library/CoreServices/.

Posted in How To, Technology | Leave a comment

Solaris 11.2, a perfectly timed release and how it saved me hours of resilver time.

Solaris 11.2 was released on July 31st, 2014, with the release brings version 35 of zpool among a many other things, most of which are irrelevant as a home user. However, with the new zpool version 35 there is an markedly increase in resilver performance which I delve into with this post. Below you can see the description Oracle has given zpool version 35 from the output of zpool upgrade -v below:

gullibleadmin@epijunkie:~# zpool upgrade -v
This system is currently running ZFS pool version 35.

The following versions are supported:

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 Slim ZIL
24 System attributes
25 Improved scrub stats
26 Improved snapshot deletion performance
27 Improved snapshot creation performance
28 Multiple vdev replacements
29 RAID-Z/mirror hybrid allocator
30 Encryption
31 Improved 'zfs list' performance
32 One MB blocksize
33 Improved share support
34 Sharing with inheritance
35 Sequential resilver
 
For more information on a particular version, including supported releases,
see the ZFS Administration Guide.

Oracle is defining “Sequential resilver” as follows:

The previous resilvering algorithm repairs blocks from oldest to newest, which can degrade into a lot of small random I/O. The new resilvering algorithm uses a two-step process to sort and resilver blocks in LBA order.

The amount of improvement depends on how pool data is laid out. For example, sequentially written data on a mirrored pool shows no improvement, but randomly written data or sequentially written data on RAID-Z improves significantly – typically reducing time by 25 to 50 percent.

I can attest to the performance increase this provides as a single drive resilver is only requiring 1/2 of the previous time. I suspect that I would not reasonably be able to replace all the drives in the vdev simultaneously without massive IO problems reflecting as more than a weeks worth of dedicated resilver time without this new algorithm in zpool version 35.

This project began with buying 6x 5TB drives to add as a RAIDZ2 vdev to my current pool. As I already have a 6x RAIDZ2 vdev using 3TB drives, I plan to swap the 5TB drives in place of the current 3TB drives utilizing the autoexpand=on setting. After the pool with the single (6x 5TB) vdev has resilvered and autoexpanded, I will then add the 3TB drives as a vdev. In doing this the pool will be closer to equal free space between the vdevs than if I were just to add the 5TB drives as a vdev. From what I’ve read and watched, ZFS utilizes dynamic stripping [Page 37], meaning that data is adaptively striped across the vdevs. As you can see below, I gain a distinct advantage swapping the drives to 5TB before adding the second vdev due to the way ZFS dynamically stripes the data across the vdevs using free space as the primary metric. Also worth mentioning, by using zfs send and zfs recv to a new pool (consisting of the 6x 5TB drives) instead of swapping the drives I would effectively defrag the pool but to me this is not soo important.

ZFS write balance

This table shows how the pool’s free space would be divided after adding a vdev. The table is presented in RAW space in terabytes, as manufactures define a terabyte. According to George Wilson, a former developer at Sun and Oracle, the code responsible for writing data across vdevs at least up to zpool version 28 would attempt to balance the free space between vdevs but uses a weak algorithm with a maximum preferential write load of 25%. In my case the vdevs would never reach a balance of the new data being written to my pool if I were to just add a 6x 5TB vdev. I speculate that this would cause a catastrophic performance loss as the full vdev reached it’s capacity and the 512 byte striping requirement would struggle to find space in the fragmentation as it writes data across all drives in all vdevs. There is hope for the users of the feature flag branch of ZFS as the developers at the OpenZFS have enhanced the write balance to free space against varying aged vdevs.

I am replacing the drives using the zpool replace command. When replacing the drives with this command, as long as the replaced drive(s) is still attached the drive will remain online and current; because of this the pool is not at a higher risk of data loss in terms of the parity loss because the drive and parity is still online. The output below is from the pool while using zpool version 34, as you can see the resilvering process is dragging along due to the IO limit of the vdev being asked to reassemble blocks from oldest to newest. This is actually where my experience with zpool version 35 started. I happened to come across the news that Solaris 11.2 had been released from beta and I happened to notice new zpool version described as “Sequential resilver” which peaked my interest as my current resilver process was dragging along. After reading as much as I could with a newly released product I decided to test it out and ended up detaching the drive before the resilver process completed on version 34 due to my impatiences.

gullibleadmin@epijunkie:~# zpool status version34
  pool: version34
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Aug  2 16:33:38 2014
    3.12T scanned out of 12.9T at 45.4M/s, 62h28m to go
    1.04T resilvered, 24.27% done
config:
config:

        NAME                         STATE     READ WRITE CKSUM
        version34                    DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            replacing-0              DEGRADED     0     0     0
              c0t50014EE6ADAECACFd0  ONLINE       0     0     0
              c0t5000C50073B0B76Ad0  DEGRADED     0     0     0  (resilvering)
            c0t50014EE6ADAF9785d0    ONLINE       0     0     0
            c0t50014EE0037F34DAd0    ONLINE       0     0     0
            c0t50014EE2B2B5AF19d0    ONLINE       0     0     0
            c0t50014EE6583B7CA1d0    ONLINE       0     0     0
            c0t50014EE6585382A7d0    ONLINE       0     0     0

errors: No known data errors

Here is the output when I ran the same replace command, now on zpool version 35. As you can see below the process is different with the algorithm change (Stage 1 – Scanning):

gullibleadmin@epijunkie:~# zpool status -lx
  pool: version35
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Sun Aug  3 15:14:45 2014
    4.58T scanned out of 12.9T at 3.51G/s, 0h40m to go
    0 resilvered
config:

        NAME                         STATE     READ WRITE CKSUM
        version35                    DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            replacing-0              DEGRADED     0     0     0
              c0t50014EE6ADAECACFd0  ONLINE       0     0     0
              c0t5000C50073B0B76Ad0  DEGRADED     0     0     0  (resilvering)
            c0t50014EE6ADAF9785d0    ONLINE       0     0     0
            c0t50014EE0037F34DAd0    ONLINE       0     0     0
            c0t50014EE2B2B5AF19d0    ONLINE       0     0     0
            c0t50014EE6583B7CA1d0    ONLINE       0     0     0
            c0t50014EE6585382A7d0    ONLINE       0     0     0

errors: No known data errors

The first stage of the resilver process is where ZFS scans the entire pool and waits until the second stage to write the resilvered data to the drive. I speculate that in stage one, ZFS reads the entire vdev in LBA order, that is to say in the order the data is currently written on the disk from address 0 to address xxxxxxxxxx. I speculate that after ZFS knows the LBA order, it compares that against the block age and by block age I mean as the pool experienced writing blocks in chronological order which does not typically correlate with LBA order when data is no longer sequential and fragmented. I also speculate that ZFS creates a reference table from these two data points to optimize the drive resilver to maximize drive bandwidth by utilizing this new way of doing a resilver in sequential order rather than by block age where fragmentation plays a large role in the performance.

Once Stage 2 starts, the data is being written to the new drive(s) and the speed inevitably drops to a lower but consistent speed, the numbers I observed ranged from 109M/s to 145M/s with a median of 119M/s. This is a large difference from zpool version 34 where the bandwidth ranged at a greater delta and also performed at a much lower median. The numbers I observed on the zpool version 34 resilver would range from 1M/s to 110M/s with a median of 25M/s. As you’ll notice from the output below from stage 2 of the resilver (zpool version 35), the speed observed is likely the write-bandwidth-limit of the new drive limiting the resilver process, not the vdev IO limit. still up for debate, see the end of this post as I updated with more experimenting.

Stage 2:

gullibleadmin@epijunkie:~# zpool status version35
  pool: version35
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Sun Aug  3 15:14:45 2014
    12.9T scanned
    638G resilvered at 129M/s, 29.01% done, 20h42m to go
config:

        NAME                         STATE     READ WRITE CKSUM
        version35                    DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            replacing-0              DEGRADED     0     0     0
              c0t50014EE6ADAECACFd0  ONLINE       0     0     0
              c0t5000C50073B0B76Ad0  DEGRADED     0     0     0  (resilvering)
            c0t50014EE6ADAF9785d0    ONLINE       0     0     0
            c0t50014EE0037F34DAd0    ONLINE       0     0     0
            c0t50014EE2B2B5AF19d0    ONLINE       0     0     0
            c0t50014EE6583B7CA1d0    ONLINE       0     0     0
            c0t50014EE6585382A7d0    ONLINE       0     0     0

errors: No known data errors

I imagine there is a low number of home users of Solaris but to those who do, I highly recommend upgrading to version 35 of the zpool. It will be curious to find out how this new algorithm impacts drives that are beginning to fail, especially in the home environment where lower performance and lower quality desktop drives are typically used. I say this in part due to something I observed when I initiated the zpool replace command on the remaining 5 drives before the first drive’s resilver process completed. The curious event was the resilvering process restarted for the pool/vdev/drive of which had already progressed to 80% on the first drive. I’m not sure if this was because the replaced drive was still online, or a byproduct of the new code in zpool version 35 / Solaris 11.2, or simply a bug in the code. I do not suspect it was a bug in the zpool status output command as well after the projected time elapsed for the first drive to complete, as indicated before issuing the replace command for the 5 other drives; the drive IO was still indicating it was being resilvered as well as the zpool status output was still indicating the drive was being replaced/resilvered. As I did not lose any data from this blip I am not terribly concerned, but a curious ‘feature’.

This post will end with Some more output from commands, because who doesn’t love tables/graphs:

gullibleadmin@epijunkie:~# zpool iostat -v jackscoldsweat

                               capacity     operations    bandwidth
pool                         alloc   free   read  write   read  write
---------------------------  -----  -----  -----  -----  -----  -----
jackscoldsweat               12.9T  3.41T  1.18K      0   147M  1.59K
  raidz2                     12.9T  3.41T  1.18K      0   147M  1.59K
    replacing                    -      -  1.16K     20  36.8M  90.1K
      c0t50014EE6ADAECACFd0      -      -  1.03K      0  36.9M  3.71K
      c0t5000C50073B0B76Ad0      -      -      0    998      0  36.9M
    replacing                    -      -  1.16K     20  36.8M  98.1K
      c0t50014EE6ADAF9785d0      -      -   1020      0  36.9M  3.71K
      c0t5000C50073B0CDEDd0      -      -      0    982      0  36.9M
    replacing                    -      -  1.15K     21  36.7M   105K
      c0t50014EE0037F34DAd0      -      -   1010      0  36.7M  3.71K
      c0t5000C50073B0D5DFd0      -      -      0    987      0  36.8M
    replacing                    -      -  1.16K     21  36.8M   101K
      c0t50014EE2B2B5AF19d0      -      -   1008      0  36.9M  3.71K
      c0t5000C50073B4E90Cd0      -      -      0    970      0  36.9M
    replacing                    -      -  1.16K     20  36.8M  99.6K
      c0t50014EE6583B7CA1d0      -      -   1006      0  36.9M  3.71K
      c0t5000C50073B1031Dd0      -      -      0    993      0  36.9M
    replacing                    -      -  1.15K     22  36.8M   111K
      c0t50014EE6585382A7d0      -      -   1006      0  36.8M  3.71K
      c0t5000C50073B08181d0      -      -      0    972      0  36.9M
---------------------------  -----  -----  -----  -----  -----  -----

gullibleadmin@epijunkie:~# zpool status jackscoldsweat
  pool: jackscoldsweat
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Mon Aug  4 15:52:17 2014
    12.9T scanned
    7.49T resilvered at 124M/s, 58.11% done, 12h41m to go
config:

        NAME                         STATE     READ WRITE CKSUM
        jackscoldsweat               DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            replacing-0              DEGRADED     0     0     0
              c0t50014EE6ADAECACFd0  ONLINE       0     0     0
              c0t5000C50073B0B76Ad0  DEGRADED     0     0     0  (resilvering)
            replacing-1              DEGRADED     0     0     0
              c0t50014EE6ADAF9785d0  ONLINE       0     0     0
              c0t5000C50073B0CDEDd0  DEGRADED     0     0     0  (resilvering)
            replacing-2              DEGRADED     0     0     0
              c0t50014EE0037F34DAd0  ONLINE       0     0     0
              c0t5000C50073B0D5DFd0  DEGRADED     0     0     0  (resilvering)
            replacing-3              DEGRADED     0     0     0
              c0t50014EE2B2B5AF19d0  ONLINE       0     0     0
              c0t5000C50073B4E90Cd0  DEGRADED     0     0     0  (resilvering)
            replacing-4              DEGRADED     0     0     0
              c0t50014EE6583B7CA1d0  ONLINE       0     0     0
              c0t5000C50073B1031Dd0  DEGRADED     0     0     0  (resilvering)
            replacing-5              DEGRADED     0     0     0
              c0t50014EE6585382A7d0  ONLINE       0     0     0
              c0t5000C50073B08181d0  DEGRADED     0     0     0  (resilvering)

errors: No known data errors

Another thanks goes out to Allan Jude as the inspiration of swapping the drives to create a large vdev to better balance the free space ratio.

 

UPDATE: 2014 August 6, Wednesday 1137

Keeping my options open, I left the autoexpand flag to “off” until I was sure that was the course of action I wanted to take; my reservation being the benefit of defragmenting the pool by instead creating a new pool and zfs send-ing the data over instead of swapping the drives in place; this would have required me to re-replace the 5TB back with the 3TB, which I was prepared to do. Curiosity had me see what would happen if I were to swap the drives back using the zpool replace command.

gullibleadmin@epijunkie:~# zpool status jackscoldsweat
  pool: jackscoldsweat
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Wed Aug  6 10:52:37 2014
    12.9T scanned
    488G resilvered at 331M/s, 3.70% done, 10h54m to go
config:

        NAME                         STATE     READ WRITE CKSUM
        jackscoldsweat               DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            replacing-0              DEGRADED     0     0     0
              c0t5000C50073B0B76Ad0  ONLINE       0     0     0
              c0t50014EE6ADAECACFd0  DEGRADED     0     0     0  (resilvering)
            replacing-1              DEGRADED     0     0     0
              c0t5000C50073B0CDEDd0  ONLINE       0     0     0
              c0t50014EE6ADAF9785d0  DEGRADED     0     0     0  (resilvering)
            replacing-2              DEGRADED     0     0     0
              c0t5000C50073B0D5DFd0  ONLINE       0     0     0
              c0t50014EE0037F34DAd0  DEGRADED     0     0     0  (resilvering)
            replacing-3              DEGRADED     0     0     0
              c0t5000C50073B4E90Cd0  ONLINE       0     0     0
              c0t50014EE2B2B5AF19d0  DEGRADED     0     0     0  (resilvering)
            replacing-4              DEGRADED     0     0     0
              c0t5000C50073B1031Dd0  ONLINE       0     0     0
              c0t50014EE6583B7CA1d0  DEGRADED     0     0     0  (resilvering)
            replacing-5              DEGRADED     0     0     0
              c0t5000C50073B08181d0  ONLINE       0     0     0
              c0t50014EE6585382A7d0  DEGRADED     0     0     0  (resilvering)

errors: No known data errors

Hmm, this is curious. Did the resilver from the first replacement of the drives defragment the drive? It seems like it. The first replacement ended up taking 35 hours and 29 minutes. This is projecting about 1/3 of the previous time to resilver the drives, after what I suspect was a resilver AND defragmentation.

gullibleadmin@epijunkie:~# zpool status jackscoldsweat
  pool: jackscoldsweat
 state: ONLINE
  scan: resilvered 12.9T in 35h29m with 0 errors on Wed Aug  6 03:21:22 2014
config:

        NAME                       STATE     READ WRITE CKSUM
        jackscoldsweat             ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t5000C50073B0B76Ad0  ONLINE       0     0     0
            c0t5000C50073B0CDEDd0  ONLINE       0     0     0
            c0t5000C50073B0D5DFd0  ONLINE       0     0     0
            c0t5000C50073B4E90Cd0  ONLINE       0     0     0
            c0t5000C50073B1031Dd0  ONLINE       0     0     0
            c0t5000C50073B08181d0  ONLINE       0     0     0

errors: No known data errors
Posted in Technology | Tagged , , | Leave a comment

A rant on privacy, the interwebs, and publicly accessible information.

Years ago I read a news article in which an employee was terminate for the contents on his personal blog. One of his post was something his employer did not agree with, the article was requested to be removed, and was, but the employee still ended up without a job. Ever since then I have been very aware of the non-forgetful nature of the interwebs and how that reflects on me and my name.

What is posted on the internet usually stays accessible on the internet forever. To pull an excerpt from Archive.org: The Internet Archive is working to prevent the Internet – a new medium with major historical significance – and other “born-digital” materials from disappearing into the past. This is just one entity with internet access and hard drive space. Another example is in regards to celebrities, it seems common now for celebrities to be ‘caught’ with compromising photos/opinions that are ‘online’ for just minutes and yet someone in that time grabs the data and saves it locally to their harddrive, then reposts it to a “news” organization such as TMZ. I have also recently became aware of a practice by some data hoarders of ‘racking the internet’ also known as automated data collection which content is saved from any online source to their local archival storage. I am not a celebrity nor have the same number of eyes on me but all it takes is one person or entity with my content to impact the world longer than I might of intended.

What is really interesting about typically private people is how reckless that same person can be when it comes to digital forms. I am amazed what people have posted, everything from the nightclub they currently in (likely inebriated), to the name and photos of their children, to private thoughts or views of the world. I find there to be a misplaced hope that people will respect their “privacy” and not search the internet, a publicly searchable index of most things digital. The problem with that hope, I think, is an assumption that all people are good natured. While I’d like to live in that world, it is not the world I currently occupy and any person can do a search of my name, user handles, email addresses, or other identifiers and find a trove of information. If I was driving on the highway and passed a billboard with a photo of me on it, I would do everything in my power to have it removed. There is an obviously difference for a digital medium, however most people will still cringe when a baby photo is shown to their significant other (a person they trust.) But strangely we are giving that ability to browse photos of us to any entity on earth with an internet connection.

So far I have largely only mentioned content we create reflecting a person individually but lets consider our offspring. I feel it is a great disservice we are doing to our children. Frequently their lives are digital before the choice is theirs’ and many children/teens are given a means to post to the internet in some capacity before they realize what they are doing. I think this problem stretches from newborns to young adults and beyond. I know for me personally, I am glad that my life as a teen was largely before social media. I feel like my life would of been recorded in a haphazard way on a permanent storage device called the internet. Teens have incredible capacity for stupidity, myself included.

So far this rant has been from the perspective of a preventative stance but what if the content is already there? This is why I do periodical searches on myself with different starting points. I might start a search with just my name, email address, or username and see how much information I can spawn from that. The results I find I typically have control over the content in some way, such as a forum post, or a website I directly run. Occasionally results will come up on a medium I don’t have any control over such as a news article which mentions me. For those I typically try and limit the information supplied initially. If that’s not enough you can typically request the information be removed or redacted. Legal action is also an avenue to investigate if your online presences really needs to reflect you in a prescribed way.

A method recommend for reducing your cross-searchablility is using profiles. In this context, using a profile means to use a set of unique identifiers such as email addresses and user names that all eventually point back to a single person either by name or association. Many people I know will have a profile that deals strictly with personal matter; one that deals business or professional matters; and perhaps another for spam,  or stuff that there is anticipate junk mail or a temporary usage; Each profile with a different set of user names (and passwords obviously) and email addresses that do not intermingle.

The reason I suggest using different email addresses is that many sites will allow you to do a search for a user by their email address, for example Facebook. You might not know the person’s exact spelling of their name but if you have their email address, you can do a search on Facebook and typically pull that person up.

I also highly recommend doing these searches and browsing in a private browsing session. The reason is, that private session will lack all cookies pointing to you, including unique identifiers and saved logged-in sessions such as Gmail which will influence the results you get; the point is to start from a fresh perspective without context, to see as any person would see. Another option if you want even more certainty of a pure search is using a live linux CD/DVD.

While you are not who you appear to be as represented in an online presences or really any constituent; the ‘whole person’ perceived  is how you represent yourself including in-person interactions, the subsequent opinions of those social interactions, your writings and works, and the digital footprint that you impress upon the world. While some of those impressions upon the world are on fallible recording devices such as human brains, the interwebs is typically stored on redundant disks arrays that typically never fail. Unfortunately many people treat internet postings as fallible things, or things that will expire at some point which they are largely not.

Posted in Commentary, Rants, Socially Curious | Leave a comment

Burnt IC replacement project.

Recently I purchased a used rack mountable 16 outlet power strip (Synaccess netBooter NP16) which has both serial console access and network access. This is a really cool piece of gear as it allows you to remotely turn on individual power outlets from (if configured) anywhere in the world.

IMG_4818IMG_4819The problem though is the power strip was broken. It appeared to be working, or at least the 16 LEDs to indicate the outlets turned on when powered up. Only 8 of the 16 internal relays (yellow components below) were cycling when powering on.

IMG_4812After some testing with my Fluke multimeter, I found power was getting to the both halves of the board, both outlet banks were still conducive, and that’s when I found one of the integrated-circuits [IC] was completely burnt out.

IMG_4815-2In my past I have repaired burnt out traces on boards but not a component, such as an IC. I figured I didn’t have anything to lose as this broken unit became free. I contacted the seller and he shipped me another working unit and didn’t want the old one back. Also figured that if I repaired the broken one I could pay for all the components and gear I buy to pay for the project.

IMG_4813I bought a nicer soldering iron and a solder sucker. As for the components, I was lucky enough that there was an identical chip design internally which allowed me to pull the part number for the IC.

IMG_4817I ordered the parts from Mouser and waited a week for the IC to be delivered. The part came in, I replaced the IC after watching a video online, reassembled everything, and plugged in the unti to the wall, and !POP! The brand new IC blew across the garage. I was stumped, so I asked a friend, an engineer what he thought. His suggestion was to check the components against the good board and he would forward the information to his father, an electrical engineer. One more week later his father got back to us and suggested I check the operation of the AC bridge rectifier. I was getting inconsistent readings on the bridge rectifier against the good board so I decided it need to be replace. I also noticed the capacitor was bulging so I decided to spend the extra $3.30 for it and buy it rather than wait. So I ordered a bridge rectifier (DF06-G), a capacitor (EEU-ED2G330S), and another IC (TNY267PN) from DigiKey and a week later the parts were here. At this point my roommates had heard of my peril of exploding IC chips and wanted to witness it first hand. They were disappointed when it worked after plugging in the unit to the wall after replacing the components. A few weeks later everything is still working, pretty good for someone with most of their knowledge coming from the interwebs.

Posted in Technology | Leave a comment

Wireless SSIDs tied to VLANs using a Cisco Aironet and pfSense

Background

This project came about while searching for a new housing arrangement, one involving housemates. You see, many housemates are not techies; most are open to you dropping in your hardware for the internet service and local network. They like it because their bills go down because they are no longer renting equipment from the ISP and also like the situation because they have on-site technical support. These type of arrangements are great for me because I like the control and my hardware performs better than the rented equipment. Also there is typically no threat in these non-techie situations that your equipment is going to be pen-tested. It is sort of like sitting your grandmother at a terminal and saying ‘go crazy’; no matter what she does, if she’s like most grandmothers, she’ll never run metasploit against your devices, or port scan the network, or innocently explore all the directories in your network shared folders. However, if you do decide to live with people that self identify as a ‘techie’, you do run the risk of one or all of those situations happening to some degree.

One possible solution to this, is to obfuscated tentacle hentai as a document folder full of PDF diary entries on a password-less network share to send a subtle message to that nosy/curious roommate. However this method only works to a point, before curiosity turns to interest. Instead, I’ve decided to avoid the situation all together and logically separate the networks with VLANs including using separate SSIDs.

This project, like many projects as of late is something I find really cool. Cool because it’s over most people’s head, including my own at first, so it was a fun challenge to setup. These articles further my own knowledge in the research required to fully understand it, they also document the process should I have catastrophic failure of hardware (brain, storage, and backups), and last I hope these articles provide instruction to at least one other person besides myself.

 

 

How does this VLAN stuff work?

VLANs allow isolation of physical hardware into logical groups, on switches this isolation is port based. However on a wireless access point, the VLAN isolation can be based on the SSID. VLANs are most apparent from the perspective of a switch. Managed switches are the heart of VLAN operation. Lets take a managed switch.

2014-04-30 17_00_04-Managed Switch

When you buy a managed switch, you plug it in and connect your network-printer, PC, and NAS; everything just works; it’s features are being wasted but it works out of the box.

2014-04-30 23_34_36-YourLAN

We’ll say in this hypothetical situation, you get a dorm-mate. He has a PC and router he wants to use in conjunction with your switch. No big deal, you have extra ports, you know about VLANs, you don’t want interwebs, and there doesn’t need to be connectivity between your ports and the dorm-mate’s.

2014-04-30 23_33_41-VLANed

So you get on your managed switch and create two VLANs. VLAN33  is assigned to port 1 (network-printer), port 2 (PC), and port 3 (NAS.) On VLAN44 your dorm-mate’s PC is on port 4 and the router on port 5. The switch logically separates and groups ports 1,2,3 into one group; ports 4,5 are in another group. Communication between the two VLANs is impossible for two reasons; one, the switch is no longer bridging those ports and sees them as completely separate groups. Two, best practice says to put each VLAN on it’s own subnet; due to using multiple subnets, a router or a L3-switch is needed to communicate between the VLANs. With this demonstration your VLAN does not have internet access because the router is on the other VLAN. BUT wait, I just said you need a L3 device to communicate between VLAN, what’s different in this case? That will be answered next.

In the diagram above none of the devices except the managed switch are even aware VLANs are being used. But lets say your dorm-mate then wants to use your printer and you’d like to connect to the internet. This is possible by using switchport trunking and VLAN tagging. Tagging actually was occurring before in the last arrangement; when a frame enters a port it is tagged with which ever VLAN id is assigned to that port. But when the frames were leaving out to the destination port the frames were untagged. So why is tagging even used? Tagging is used to keep track of where a frame can and can’t go. The next question is, well if the tagging is stripped when it leaves the destination port (an access port to be more precise now), how then do frames that are tagged get to a router? The answer, is to configure the switch port going to the router as a trunk port and also configure which VLAN ids are allowed to be on that trunk port. The router’s interface is then configured to have sub-interfaces for each VLAN it’s going to handle. The sub-interfaces act as regular interfaces but share the same physical port.

Below is the new configuration in our hypothetical situation. The router is now acting as ‘router-on-a-stick’. This means that the router now handles communication between the VLANs, which personally to me is amusing considering the VLANs are on the same switch at least in this situation.

2014-04-30 23_31_54-VLANed - Trunked

At this point you can access the internet and your dorm-mate’s PC and she can access your devices including your printer. A firewall rule on the router easily handles restricting traffic from your dorm-mate’s subnet to only your printer.

Another cool thing you can do with VLANs, tagging, and trunking is to carry VLAN traffic from one switch across a campus to another location. This effectively allows you to group individuals in an enterprise setting by department rather than geographical location. This might be extremely effective for a company that might have developers in a few buildings on their property that share network resources containing trade secrets but share their office space with other ‘less-secure’ employees such as interns. To further our example, lets say you want to play a LAN based multiplayer game such as CS1.6 with a friend in another dorm room. Setting up the routers and switches to pass VLAN tagged frames throughout the network would allow you to do this. As you can see below VLAN33 is in both your dorm room and your friends door room.

2014-04-30 22_42_57-Expansion

Lets now proceed to applying VLANs to wireless devices and instead of using ports as the VLAN boundary we’ll use the wireless SSIDs.

 

 

Hardware and software used

The hardware and software we’re using here: a Cisco Aironet 1040 Series wireless access point for radio communication and pfSense 2.1 whitebox to function as a router-on-a-stick for restricted access between the subnets/VLANs.

 

 

Network reference data

Guest Network

  • VLAN20
  • 10.0.20.0/24
  • Default Gateway, DNS, and DHCP server – 10.0.20.1
  • SSIDs public and public-5G

VLAN30 – Your Network

  • VLAN30
  • 10.0.30.0/24
  • Default Gateway, DNS, and DHCP server – 10.0.30.1
  • SSIDs epijunkie and epijunkie-5G

VLAN40 – Roommates Network

  • VLAN40
  • 10.0.40.0/24
  • Default Gateway, DNS, and DHCP server – 10.0.40.1
  • SSIDs roommate and roommate-5G

The native VLAN is left with the default configuration of 1.

The WebGUI is located at 10.0.1.2 as configured through the BVI interface.

 

 

Configuring the Cisco Aironet

These are the bare minimum configurations need to get both the 2.4Ghz and 5.0Ghz radios working with VLAN segmentation between the SSIDs. This guide assumes you are using CLI to configure the AP and that you’ll be using WPA2-PSK for authentication.

 

This first block of configuration sets up the SSIDs and which VLAN to associate with it, the passwords (encrypted upon entry into the configuration but easily decrypted with this tool), and the key management method.

dot11 ssid public
 vlan 20
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii guestpassword
 mbssid Guest-mode


dot11 ssid epijunkie
 vlan 30
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii temptemp
 mbssid Guest-mode


dot11 ssid roommate
 vlan 40
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii temptemp
 mbssid Guest-mode


dot11 ssid public-5G
 vlan 20
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii guestpassword
 mbssid Guest-mode


dot11 ssid epijunkie-5G
 vlan 30
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii temptemp
 mbssid Guest-mode


dot11 ssid roommate-5G
 vlan 40
 authentication open
 authentication key-management wpa version 2
 wpa-psk ascii temptemp
 mbssid Guest-mode

 

This block of configuration configures the base radio interfaces, which encryption scheme and cipher to use, defines the SSIDs names on the radio interface, and then turns on the radio.

interface dot11Radio 0
 mbssid
 encryption vlan 20 mode ciphers aes-ccm
 encryption vlan 30 mode ciphers aes-ccm
 encryption vlan 40 mode ciphers aes-ccm
 ssid public
 ssid epijunkie
 ssid roommate
 no shutdown

interface dot11Radio 1
 mbssid
 encryption vlan 20 mode ciphers aes-ccm
 encryption vlan 30 mode ciphers aes-ccm
 encryption vlan 40 mode ciphers aes-ccm
 ssid public-5G
 ssid epijunkie-5G
 ssid roommate-5G
 no shutdown

 

This block of configuration sets up the subinterfaces for each VLAN, the bridge groups, and also points to the dhcp server to relay from.

interface Dot11Radio0.20
 encapsulation dot1Q 20
 bridge-group 20

interface Dot11Radio1.20
 encapsulation dot1Q 20
 bridge-group 20

interface GigabitEthernet0.20
 encapsulation dot1Q 20
 ip helper-address 10.0.20.1
 bridge-group 20
 no bridge-group 20 source-learning
 bridge-group 20 spanning-disabled

 

interface Dot11Radio0.30
 encapsulation dot1Q 30
 bridge-group 30

interface Dot11Radio1.30
 encapsulation dot1Q 30
 bridge-group 30

interface GigabitEthernet0.30
 encapsulation dot1Q 30
 ip helper-address 10.0.30.1
 bridge-group 30
 no bridge-group 30 source-learning
 bridge-group 30 spanning-disabled

 

interface Dot11Radio0.40
 encapsulation dot1Q 40
 bridge-group 40

interface Dot11Radio1.40
 encapsulation dot1Q 40
 bridge-group 40

interface GigabitEthernet0.40
 encapsulation dot1Q 40
 ip helper-address 10.0.40.1
 bridge-group 40
 no bridge-group 40 source-learning
 bridge-group 40 spanning-disabled

 

This configures the base ethernet device and also VLAN1 subinterface.

interface GigabitEthernet0
 no ip address
 no ip route-cache
 duplex auto
 speed auto
 no keepalive

interface GigabitEthernet0.1
 bridge-group 1
 encapsulation dot1Q 1 native
exit

 

This configures the default Bridge Group Virtual Interface 1. This interface is where you configure the IP for managing the AP.

bridge irb
bridge 1 route ip
interface BVI1
 ip address 10.0.1.2 255.255.255.0
exit

 

Not required but sure helpful for not getting interrupted while typing in commands.

line con 0
 logging sync
exit

 

 

Configuring pfSense

This is a fairly straight forward configuration of pfSense. It just works and is pretty powerful. I highly recommend using pfSense as your router if you aren’t already. I actually spent most of the time wrestling the Cisco AP to communicate with pfSense; both of which use a standard called 802.1q which covers VLANs.

 

First you’ll need to create VLANs under Interfaces > (assign) > VLANs > “+

2014-05-11 03_00_01-pfSense.localdomain - Interfaces_ VLAN

Add each VLAN. In the case below we are configuring the guest VLAN.

2014-05-11 03_16_08-pfSense.localdomain - Interfaces_ VLAN_ Edit

 

Should look like this after you’ve created all the VLANs.

2014-05-11 03_02_22-pfSense.localdomain - Interfaces_ VLAN

Next to create the sub-interfaces for each VLAN.

2014-05-11 03_03_02-pfSense.localdomain - Interfaces_ Assign network ports

 

Afterwards:

2014-05-11 03_03_38-pfSense.localdomain - Interfaces_ Assign network ports

 

Now enable each sub-interface and configure an IP address. I used the 10.0.X.1 scheme, where X is the VLAN id. This can be done under Interfaces > OPTx.

2014-05-11 03_04_44-pfSense.localdomain - Interfaces_ OPT1

 

After each sub-interface is enabled and IP address is configured, the DHCP service needs to be enabled on each of the sub-interfaces. Services > DHCP Server > Sub-interface Name.

2014-05-11 03_06_42-pfSense.localdomain - Services_ DHCP server

 

For simplicity sake I have basic (highly insecure) firewall rules set in this guide. I do however recommend you create rules that restrict traffic between interfaces, especially your guest network. If you aren’t familiar, these rules are evaluated from top to bottom, the first rule to match is the one that is used to handle the packet (drop, reject, pass.) There is also an implicit deny all  (catch all) at the end which will drop any packet that made it that far without matching another rule.

2014-05-11 03_25_44-pfSense.localdomain - Firewall_ Rules

 

An allow any (bad practice) firewall rule.

2014-05-11 03_26_13-pfSense.localdomain - Firewall_ Rules_ Edit

 

Create rules as you see fit and keep your data safe from prying eyes.

2014-05-11 03_30_05-pfSense.localdomain - Firewall_ Rules

 

 

Final Thoughts

While VLANs are not absolutely secure, they are a measure of deterrent. As with all security matters there is a trade-off of convenience and true security. In my case they provide decent separation of the LAN traffic while still providing the possibility of sharing resources.

Posted in How To, Technology, Wireless | Tagged , , , , , , | Leave a comment

Thoughts and feelings on data storage implementations after four years of immersion.

Background:

Four years ago I got serious about the way I stored my data. At that time I had four harddrives, each with a different filesystems, each with different types of data, and the drives were scattered between different PCs. I transferred files between the PCs with thumbdrives, DVDs, and occasionally SFTP. Overall the system in general was inefficient. So I set out to build something that: insured data-integrity, would allow me to access all the data from multiple devices over a network simultaneously, and would be fast and streamlined. ZFS was the obvious and indisputable choice. The choice I struggled with for a while was the OS to implement ZFS with; during my search, my system was optimized, time pasted and different configurations were tested; eventually leading to data storage nirvana.

After reading extensively, I dove deep into this journey. It started by using Solaris running as a virtual machine running on top of ESXi. An ESXi whitebox was setup as an All-In-One [AIO] server. The ESXi hypervisor ran off a USB boot device. In addition there was a very small VMFS datastore stored on a couple of junk harddrives in a hardware-RAID1 configuration. This small VMFS datastore had a Solaris virtual machine [VM] and a pfSense VM. pfSense would boot-up my network services (WAN and LAN DHCP) and then the Solaris VM would boot-up the primary data including the iSCSI VMFS. Both of these VMs used PCIe passthrough: Solaris received a HBA through PCIe passthrough and pfSense a dual port NIC. When Solaris finally booted I would have to reconfigure the iSCSI LUNs due to the encryption. After the LUNs/zvols were reconfigured they were presented to the ESXi machine via iSCSI as a datastore which contained the non-critical VMs.

Even at the time I found it kinda funny the system relied on a nested VM to give itself more data. However this setup was limited, it did not lend it self to being pushed. This setup with the nested storage was fairly easily to bring a crawl or entirely down by running too many VMs or IOs. To further my own caution the thought of restarting, a 30 minute process, hindered my desire to be creative. To me, 30 minutes from the time of pressing the power button to the time of having the nested datastor back up and running was too long. There were a number of non-scriptable (at least at the time by the talent I had then) commands that were needed to get everything to startup in the correct order on the separate OSes. Another contributing factor was once the AIO went down so did the WAN connection.

During one crash, the entire system collapsed. The smaller datastore became corrupted and I couldn’t access the massive datastore which had the disk images to reinstall pfSense and Solaris 11. This equated to about a week of downtime while I downloaded the install ISOs on a slow connection.

I recently switched to a multiple-independent-server-setup (4x Dell R610s, 1x Dell R510, 1x Norco DAS). One of the R610s is used as a 12-port-router, the R510 is used for data storage (both SAN and NAS), and the remaining R610s are used as a lab for testing configurations and breaking them. This latest setup is the best yet, as the ability to be reckless on the lab without consequence has allowed for me to learn things I would of been too cautious to try before.

 

So here are some quick tips from my experience:

  • Unless you are building a strictly-play-only-lab don’t use Intel Engineering Sample CPUs. The CPUs are cheaper than the retail releases, the CPUs typically work without issue most of the time, but for me it seemed to Purple-Screens-of-Death on VMware at the most inopportune times.
  • Whitebox servers are definitely cheaper than same-component generation server equipment from a brand but sometimes at the cost of stability. For example I had a SuperMicro motherboard in a whitebox system with an unsolvable IRQ issue. It seemed that no matter the configurations I tried, occasionally a hard to reproduce PSoD occurred due to an IRQ conflict. I ended up going with one-generation-past (12G is current as of this writing and at the time of my purchases) servers from Dell and haven’t had the same issue in a similar configuration. Also the Dell servers are way quieter at idle than my whitebox setup and they also have more granular fan control under load.
  • With Dell servers, you should assume the RAM configuration is the one you are going to use from now until the end of time. In my experience, Dells are finicky about the RAM modules being used and which slots are populated. Differences in size, speed, rank, and sometimes even model number of the RAM modules will create less than optimal running configurations. My suggestion is to either buy a used system with no RAM (more expensive in the end but you get exactly what you want in terms of speed and modules) to populate it yourself OR buy a Dell system with enough RAM to begin with (cheaper route.) This exacting nature may be the case with all [server-]motherboards, but my experience has been limited to Dells so far. My experience with Dells has been RAM modules were disabled because of the physical slot configuration was less-than-optimal in the BIOS’s opinion; also worth mentioning these Dells are the first time I have dealt with a system with 128GB of RAM.
  • Another thing to really consider is where and how you store your data. Take consideration to how you are locking your data/drives to a particular OS / RAID card / Software RAID / or possibly ZFS pool version. For example most hardware RAID is locked to that particular model of card and sometimes even to that particular version of firmware on the hardware RAID card. As for ZFS, there are four common versions in the wild: 15, 28, 5000, 34; the latter two are complete forks with no inter-operable ability.

 

My own brief experiences:

On a FreeNAS 0.7 / NAS4Free install which was particularly buggy at the time, especially with the software RAID5. I used this configuration for about a year.

  • Pros: It was the only thing available meeting my needs at the time, NAS and plug-ins. My data was a little safer than just having a single copy on a single drive.
  • Cons: Crashed a lot. Had many data-loss-scares. My drive were locked to the hardware configuration and software-raid5.

I used FreeNAS 8 for a brief time only to switch back to Solaris soon after. This was due to personal preference but I hold FreeNAS in the highest regard, along with the company that backs it, iXsystems.

I originally started with Solaris, a noble cause no doubt. I nearly immediately switched to the FreeNAS .7 / NAS4Free setup due to my intimidation with near exclusive configuration from command line. I then switched back to Solaris 11 a year later after briefly testing other platforms, only to realize what I was missing on Solaris. Solaris 11 is not for many people, but it now fits my needs wholly.

  • Pros: Very Stable, Native Data Encryption, built-in CIFS support, pool can be transferred to a fresh install without issues. SAN technologies built into the OS. Enterprise grade OS.
  • Cons: Solaris is a beast to wrestle depending on what you want to do with it. Native encrypted data is locked to Solaris 11+. Largely managed through command line interface.

 

More considerations:

Also, it’s worth noting that most HBAs and RAID cards are temperamental when it comes to harddrives timing out aka TLER/ERC-timeouts on high-recovertime-drives aka typical desktop harddrives. Sometimes I would have problems with my aging drives dropping from the array which would cause “ZFS UNAVAIL There are insufficient replicas for the pool to continue functioning.” which always stopped my heart. The solution is to use Enterprise drives, which are very costly; or to use NAS oriented drives which are closer to desktop drives both in price and hardware but with a firmware that is flashed to allow for different error handling, a handling method that resembles how enterprise drives handle errors. My personal preference for NAS drives are WD’s Reds but each manufacture has a NAS-variant drive now.

With btrfs now being considered mostly stable, it should definitely be a consideration as it looks to have a really cool feature set. At the time I was deciding my long term data solution btrfs was not nearly stable enough for me to have considered.

Consider buying long-warrantied (>=3 year) used drives on Amazon, yes used. My reasoning is, most drives either fail in the beginning week or toward the end of their warranty cycle. So I buy used effectively having other people test out the drives for me.  I have checked the SMART data on these used drives; the “Power On Hours Count” has typically been around 300 hours, a value I find acceptable. My personal hypothesis why people return drives with such low times, is that people “rent” hard drives from Amazon to transition to larger datasets on new storage systems and take the return fee hit as compensation. Then again I could of just got a batch from someone who had done that. Another advantage to buying used is, the potential for considerable savings depending on the number of drives you are buying. For me, sixty percent of the first batch (all new, 1TBs, 3 year warranty, various manufactures) of drives I bought failed. I was fortunate enough that most drives typically failed around seventy percent through their warranty. One drive did fail after one week. It typically took two weeks for the manufacture to return a refurbished drive with a renewed warranty. I ended up buying a spare drive to use as a cold spare during these exchange times. None of the second batch of drives (all used, 3TBs, 5 year warranties, WD Reds) have failed after two years of use. Not exactly apples to apples but an idea to consider. Google and BackBlaze have interesting hard drive failure data to look into as well.

 

Final Thoughts:

With bit rot, failed drives, and data loss being something I’ve personally experienced long ago before ZFS, I couldn’t imagine storing my data on anything but ZFS from now on. Getting the correct software and hardware configuration just right for your use and needs is tough. I hope this post can illuminate your way if you are still trying to achieve data storage nirvana.

Posted in Commentary, Technology | Tagged , , , , , , , , , , , , , , , | 1 Comment

Joined the IPv6 Bandwagon

My ISP has been assigning IPv6 /128 addresses for a while. My ISP has also just recently started offering /60 addresses. pfSense has been implementing better support for IPv6 as well in their snapshots.

Here is a speed test on IPv4 and IPv6:

2014-03-16 20_29_58-XFINITY Speed Test

Posted in IPv6 | Tagged | Leave a comment