What Every Beginning Linux Sysadmin Needs to Know

Many new sysadmins focus on process and technology issues, devouring books and guides.  That's awesome, and important, but it's only the tip of the iceberg.

Let's talk about attitude.  The difference between a good systems administrator and an outstanding systems administrator is attitude.

A sysadmin has to be extremely intelligent.  That goes without saying, pretty much.  If you confuse "affect" and "effect", "server" and "service", and "is it possible to" vs "please do this for me", you'll want to find another profession.  But intelligence alone is FAR from sufficient.  The five H's cover other attributes you'll need to have to be truly outstanding as a sysadmin.

humble

Humble is good.  It means you understand yourself.  You neither over-estimate your abilities nor do you have poor self-esteem.  You know your limits, your strengths and weaknesses, and you're sober-minded.

A humble person assumes the position of a learner during conversation.  You would never ever consider yourself "the smartest person in the room".  You're interested in understanding the other person's perspective, and you make the others in the conversation feel smart and competent.

No job that needs to be done is beneath you.  You're not focused on your title, position, or status relative to others in the organization - instead you're focused on the success of the organization.

You value "the little people" in the organization, and treat them as peers.

honest

An honest person delivers bad news first.  Exaggerating, bragging, or misrepresenting facts are TOTALLY foreign concepts to an honest person.

My employer values honesty extremely highly.  So highly, in fact, that if a co-worker told me, with a straight face, that they had gone over Niagara Falls in a wooden barrel, I would believe them.

Wrapped up with honesty is integrity.  If you make a promise, keep it - no matter how inconvenient, difficult, or personally expensive it is to do so.  

holistic

Don't think "inside the box" - think holistically.  Let's look at an example.  A server with 6 drive bays, currently using only 2 of them, is out of free drive space.  How much time to do you spend finding files/directories that can be deleted?  If the filesystem is using LVM, think instead about adding drives and expanding the filesystem.  Even without LVM, the cost of NOT adding additional drives could well exceed the cost of adding them.

Ticketing systems, used religiously, are awesome for documenting trouble as it happens.  One often-overlooked feature is the timestamping.  

Expensive alternatives to fixing recurring problems are hard to justify, but the timestamping makes it possible.  You can assess how much sysadmin time was spent on a problem by subtracting the start-time from the end-time.  Get an "average cost per hour" for sysadmin time in your organization, and multiply.  You can extrapolate out an average cost per month of not solving the problem in a more permanent way.

Hopefully you can identify recurring problems to management somewhere below the "50%" mark.   You'll be able to say:  "This is an ongoing problem.  So far it has cost us X.  By it will have cost us Y, which is the cost of solving the problem.    If we spend Y to fix the problem now, then by we'll have broken even, and we'll be SAVING money every month after that."  This is exactly what your boss needs to be able to justify the expense up the ladder.  

The reason this H is so important is that quite often, even though there may be people in the organization charged with implementing these sorts of cost-saving measures, they don't have the necessary information or holistic perspective into what YOU are doing, to be able to identify what needs to be done for particular situations.

hungry

Don't dwell on your past success.  Set your standards higher and higher.  Have an insatiable appetite for new information.  Be an active listener in meetings - ask questions, take notes.  Follow up on open action items.  Put shoulder to everything you do.  You're being paid for 8 hours per day... deliver at LEAST 8 hours worth of value.  And if that took you 6 hours... deliver just as hard for the next two, because you've set your bar too low.  

helpful

Having a helpful attitude is crucial.  ALWAYS be helpful.  ALWAYS be part of "the solution", not part of "the problem".  ALWAYS ensure that everyone in the loop is helped sufficiently.

It's useful here to have the ability to say no using the letters "y", "e", and "s".

For example, say a customer, or manager, or end-user, wants to solve a problem in a way which simply won't work.  YOU know it won't work, all of your co-workers know it won't work, and the whole world except that one person knows... it won't work.

How to say "no"?  Find three alternatives that WILL work, and present those.

Depending on the dynamic, you might take the following stance.  "The approach you're recommending isn't viable.  We can go into why, if necessary, but it might be more effective to look at alternatives.  Here are three good ones: ..."

So, instead of showing them why what they're asking for won't work (and making them feel less competent in the process) you're putting them in a position where they can CHOOSE a workable solution from the options.

They'll feel better about that, and the problem will get solved in a workable manner.

Another way to be helpful is to share not only data, but a quick statement of how you generated the data.  Consider the tremendous difference:

Dear Customer,

You'll need to clear some drive space - your drives are full.  Here is a list of files >100MB on your server for your review, and possible deletion, compression, or relocation to another device.

9999MB /really/big/file
1234MB /other/big/file
123MB /somewhat/big/file

Regards,
-your helpful sysadmin

...compared to...

Dear Customer,

You'll need to clear some drive space - your drives are full.  Here is a list of files >100MB on your server for your review, and possible deletion, compression, or relocation to another device.

[root ~]% find / -xdev -type f -size +102400k -exec stat -c'%s|%n' {} \; | awk -F\| '{ print $1/1024/1024 "MB " ": " $2 }' | sort -nr

9999MB /really/big/file
1234MB /other/big/file
123MB /somewhat/big/file

Regards,
-your helpful sysadmin

In the first version, you've given the customer some information to work with.  However, they have no idea how you got that information.  You've left them powerless to search, for example, for files >50MB as well.

In the second version, you've provided just enough information that if they're at all capable on the commandline, they'll be able run another scan themselves.

THAT is helpful.  Forcing them to come back to you for another listing of files, this time >50MB, might on the surface seem like you're helping them more - but what you're really doing is forcing them to be dependent upon you.

     ~~*~~

So... to recap:  humble, honest, holistic, hungry, and helpful.  Integrate these 5 H's into your very being... and you're well on your way to becoming an outstanding systems administrator.