I’ve been a Systems Administrator for going two decades now, and I’ve learned a thing or two about how dumb I am. This blog is mostly errata that I think might help my users: if I googled it, they might eventually too, and so I want to be able to point them to something that will work in our environment.

Having said that, stuff that I blog about here should work for you too. If you have questions or problems, just ask!

In my computer career I’ve worked at Johns Hopkins University Computer Science, the Enoch Pratt Free Library, the State of Hawaii Clean Air Branch & various NASA contracting firms.

I’m currently the senior UNIX & Mac SysAdmin at the Old Dominion University Center for Coastal Physical Oceanography. My scientists study the effects of climate change on sea level rise, among other things.

I have the best systems job in the world, because all of my users are power users. You should be so lucky. If you find a job like mine, take it! Added bonus: working at my alma mater.

A few points of troubleshooting & meta advice for junior systems administrators in general:

• Document everything you do. There are no exceptions. You will forget the specifics of your localized configurations. Documentation is non-negotiable. You & your team require reliable documentation.
• There are no stupid questions. If someone asks you a technical question that you think should be obvious to them, don’t insult them, don’t tell them to RTFM, instead just educate them (politely). They’re just grepping your brain. It’s a compliment.
• Keep the machines up to date. Set them to apply security patches (and preferably all patches) automatically.
• It’s probably just not in your $PATH.
• If it’s in your $PATH, it’s probably a permissions error.
• If it isn’t a permissions error, and you tried it with an absolute path, make sure you’re executing it as the proper user.
• Don’t su -. Just don’t. sudo instead.
• Since all senior admins are going to ignore that last one, at least don’t enable color in your root .tcshrc or .bashrc or whatever. Then remember: If it’s white, make sure it’s right. Or, if you use a terminal with black text and a white background, If it’s black, you can’t go back.
• There will always be someone smarter & more experienced than you. Those people are a resource. Use that resource if you can. Likewise, you may know more than the person next to you. Be their resource.
• You can fix it.  Step number one is don’t panic. If you can’t fix it, maybe somebody will help you (see above). If nobody will help you, then maybe you could try another solution. If you can’t try another solution, you probably have another computer. In any case, you can solve the problem.
• If you’re in systems, you need to work on your customer service. We all do.
• Always ask the user what she’s actually trying to do. Sometimes we all get too annoyed with details. There may be a much simpler, faster way to do something that your users don’t know about, or even services you have that are poorly advertised.
• Begin to accept that you aren’t just root, you’re also a user. They are you & you are them.
• Never argue or evangelize about your chosen $EDITOR or your beloved $SHELL. Give the user the tools he wants to do his job. And smile about it.
• Everyone has something to teach you. Therefore everyone is your teacher. Respect them. Soft skills are as valuable as technical ones.
• Disks that are full will cause you problems. They will be weird problems. If weird shit happens, $ df -h.
• Make sure you lsof -i to see what the system is opening to the world, then run nmap on your systems from the outside.
• If you don’t know how to harden a linux machine, try bastille.
• denyhosts. For days.
• Every Single Time:
  • whoami
  • sudo cp aFileImAboutToEdit aFileImAboutToEdit.date
• If you call yourself a systems administrator, you better make sure you can restore from your backups.
• You should hold yourself to the highest ethical standards. Consider posting the Systems Administrator’s Code of Ethics by your door, and encourage your users to hold you to it.
• Never take a service down without notifying your users.
• Remember that users come to rely upon your services. If you offer a service, then take it away, they’ll become understandably upset. So don’t. Or, if you have to, offer an upgraded version, and understand that change is a process.
• Your job is to empower excellence. That is your goal. You maintain the tools your users need to succeed. Satisfaction guaranteed is the minimum level of service you should shoot for.
• Never change anything on Friday after noon.