System administrator leveling matrix


1	Compiled by sistemshik		http://nixsrv.com
2	Credit for the idea and and several entries		http://www.indiangeek.net/wp-content/uploads/Programmer%20competency%20matrix.htm
3	Sage Level Job Description		http://static.sage.org/field/jobs-descriptions.html
4	One cannot reach level cap in all areas, so this matrix should be viewed as development branches. A good sysadmin must be an expert in one chosen area, so he usually will know other areas totally not as deeply. Attention! This leveling matrix has an obvious bias towards Unix-like systems, but entries often can be directly interpreted towards Windows platform. For example, in the row «System configuration control» entry «Centralized configuration management using tools like chef, puppet, cfengine» may be interpreted as «System confirutations are managed with SCCM».
5	Domain knowledge
6		level 1	level 20	level 50	level 80 (level cap)	comments
7	Operating systems/ platforms	Almost no understanding yet "Knows" that <your distro name here> is an OS Is able to: Install an OS to his computer	Basic understanding of operating systems RHCT, RHCSA MCSA, MCITP: Server Administrator Know: What an OS is What a kernel is Is able to: Install an OS to server Add, remove users Install software using package manager of choice	Good understanding of what an OS is RHCE MCSE, MCITP: Enterprise Administrator Knows: difference between kernel and user mode what are system call and how they are used OS process model OS security model, limits, how to use access use control lists OS user model OS boot process, what happens when OS starts, stops, what runlevels or similar concepts in os of choise are OS networking model Is able to: read and sometimes understand source code of OS of choice use strace, ltrace, ktrace or similar for toubleshooting make packages from sources using pakage format of OS of choice set up and use console redirection troubleshout networking problems using tools available limit proccesses using mandatory access control	Expert understanding of an OS of choice RHCA MCA Knows: Understands the whole process execution: hardware (CPU + memory + cache + interrupts + microcode), assembly code, static and dynamic linking, interpreting, heap, memory allocation... OS kernel options, process schedulers, IO schedulers, memory allocation Is able to: Build any package from source Patch package for compatibility Use gdb to fix broken software	Working with someone who has a superstar ranking would be an unbelievable piece of luck!
8	Computer architecture	Almost no understanding yet Know how computer peripherals connect to the computer Is able to connect monitor, keyboard and printer to the computer	Basic understanding of Von Neuman architecture, different system components like CPU, memory, hdd and how this components inter-operate	Good overall knowledge of computer architecture Knows processor types, motherboard types, memory types (ECC, FB-, ...), how this components are clocked, how cache works, how memory architecture works (NUMA, UMA), what expansions slots there are, what parameters in UIFI/BIOS mean, etc.	Expert knowledge of contemporary computer architecture and how to design and implement, using available technology, complex clustering solutions
9	File Systems	Knows: File system is that thing which keeps files on it Is able to: write a CD, format a USB-dongle transfer >4GB files using USB-dongle	Basic filesystem understanding Knows: What partition table is What journal is Is able to: Patition disks Check filsystem Install bootloader	Good understanding of filesystems Knows: What i-node is What filesystem journal is What files there are (block, char...) stat command output How ACL works What umask is What copy-on-write is and why it's good What FHS is and why it's needed What parition alignment is What VFS is Is able to: Partition disks properly depending on usage scenario Use LVM or similar partition management tool Modify filesystems with tune2fs Resize filesystems Use debugfs	Expert knowledge of filesystems Knows: Read and understood File System Forensic Analysis and/or filesystem code of the OS of choice Is able to: Find and read file using HEX-editor only Repair badly damaged filesystem
10	Storage	Almost no understanding yet	Basic storage systems understanding Knows: There are many different types of storage out there Is able to: Use RAID-1 for redundancy	Good storage systems understanding Knows: All RAID levels Why one almost never should use RAID-5 What storages there are: NAS,SAN,DAS What iSCSI is What contemporary hdd interfaces are Is able to: Employ a decent storage model for the tasks	Expert storage systems understanding Is able to design and implement storage system for many highly-loaded systems, with geographically distributed replication
11	Networking	Almost no understanding yes Knows: What an IP-address is Is able to: Use ping utility	Basic to mediocre networking understanding, CCNA Knows: What is OSI-model What is MAC-address What is a switch, how it differs from a hub What is a subnet What is DNS What is DHCP Is able to: Configure networking on the workstations using DHCP or static networking Confgure static routing on a workstation	Good networking understanding, CCNP Knows: OSI model What OSI level means in network gear specifiations What is network mask What is classless networking TCP/IP stack, which headers are add on which level VPN and VLANs and how to use them Routing protocols and how to use them Wireless networking and how to use it and secure it Is able to: Design and implement company network Set up VPN for remote access Use telnet, netcat, tcpdump for troubleshooting Use telnet for checking www, mail servers, etc.	Expert networking understating, CCIE Is able to design and implement geographically distributed highly-loaded network with thousands of users. Is and expert is one or several of the following areas: http://www.cisco.com/web/learning/le3/ccie/track_comparison/index.html
12	Virtualization	Almost no understanding yet Maybe heard the word "Cloud"	Basic virtualizatoin understanding Knows: What you can get a Virtualbox or VMWare and run an os inside an os Is able to: Install an OS in Virtualbox or VMWare workstation	Good virtualization uderstanding Knows: Theory, how virtualization works What difference there is between full virtualization and paravirtualization What binary translation is What Ring -1 is What AMV-V/Intel-VT is What SR-IOV is What live-migration is What storage system to use and why partitin aligment in virtualization is especially important Is able to: Use virtualization in production using Citrix XEN server, KVM, VMware ESXi, etc. Migrate physical servers to virtual machines and vice versa	Expert virtualization understanding Has expert-level understing of virtualization system of choice. Is able to design and implement higly-avaliable, georaphically distributed solution.
13	Security	Almost no understanding	Basic security understanding Maybe CCSA, SGFWA Knows: What permissions are Why one shouldn't install software from untrusted sources Why one shouldn't work on the system using administrator privileges Is able to: Run antivirtus solution	Good security understanding Comptia Security+, SSCP, CCNP Security Knows: What is information security, and that it is protecting information from internal and external threats, which are: -- illegitimate access and usage -- disclosure -- loss Main goals of information security, which are: -- integrity -- confidentiality -- availability What is the RFC2196 Site Security Handbook What is CVE Is able to: Securely configure a system Implement least needed priviligies security policy Update the systems using security patches Track security vulnerabilities for software for which he is responsible using security advisory mailining lists and online security databases	Expert security understanding CISSP+CISA+CISM, CCIE Security Is able to design and implement company-wide security policy including protection from techical threats and protection from malicious people as well
14	System administration
15		level 1	level 20	level 50	level 80 (level cap)
16	System configuration control	cp conf conf.bak	VCS, RCS basics	All configuration files are managed through SVN/GIT/etc.	Centralized configuration management using tools like chef, puppet, cfengine
17	OS installation automation	Installs OS manually	Is able to create unattended install image	Uses automated installation over network	All OS images are configured via configuration management tools and deployed automatically
18	Monitoring	Thinks that monitoring is then users are screaming "It doesn't work!"	Uses basic monitoring tools like ping monitoring	Uses monitoring server with agents on the monitored hosts	Uses monitoring server with agents on monitored hosts and in-depth monitoring scripts for automated checking of ability to perform usage scenario, for example logging into the system, executing queries on the databases, etc.
19	Best practices	What's that?	Doesn't work under root	Good understanding of best practices, knows where to get them and follows them for the platform of choice	Writes them
20	Number of supported servers	Zero	1--10	10--100	Many hundreds
21	Problem decomposition	Reinstall is the only way	Is able to basic problem decomposition, check hardware, software, network...	Is proficient with problem decomposition, understands that contemporary systems consist of many components, breaks up the problem and troubleshoots it piece by piece. Is proficient with log analysis, OSI-model for networking troubleshooting, etc.	Expert understanding of different levels on which the problem may lie. Creates a model for each problem and uses systemic methods to troubleshoot it
22	System decomposition	Unable to think of more than one computer	Able to break up problem space and design solution as long as it is within the same platform/technology	Able to design systems that span multiple technologies/platforms.	Able to visualize and design complex systems with multiple product lines and integrations with external systems. Also should be able to design operations support systems like monitoring, reporting, fail overs etc.
23	Communicability	Cannot express thoughts/ideas to peers. Poor spelling and grammar.	Peers can understand what is being said. Good spelling and grammar.	Is able to effectively communicate with peers	Able to understand and communicate thoughts/design/ideas/specs in a unambiguous manner and adjusts communication as per the context	This is an often under rated but very critical criteria for judging a system administrator. With the increase in outsourcing of programming tasks to places where English is not the native tongue this issue has become more prominent. I know of several projects that failed because the programmers could not understand what the intent of the communication was.
24	Configuration files organization	Absent	Configuration files are grouped logically	Configuration files are grouped logically and referenced to related configuration files. Big configuration files are split in several small using includes.	Every configuration file has a summary, is well commented, consistent white space usage. The file should look beautiful.
25	Services data organization	Generally, just a mess.	One directory per service	Generally, one partition per service	One partition per service, all names are consistent between all servers
26	Configuration files readability	No comments, just a mess	Simple comments	Comments, configuration parameters are grouped logically	Additionally has explanations as to why parameters are chosen as such
27	Ability to read manuals	Almost absent, tries first google result	Is able to read man, does this often	Knows many solutions from experience	Is able to use application source code as documentation
28	Scripts	Unable to script	Simple bash scripts	Perl/Python/Ruby/VBScript/Powershell	Writes and published good scripts which can be reused
29	Specification	Implements solution according to specification	Come up with questions regarding missed cases in the spec	Understand complete picture and come up with entire areas that need to be speced	Able to suggest better alternatives and flows to given requirements based on experience
30	Databases	Thinks that Excel is a database	Knows basic database concepts, normalization, ACID, transactions and can write simple selects	Can do basic database administration, performance optimization, index optimization, write advanced select queries, able to replace cursor usage with relational sql, understands how data is stored internally, understands how indexes are stored internally, understands how databases can be mirrored, replicated etc. Understands how the two phase commit works.	Good understanding of database administration, is able to design and implement geographically distributed highly available redundant system
31	Experience
32		level 1	level 20	level 50	level 80 (level cap)
33	Platforms with professional experience	1	2--3	4--5	6+
34	Years of professional experience	1	2--5	6--9	10+
35	Domain knowledge	No knowledge of the domain	Has worked on at least one product in the domain.	Has worked on multiple products in the same domain.	Domain expert. Has designed and implemented several products/solutions in the domain. Well versed with standard terms, protocols used in the domain.
36	Knowledge
37		level 1	level 20	level 50	level 80 (level cap)
38	Instrument knowledge	Almost none	Basic CLI skills	Advanced CLI skills, is able to use filters	Has actually written tools and scripts, added bonus if they've been published.
39	OS source code knowledge	Has never looked at the codebase	Basic knowledge of the code layout and how to build the system	Good working knowledge of code base, has implemented several bug fixes and maybe some small features.	Has implemented multiple big features in the codebase and can easily visualize the changes required for most features or bug fixes.
40	Knowledge of new and emerging technologies	Has not heard of the upcoming technologies	Has heard of upcoming technologies in the field	Has downloaded the alpha preview/CTP/beta and read some articles/manuals	Has played with the previews and has actually built something with it and as a bonus shared that with everyone else
41	Platform internals	Zero knowledge of platform internals	Has basic knowledge of how the platform works internally	Deep knowledge of platform internals and can visualize how the platform takes the program and converts it into executable code.	Has written tools to enhance or provide information on platform internals. For e.g. disassemblers, decompilers, debuggers etc.
42	Books	Google HOWTO, for Dummies series	O'Reilly Network Administration, other O'Reilly books UNIX and Linux System Administration Handbook" "Modern Operating Systems" "Computer Networks"	O'Reilly Network Administration, other O'Reilly books UNIX and Linux System Administration Handbook" Jeffrey Friedl's Mastering Regular Expressions	Fundamental books like "The Practice of System and Network Administration", "Windows Internals"; Linux source code, "Lions' Commentary on UNIX 6th Edition, with Source Code"
43	Blogs	Has heard of them but never got the time.	Reads tech/programming/software engineering blogs and listens to podcasts regularly.	Maintains a link blog with some collection of useful articles and tools that he/she has collected	Maintains a blog in which personal insights and thoughts on system administration are shared