|Home » Industry Watch (» The Technological » Hall of Monkeys » Heroes Banquet)
Define 'journalist'. Try. The story of San Francisco's strange IT department.
The advent of the blogosphere has caused many to wonder if mainstream journalists have any relevance any longer. Their coverage of the Terry Childs case shows convincingly that they don't.
It's one thing if bloggers copy info without attribution. It's quite another if they do so without corroborating information. After all, they're just bloggers.
But it's another thing entirely when the mainstream journalists do so - and do so without bothering to corroborate information in the slightest.
Bloggers do it better.
The crap spread in the media immediately Terry Childs was arrested was bizarre. About the only story not concocted by those so-called journalists was that Terry Childs was sent by aliens to sabotage critical infrastructure to help a planetary invasion. But that story could still turn up somewhere, given the ambition and integrity of today's so-called journalists.
Paul Venezia is one journalist who didn't take the cheap route. He's different from mainstream journalists in one very important regard: chronology notwithstanding, he's at heart an admin first and a writer second.
For that's another thing about mainstream journalism: the writers aren't required to have any in-depth knowledge about their subject matter (and rarely do). They only need be journalists. (And there's the #1 reason you never trust a professional pundit.)
Paul Venezia writes about things that would leave ordinary journalists tongue-tied. Paul's on the inside of the profession; ordinary journalists sit on the outside, have no clue what's going on, and just 'wing it' as best they can (which often isn't very good at all). Their bosses know less than they do, so they keep getting those fat paychecks.
Every IT professional knows of the 'egregious fucktard'. It can be a systems analyst proud to tell everyone his work is completely abstract and he can't write a single line of BASIC, it can be a weasel in a pinstripe suit dropping cool technical jargon he doesn't understand for other weasels who don't understand it either but wouldn't dare admit it, it can be an up and coming management lioness who has no clue how deep the technology really goes and who consequently trivialises the work others have done for half a century, it can be nothing more than your run of the mill outsider ego-tripper.
The scorn for these people is universal amongst people on the inside.
And there are good (and instinctive) reasons for this scorn as well. Those fucktards invariably screw things up, they're very rude, they sow dissension, and they are so lacking in clues that they're regarded as anathema.
IT pros aren't ladder climbers. They gravitate to their profession because they don't want that horseshit. They want to be left alone to do their jobs. They're fascinated by their jobs. They live in an environment without ego - their field of study is the only reward they want. They want to be removed from the rat race they see all around them. Their field of study is their world and they want to be left alone. And guess what? You can't make IT work without them. So don't even think about it, asshole.
Eric Raymond's had quite a bit online about this - rather quite a bit about how the clueless should approach the high priests and why. It's no coincidence that IT pros used to have this status. Really. At least until the PC revolution came and people saw 'Intel Inside' ads at football matches.
The IT high priests were never interested in the 'status' per se. But the perceived status meant the clueless gave them wide berth. And that's what they wanted (and needed).
Things are different today in the PC era. Not because ordinary people have direct access to computers but because management pinstripe suits are also included in that 'ordinary people' demographic. Suddenly the same fat cats who always thought they knew everything (and are great at negotiating golden parachutes) think they know everything about IT as well. They know how to make a spreadsheet in Excel.
Management will never trust IT pros. It's easy for IT pros to see through management (and they spend considerable time laughing at them behind their backs) but management - not always the sharpest tools in the shed - simply can't figure IT pros out. They simply can't understand people who aren't conniving and greedy like they are.
Management can't control IT, can't control IT pros, and that really pisses them off. It makes them feel insecure. And the management referred to here isn't general corporate management - it's specific IT management. Even those managers are hopelessly clueless. And that's a huge problem.
Why does Spotify work so well? Because Spotify is a flat organisation with effectively no layer of management whatsoever. Daniel Ek is just another employee. Often not even in the office. As his 'job' - a job like any other job there - is to go out and negotiate new contracts and spread the use of Spotify.
Why didn't Steve Jobs like John Sculley? Same difference. Jobs might have been the cofounder of Apple but he didn't sit around the office looking at his stock portfolio - he worked.
It's thanks to the investigative journalism and background of Paul Venezia that people know something about Terry Childs today. Not a single other journalist bothered to look.
Looking at Terry Childs
Terry Childs had a rough and tumble childhood. His mother was a junkie. His younger brother Brad describes her as a 'loser - and she still is'. Terry had some brushes with the law back in those days. And he spent four years as the honoured guest of a correctional facility - something mainstream journalists salivated over.
But Terry went straight - and became for the most part flawless to a 't'. Even his brother's complained that the Terry of today is 'too correct'.
Terry Childs is today one of about 12,000 people worldwide who have the CCIE certificate from Cisco. This is a hard goal to reach - only about 10% of all applicants ever pass the exam. Terry's worked very hard.
Terry build the FibreWAN network for San Francisco. He didn't design the network - he built it. He did all the purchasing, implementation, installation, and so forth. The network was his baby. He even got a copyright for it as a 'work of art'.
[Note: all stories agree he got the copyright but some say the city somehow got it rescinded - whatever that means.]
Things hadn't been good in Terry's department for some time. The IT work force was cut from 350 staff to 150. Remaining staff were worked to the bone. Terry himself worked overtime until it was coming out of his ears but received no compensation for it - only holiday hours which he like most of the others had no chance of ever redeeming.
The upper management of Terry's department - the DTIS - were a group of incompetent fools. Terry had petitioned for years for an official security protocol - and written several proposals for it - but management ignored him. They finally brought in a security supervisor from outside who proceeded to walk all over Terry. That didn't go down well.
The story is Terry and this woman had a lively argument in his office and Terry pulled out his cellphone and started taking pictures of her. And no one's understood why or even suggested an explanation. The lady may have been misbehaving and Terry wanted digital evidence of her behaviour.
Terry heard shortly thereafter that DTIS management were going to 'reorganise' and Terry was to be given a 'horizontal promotion'. Here's where the fun starts: Terry has no 'rucksack' - there's no one else in the organisation who has a clue how the network works. Terry's built and protected this network with his blood, sweat, and tears - and the people who want to take over the network are fools. And they never backed him up.
What to do?
The smug outsiders - the besserwissers, the rats at Slashdot, the clueless - all say Terry should have turned over access to the network to anyone at all if he was in fact being relieved of his duties. And that does ring true.
But to whom? There's the rub. For a number of interesting things can happen here, as Terry and any IT pro know full well.
- The suits at DTIS are demonstrably ignorant. And arrogant. They can't possibly handle the network if given access. Bad things will happen (and bad things did).
- Suits are fucking up all the time but they're conniving enough to weasel their way out of any situation. They'd still blame their own failures on Terry Childs and Terry knew it.
Terry's one chance was to find a single person who had a reasonable chance to take over without screwing up. There was nobody.
And so Terry Childs was arrested - on trumped up charges, charges that were written by suits, by outsiders. Most of the charges listed in that first warrant as supposedly 'dangerous' behaviour on Terry's part are 'standard operating procedure' for network engineers - it's just that the people responsible for that document couldn't find the inside of a ferrite core CPU in broad daylight.
Jason Chilton 'Thinks'
Jason Chilton was juror #4 in the Terry Childs trial. Chilton is also a CCIE. He spent five months away from his job for the trial. No explanation has been given how Chilton's company could survive without his services for so long. Perhaps matters aren't as dysfunctional there - perhaps management aren't working staff to the bone and don't let key staff get isolated as DTIS did.
IT pros who've read Bob McMillan's interview with Chilton aren't impressed - but then again McMillan is not an insider. Here's the gist of the interview.
- Why is Childs guilty? Chilton says they were given specific directives as to the letter of the law and how it should be interpreted. They were also given definitions of technical terms they had to abide by, no matter how things pan out in 'reality'.
Their first question, says Chilton, was to determine if Childs knew he caused a 'denial of service'. Note that this completely circumvents the question of whether there actually was a denial of service: the network ran fine, thanks to Terry, until mayor Gavin turned the passwords over to DTIS management.
Note as well that anyone with technical acumen could have bypassed the password requirement with physical access - something that calls the competence of Chilton into question. The passwords were needed because the people needing them were incompetent.
And the DTIS did in fact proceed to completely screw up - hundreds of classified documents were released into the public domain. This was the first 'denial of service'. (Yet no one from DTIS will be charged with a crime. Of course not.)
- The second aspect. According to Chilton, this was 'denial to an authorised user'. This too is silly - there was no one authorised. Certainly no one competent. You don't work five years on the project of your life only to let greedo pinstripes yank it away and ruin it.
But Chilton says the court didn't even explain for them what an 'authorised user' was - so they had to work that out for themselves.
'It was really hard for us to get through that part', admitted Chilton. They had a sketchy story of a conference call and they were expected to piece together a plausible way things should have played out - and on that basis issue a verdict?
'The city didn't have any procedures. There was no policy that was formally adopted that people were supposed to follow. It was this amorphous thing', said Chilton.
- Unsubstantiated conjectures. So the jury looked instead at an event earlier in June of that year. Terry's manager requested accounts be created to afford access to certain routers and switches - and Terry complied.
Here's where it starts to border on the ridiculous: the jury noted that one of the accounts was for one Richard Robinson, generally regarded as one of the boobs in DTIS management.
So the jury's conclusion? That Terry Childs claiming there was no one competent to run the entire San Francisco FibreWAN network was just a lame excuse.
You read it here, folks.
- So what transpired on that conference call? Again: this is pure conjecture but Chilton of course has a qualified opinion.
'I think he went into that meeting probably thinking he was being fired. Definitely he knew that there were employment changes coming', says juror #4 Chilton.
Chilton also claims to somehow know Terry Childs had been given advance notice from a friend that he was going to be reassigned.
Chilton continues to speak off the cuff with McMillan, speculating on how Terry Childs felt at the time and even speculating about the character of Terry's boss whom he didn't know.
Chilton's ubiquitous qualifier 'I think' is all over his interview with McMillan - eleven instances all told. That's a lot of 'I think' for such an important verdict.
Chilton also speculates on Terry's motives - assuming in his amateur wisdom that it's already been established a crime has been committed.
Jason Chilton may have been a CCIE - although how good a CCIE is not known - but what about the other eleven jurors? What can possibly suggest this was a 'trial by peers'? Who of the eleven other jurors had the faintest clue?
What would the verdict have been if all twelve jurors were CCIEs?
Dana Hom Winces
Dana Hom has been behind Terry Childs all the way. He's turned up at the courthouse in support of Childs and spoken out repeatedly on the affair.
Hom is Childs' former boss at the DTIS - and the one who hired Childs on. He's had only good things to say about Childs - and less than flattering things to say about DTIS management. (He left his position after four years because of the corruption.)
'The mayor, along with the executive leadership at DTIS, should open their eyes and see that the information technology department is very dysfunctional. I challenge mayor Newsom to do a ground troops evaluation by speaking with the line level employees as to what is really going on. I would eat my hat if mayor Newsom did not find the MAJORITY of IT professionals that would like to see the senior management of DTIS fired and replaced', wrote Hom.
'We all know that Terry Childs is behind bars on trumped up charges based an an overreaction by DTIS senior executives who then got the SFPD involved and later the DA's office who jumped all over it like it was the last meal on earth.'
Hom published a longer piece at Blogger News Network.
When I headed up the division that Terry worked at the City & County of San Francisco, Terry did an excellent job, loved every minute of leading his network engineers. Terry is a dedicated principal network engineer, with an equivalent Cisco CCIE certification. He is not a system administrator, server administrator, or any of the other labels that the media has given him.
I was the Chief Operations Officer for DTIS (Dept. of Telecommunications Information Services) from 2000 to 2004.
I do not believe Terry should be in jail at all. Under my leadership at DTIS, I have had far worse employees who did not do any extended jail time and certainly did not get a bail of that nature while their cases were pending. I believe Terry is being subjected to the horrible politics of San Francisco and incompetent senior executive leadership of DTIS. Let me point this out as senior executive that actually understands technology unlike the current senior leadership at DTIS:
Some journalists are writing stories such as 'Out of Control! Felony Charges for System Crasher' from examiner.com and other journalist articles calling Terry a 'hacker'. Let's set the record straight, and I'm sure that most technology professionals would agree:
- First of all, what Terry allegedly did was change passwords on the city's network infrastructure. That has yet to be heard in court as there are always 2 sides to the story.
- There is no 'system crash', the reality is that all servers are most likely running fine.
- There is no 'denial of service' as some journalists are accusing him of doing. Denial of service in IT terms is used to define repeated attacks using data that can overwhelm a corporate network so that end users are affected, or slowing or stopping the network which can create a denial of service to end users. That simply is not the case here.
- Hacking is breaking and decrypting passwords. Changing passwords should not be referred to as hacking.
- Realistically, unless the city network engineers need to do maintenance on the network, all systems are running fine. Any new network changes will need to be rescheduled until all of the passwords have been reset (and network infrastructure configurations restored).
- A well executed plan to restore the configurations and reset the passwords should cause minimal disruption to the various departments and operations unless the configurations have not been backed up recently.
If item #6 is the case that the network infrastructure which consists of Cisco core backbone equipment, edge routers, fiber optic provisioning equipment, SMNP managed switches, Pix firewalls, etc has not been backed up, then the leadership at DTIS should be held accountable. What CIO, deputy director, and other IT leadership would not have the technical background and vision to ensure that the IT infrastructure is secure?
Above all, even with safeguards, checks and balances, and a solid backup plan, does anyone have the common management sense to build a consensus among the IT staff to encourage cooperation, team building, communications, and esprit de corps so you don't end up with disgruntled IT staff?
The mayor, along with the executive leadership at DTIS, should open their eyes and see that the information technology department is very dysfunctional. I challenge mayor Newsom to do a ground troops evaluation by speaking with the line level employees as to what is really going on. I would eat my hat if mayor Newsom did not find the MAJORITY of IT professionals that would like to see the senior management of DTIS fired and replaced.
By the way, I'm not interested in the job of CIO for SF.
Dana L Hom
Former Chief Operations Officer & Director of DTIS, City and County of San Francisco
'testsf' concurs in the same thread.
I concur with Dana that a ground troops evaluation is the best thing that could happen to all city IT groups. The devil's in the details and when first and second level managers are hired that haven't spent years in the trenches (though they may be good talkers) politics takes priority over quality and dramatic waste ensues. Upper management must come into direct contact with workers on a regular basis to understand what's really going on.
I could save my organisation an easy $5 million per year if only high level managers would set up a meeting and ask me how. Taking the initiative and going over my boss's head to present my ideas to upper management directly would be career/financial suicide. Climb down from your ivory tower and come talk to me.
A fool calling himself 'SFinformed' attacks Hom and is put quickly back under his rock by 'manictiger'. 'SFinformed':
Dana Hom is full of C**p [sic]. He was fired/laid off in 2004 when people finally found out he talked a good game and nothing else. His computer knowledge came from reading trade magazines. He purchased equipment and installed IT policies that were useless.
SFinformed, hah, that's an ironic screen name. Here's how things happened here on earth.
Dana graduated with a four year degree from San Diego State majoring in computer science. Worked his way up working with an affiliate of the SF Chronicle reaching CIO status after ten years. The company was sold so he moved on finding a position as COO of DTIS. He finally quit on his terms because of the corrupt politics. When he left everything went to hell and they wanted him back. He declined and now owns a computer repair shop that would put the geek squad at Best Buy to shame.
'Rich Robinson' - who probably isn't the notorious Richard Robinson from the DTIS - adds the following.
'Terry's solution was to disable all of the administrative passwords for the complex, effectively leaving the city with no control over their information.'
This is factually incorrect. It was Terry's job to administer the network routers and had been for years. The city set many minimum requirements for the position including years of experience and certification. The city found that Terry met the qualifications and selected Terry for the job.
Part of the job of administering the routers is to set passwords for the routers. This was and had been Terry's job for years. In addition, it is standard security practice to routinely change passwords so that in the event a password had been compromised, security would periodically be re-established. This was also part of Terry's job and had been for years.
Terry is not charged with disabling anyone's passwords. Neither is he charged with setting the passwords in the first place. He is charged with failing to disclose the password. Oddly, the law cited as having been violated (California Penal Code subsection 502) has nothing to do with failing to disclose passwords but rather applies to 'unauthorised access to computers, computer systems, and computer data'.
He most definitely did have and has had authorisation to set passwords on the routers for years.
Further, the passwords in question were not to servers where information would be stored but rather to network routers which control the flow of traffic on the network. Thus the city was not left with 'no control over their information'.
The network itself continued to function normally throughout even while Terry was held in jail and up until Terry was suspended from work on 9 July 2008 the city still had the same one and only employee available to administer the routers as it had had for years - Terry Childs. Thus no 'denial of service' existed until he was suspended and they then claimed they did not have the access which they also admitted that they knew they had not had for some time.
The network routers include a feature which allows them to be reset. The city could at any time have reset the routers and reconfigured them to regain complete administrative control of the network routers.
Managers within the Department of Telecommunications and Information Services (who were purportedly attempting to fire Terry) have stated that they did not know the configuration of their own routers.
Thus the passwords or the lack of the passwords are not the prohibiting factor in this case but rather the issue is the lack of knowledge and ability within the DTIS department to be able to reconfigure the routers -without- Terry Childs - the one person whom they have been paying for years to do so.
The outrage - and fear - felt by IT pros has been deafening.
Terry is a good guy. It is outrageous how he is being treated. Paul Venezia's article is right on.
I knew this guy. He was competent, thorough, and dedicated. The city politics would drive anyone to desperate measures and not sharing a password is NOT a crime. He has not robbed anything, killed anyone. He built the network and simply set out to protect it. Shame on those people for making a scapegoat out of an excellent engineer.
I don't normally write into these comment sections. But I have to make an exception in this case. I have known Terry for many years. I was the person who opened the IT doors for him and gave him a start. I did it because Terry was a confident go-getter person. I helped him obtain some of his Cisco certifications because he was willing to study and go the extra step required to master the IT and WAN infrastructures. This guy is NOT a maniacal, wacked out individual. He is a fun person to be around, he is very devoted in what he does and yes, he has a sense of ownership that any career minded IT person should have. I don't understand why the city is doing this to him, or how they can justify a $5 million dollar bond! This sounds more like Terry Childs uncovered something on a politician. And now the politician is attempting to cover it up at Terry's expense.
Thank you, I completely agree that there was something much deeper in this story than a 'rogue employee'. I read the original articles and knew that it wasn't the whole story. What Venezia and you have discovered sounds much more likely than the mainstream media's account.
This whole story is so sickening and will likely put many government politicians in panic mode about their IT departments. Was the guy uncooperative? Very likely but like any IT position it sounds like he was put into a position to keep the world running, did everything he could, and maybe a bit too much, to make sure that happened and will be the fall guy since it is obvious the police and administration have no clue what it is all about. IT administrators take note! We will all be getting a visit by our non-technical supervisors as soon as they hear about this one.
I could see this attitude coming from spurious and vague complaints that were more appropriate at a helpdesk than an admin's plus over-work. Even the negative articles I have read all said his coworkers were generally supportive and sympathetic.
Whatever way you carve it, for this situation to occur you need some serious failures at a management level. As for the way they are handling the situation they've created, I can only assume that these guys must be complete buffoons.
Your article is spot on. There is no way that the news is portraying this story the way it really was. I have known Terry my whole life, my dad and him were close in the past, and he may have gotten into trouble as a teenager, but he is past that now. That was a learning experience for him. Something else must have been going on for him to just give up on something he has spent years creating and maintaining. He doesn't just throw his hands up and quit when something is tough. He is not a quitter. That company has known for years that Terry was the one maintaining everything. Why now were they questioning his access after so long? There are two questions we should be asking. What made Terry feel the need to do this? And why did the company wait so long to question his access?
Yes, that's the problem. We are going to lose Terry but do we have anyone who is as competent or anyone who is willing to work for the city to fill his shoes? Also, what is the story of SFPD (and whoever involved) jumping into arresting Terry in such a way that would put the entire city's network in jeopardy (now in the hands of incompetent mgmt & inferior sysadmin)? We don't go to war when diplomacy is available, especially when this war is totally not justifiable. Terry is not a villain - he is a hero. A very lonely one.
If the security manager cared anything about security he would not be talking about possible security holes in the network to the media. What if a hacker found these supposed holes. Would they blame Terry? This is very common in the network world from what I have seen. If they convict Terry this will set a standard for network admins. I have been following the story since it broke. I just cannot believe it has gone this far. The bond is unreal. I agree Terry is a hero.
Only in San Francisco! They locked up a hero and gave the person who has a track record of messing things up a promotion to becoming the new security manager!
The IT department spokesman Vinson's comments are embarrassing to read. He is more convincing if I tell you he is a standup comedian. If you don't believe me, go ask him something slightly technical. He will crack a joke and then run for his life. While on his way out, he probably murmurs something like 'got to go, the mayor wants me now'.
This seems like yet another example of stupid people making decisions regarding internet technology. $5 million bail? Nothing was stolen, nothing was broken. That 'judge' should be sitting in jail for such a stupid-ass decision!
The only logical course of action in the next few days would be for Gavin Newsom to sack the leadership of San Francisco's IT department. If anything Childs' actions demonstrate that the department's management was completely out of touch. In this story, Childs' malice has yet to be proven but government's incompetence is in full display.
- Tim O'Brien
Two More Years, Appeals, Fallout
Terry Childs has two more years to romp around in a jumpsuit. He'll appeal if he has any sense. He's already been in that jumpsuit for over two years - 750+ days - all for allegedly not revealing a password to a system that competent engineers didn't need anyway. (And what happened to the right to a speedy trial?)
Terry Childs is IT's first high-profile political prisoner and his case exposes the buffoonery and corruption in IT management that every IT pro is painfully aware of.
Terry's story is going to have cataclysmic repercussions in the IT industry. IT students targeting work as network and system engineers are going to think twice. IT pros already in the field are going to regard their management with even more suspicion. A great many of them are going to plan career changes. The world of IT can't survive if only the managers remain. IT managers can't do shit.
Terry is always opening his heart to anyone he comes in contact with.
- Bobbi Meairs
This is what happens when non-technical people manage technical people.
You give someone your life and they turn on you like a rattlesnake. It's the hardest thing in the world.
- Brad Childs
LinkedIn: Dana Hom
Facebook: Dana Hom
SFGate: SF computer tech turned life around
CIO/Paul Venezia: Sorting out the facts in the Terry Childs case
CIO/Paul Venezia: Why San Francisco's network admin went rogue