Getting IT and Engineering to Play Nice
[August 2013] Radio engineers and IT people both work with electronic gear, mostly connected by wiring. However, although a few engineers can handle both, it often seems these two disciplines speak two different languages and operate with very different priorities.
Finding ways to work together may be a challenge at times. But in the end, it will benefit everyone.
In the beginning, only engineers, researcher types, and bookkeepers had computers.
It was 1979 when the first interactive, auto-recalculating spreadsheet, VisiCalc (The Visible Calculator), blasted personal computers out of the “techie” realm. VisiCalc was a huge time saver, allowing the kind of instant “what if” financial or numerical scenario analysis that we now take for granted.
Indeed many would claim that VisiCalc was the first “killer app” – an application that people found impossible for them to do without. As VisiCalc and other applications hit the market, virtually overnight calculators, typewriters, and manual cash registers became surplus gear at offices and shops all over the country.
As it progressed, the digital revolution resulted in innovation after innovation, and the time needed to bring products to market, as well as product lifetime, has gotten very short since 1979; today an application can be written, tested, and posted to the Internet for downloading easily in just a matter of hours.
Computers Meet Broadcasting
In a very similar way, a computer revolution also has taken place inside our broadcast stations.
At first, there was little conflict when broadcast companies started converting their stations to PCs and removed most tape machines. That removed numerous moving parts and, in most cases, dramatically improved the quality and reliability of the air product.
As computers took hold, networks to tie them together to share resources did as well. Novell had the first file server software with Netware in 1983. Windows NT followed in 1993. Most engineers updated their skills at this point, sometimes even with their own money, to support this new technology.
Now we commonly tie large groups of local computers together via Local Area Networks (LANs) and, with firewalls, hope to protect everyone from the evildoers that troll the Internet. Wide Area Networks (WANs) tie locations together, allowing central control of computers and LANs.
The challenges of administering a LAN and supporting desktop users are many. Clearly, in this age where computers are running broadcast stations, the engineering department needs IT’s full cooperation. In fact, many feel that the IT department should be an integral part of the engineering department.
A Bad Division of Responsibility
Unfortunately, some broadcast companies have allowed an ill-advised creation of a department separate from engineering: the “IT” department. Usually it is not properly thought through. The reason is that the technical and IT departments operate in some very different ways.
Broadcast engineers are trained to be very flexible in their approach to problem solving, being ready to do almost anything to get back a station on the air as quickly as possible. Dead air, even hums and buzzes, is sure to bring complaints from the staff – and the engineer often is tasked with an immediate problem solution.
On the other hand, IT departments can be very inflexible and unaware of the unique nuances, pressures, time demands, and the 24/7/365, competitive nature of our business. IT people tend to be highly oriented toward 9 to 5 operations and shutting everything down while they troubleshoot is their normal mode.
The company with an IT manager that has been a chief engineer and truly understands the bigger picture is a fortunate one indeed.
Getting Along
In some ways, it might seem some broadcast engineers are getting a “taste of their own medicine.”
As long as there have been microphones, transmitters and antennas, engineers have led a somewhat shielded existence, sheltered from brutal sales calls and the “Class Clown” world of air personalities.
However, for the first time, many engineers now have to deal with a different, often detached, and mostly unaccountable department that controls a vital resource. The shoe is on the other foot – and the fit could not be worse.
There may be many opportunities for conflicts between a typical IT department and Broadcast Operations. This is because Broadcast Engineers are responsible for on-air product – which produces the revenue stream – but most, if not all, air product is generated today by computers, usually from three or four layers or more of computers.
Relations Going Off the Rails
In fact, most stations now do some walk-away automation. When Internet protocols (IP) are used, the physical control location is almost irrelevant.
This means that people from thousands of miles away, with little or no knowledge of the local day-to-day challenges, often now are calling many of the shots for our companies’ revenue generating systems.
Yet, IT departments can often be unaware of the physical distances our LAN extensions can traverse, such as when piggybacking with our audio T1s, and the time, gas, and effort it takes to get to each location, especially if you work for a large cluster of stations.
And this is where things can get “sticky.” Once a LAN becomes “centrally managed” and secured from far, far away, moving any of the devices on the LAN (printers, computers, etc.) often requires a call to IT to coordinate and allow that change to happen.
Even more exasperating: installing a new program – or even an update to programs like TAP-SCAN, QUALITAP, Media mix, SmartPlus, X-Ray, Media Audit, etc. – either is difficult or downright impossible without “admin privileges.”
Sometimes, just opening a port to get a vendor into your LAN for an update or support can become an all-day affair.
Some attempts to justify this lost productivity and expense are laid on the often mentioned and dreaded “Sarbanes-Oxley.” Yet, it really is hard to regard that claim as much more than scare tactics, since SarbOx only covers financial data – essentially being the IT equivalent of the FCC Rules and Regulations.
Uniformity is a Lost Cause
One common IT goal is “Uniformity” and “Best Practices.” In other words, “one size fits all.”
But in most cases this is neither a rational nor a practical goal at a broadcast station because broadcast stations usually do not lend themselves well to computer uniformity.
Consider the how broadcasting works.
The typical lineup of computers at a broadcast station is not the least bit uniform. Machines in the Control Room, Production, automation racks, offices, and at the transmitter all have varying software installs, some unique to that user, some running an Apple OS.
How many editing programs do you support? I have been in shops where there are four or more in use. Production people have their own preferences for editors and plug-ins, and very few want to relearn another editor when starting a new job.
Furthermore, when a contract engineer comes in and cannot access the automation equipment system because he is not “a fulltime employee” and therefore cannot be issued an account and password, things have reached a silly point.
This is where a helpful IT department can solve a lot of problems, allowing access to specific on-air systems, but not the main LAN with the business systems on it.
The Password Hassle
The effect of centrally-managed computers on the people in the trenches can also become a morale problem in another way.
Computer security is important. But when the big brother aspects of remote management set in, the underlying theme quickly comes through loud and clear: some companies do not trust their employees to operate a computer in a responsible manner.
This time, part of this truly can be blamed on SarbOx, especially if access to the LAN means access to the entire network.
Still, those periodic forced password changes, (usually every 90 days) can cause problems. Is it really necessary for every studio work station to have passwords of 25 characters that fit complex rules, are secure – and are especially hard to memorize?
The real world end result “in the trenches” is paper notes on monitors with each user’s password written on it. That really is not very secure at all. In fact, it rather defeats the very purpose of passwords.
On the other hand, here again, cooperation between engineering and IT can easily specify those mission critical on-air systems where the passwords do not need changing all that often.
When security policies are set with that in mind, everyone’s needs can be satisfied.
Who Is In Control?
Consider for a moment how the new technology is a game changer.
Remote access via the Internet to station computers is vital to keeping operations going. Nevertheless, some IT managers resist any effort to share remote management capabilities they want to reserve for their own staffs.
Unless there is good cooperation, remote computing can cause problems. That is especially true if someone from the far away, central management point – someone who may be unfamiliar with the station’s broadcast operations – gets into a system.
Content filtering is another challenge. Keeping morning shows happy is one of the things we engineers do to please management. However, some morning shows have questionable content and need to access unconventional sites to get material. Some IT policies can make that very difficult.
While, no one wants employees to waste time surfing porn – and possibly cause a hostile work environment – a morning show trying to research breast cancer can get blocked as well. This is not a good situation.
Worse, just imagine the response to giving your highly rated, highly spot-loaded morning show a help desk number in another state when they are trying to do their show prep!
There is a better way: IT and engineering must talk and cooperate for the station’s good.
At the Most Inconvenient Time
My favorite anecdote about how all the various these conflicts can play out concerns a station where the IT people permitted a “Microsoft Tuesday Update.”
Of course Murphy’s Law demands that the subsequent reboot of all the PCs on the LAN happen right smack in the middle of afternoon drive – and with no warning. And it did.
Even better (for Murphy?), some of the remote computers failed to reboot, rendering some critical systems inoperative until they could be reached by an engineer.
Loss of even a couple of minutes of air time (or production time) – especially in afternoon drive time – can be expensive and hard to recover, especially in rated markets.
More critically, updates have been known to break things like the automation software that is running your afternoon drive spot load, the production manager’s editor, the news department’s feed, traffic software, etc.
The lesson is that when such patches are installled without the knowledge of local engineers, troubleshooting can be almost impossible. (The Microsoft restore point feature is a life saver, but only if a restore point is created immediately before patches are applied.)
Sharing Control
As noted above, one effective compromise is for a radio station LAN (and its attached devices) to be split into two parts.
One part is the LAN that can be managed from a central location. A different LAN is for locally managed engineering and operations. This is the LAN where the automation, the transmitter computers, HD encoders, HD importers, remote control systems, satellite receivers, etc. are located and controlled.
This part of the LAN can have remote applications such as VNC, LogMeIn, GoToMyPC, or Team Viewer installed and not allow access to sensitive business files.
Yes, sometimes a “bridge” device is needed that has two LAN interfaces in it to allow temporary connections to both LANs for log and other data transfers. When IT and tech talk, it can work.
The compromise with IT folks is this: if it is part of the locally-managed tech LAN, it is none of the IT departments’ business. A good selling point is to state that the IT department does not have to support this entire group of machines. But allow IT to control the “bridge” device and the centrally managed part of the LAN.
Finding Solutions
The biggest cause of stress in a work environment is responsibility for something for which you lack control.
Engineers often become responsible for something over which they have little control. A good example of why control of your tech LAN is so important might be the satellite receivers which store programming on a hard drive, and which needs to be on the LAN and Internet.
Engineers who do not control their LANs need to talk with their boss about better policies for their operations.
To be fair, not every IT department is completely inflexible over control; indeed, your mileage may well vary. There are large broadcast groups where the engineers have control of their IT department, or at least full cooperation. In those cases, if the Chief Engineer says a machine needs VNC or TeamViewer, then it gets it.
Get With the Future
If you lack the experience or training to administer your LAN, you need to update your skills.
I have an extensive LAN at my home used as a test bed to try new devices. This is a great place to “break” stuff – and repair it yourself. Those engineers that have refreshed their skills in the last ten years are able to administer their LANs – and impress their supervisors.
Finally, as we move into the future, it is clear that most electronic devices more complex than a toaster will have a CPU of some kind in it, and many of these devices will be on a LAN.
Having the right people in control of mission critical gear is very important. If there is any question, just remind your boss who gets called first when there is a problem on the air at 2 AM.
I rather doubt if it is a far away IT department.
– – –
Steven Martin is a pseudonym for an experienced engineer who is just slightly worried that somewhere an IT person is plotting against him.
– – –
Would you like to know when other articles like this are published?
It takes only 30 seconds to add your name to our secure one-time-a-week Newsletter list.
Your address is never given out to anyone else.
– – –