Pages

Jump to bottom

22 comments

1 Justanotherhuman  Jun 26, 2014 5:46:04pm

I feel lusty.

2 EPR-radar  Jun 26, 2014 5:59:56pm

Another RWNJ derpfest turns out to be a nothing-burger they blew out of their asses.

Must be a day ending in -y.

3 Charles Johnson  Jun 26, 2014 6:11:49pm
4 Charles Johnson  Jun 26, 2014 6:13:54pm
5 Charles Johnson  Jun 26, 2014 6:19:17pm
6 lostlakehiker  Jun 26, 2014 6:30:43pm

At my school, emails are remotely backed up and kept in perpetuity. This has been the case for decades. All in accordance with law. If my emails were required and not produced, talk of crashed hard drives would be pure mendacity.

And the IRS is technically way behind the times in this matter?

There’s another issue here. If emails from 7 specific further officials were requested, and all their hard drives turned out to be gone, that’d be different than if emails from 700 were requested, and 693 hard drives were available and 7 were not. Does anybody know the actual stats of requests and deliveries?

7 Fairly Sure I'm Still Obdicut  Jun 26, 2014 6:57:25pm

re: #6 lostlakehiker

At my school, emails are remotely backed up and kept in perpetuity. This has been the case for decades. All in accordance with law. If my emails were required and not produced, talk of crashed hard drives would be pure mendacity.

What law requires your school to keep all your emails in perpetuity?

Also, backups get corrupted (or turn out to have been made from corrupted versions) all the time. A two of the corporations I’ve worked at, emails got lost and were unrecoverable. And one of those was EA.

There’s another issue here. If emails from 7 specific further officials were requested, and all their hard drives turned out to be gone, that’d be different than if emails from 700 were requested, and 693 hard drives were available and 7 were not. Does anybody know the actual stats of requests and deliveries?

Nope. Which is why going around saying “Seven hard drives!” as though it’s meaningful is dishonest, stupid bullshit.

8 goddamnedfrank  Jun 26, 2014 7:11:18pm
Description of IRS Email Collection and Production

II. Physical Retention, Collection, and Production of Email
The IRS email system runs on Microsoft Outlook. Each of the Outlook email servers are located at one of three IRS data centers. Approximately 170 terabytes of email (178,000,000 megabytes, representing literally hundreds of millions of emails) are currently stored on those servers. For disaster recovery purposes, the IRS does a daily back-up of its email servers. The daily back-up provides a snapshot of the contents of all email boxes as of the date and time of the backup. Prior to May 2013, these backups were retained on tape for six months, and then for cost-efficiency, the backup tapes were released for re-use. In May of last year, the IRS changed its policy and began storing rather than recycling its backup tapes.2

A. Email Preservation
In late May 2013 and early June 2013, the IRS sent document retention notices to employees identified as having documents (including email) potentially relevant to the investigations. These notices instructed employees not to alter or destroy:
all communications, documents drafted or reviewed, spreadsheets created or reviewed, notes from meetings, notes relating to specific taxpayers and/or applications, information requests to applicants, training materials, or any other items that relate to the process by which selection criteria were used to identify tax-exempt applications for advocacy organizations for review, including but not limited to Be On the Look Out, from January 1, 2008 to the present.3
In that same time frame, the IRS sent similar document retention notices to all employees in the IRS Tax Exempt and Government Entities function and its Chief Counsel counterpart; the IRS Communications and Liaison function; and all employees assigned to respond to the Congressional inquiries.

B. Employees’ Email Storage
The IRS has approximately 90,000 employees. Due to financial and practical considerations, the IRS has limited the total volume of email stored on its server by restricting the amount of email most individual users can keep in an inbox at any given time. This is not an uncommon practice within the government or the private sector.

󿿂
According to estimates, it would cost well over ten million dollars to upgrade the IRS information technology infrastructure in order to save and store all email ever sent or received by the approximately 90,000 current IRS employees.
Currently, the average individual employee’s email box limit is 500 megabytes, which translates to approximately 6,000 emails. See Attachment B. Prior to July 2011, the limit was lower, 150 megabytes or roughly 1,800 emails. See Attachment C. The IRS does not automatically delete email in its employees’ email account to meet these limits; rather, each employee is responsible for managing and prioritizing the information stored within his/her email box.

Historically, the email of IRS employees is stored in two locations - email in an individual’s active email box and therefore saved on the IRS centralized network and email archived on the individual employee’s computer hard drive.4 If an email user’s mailbox gets close to capacity, the system sends a message to the user noting that soon the mailbox will become unable to send additional messages.

When a user needs to create space in his or her email box, the user has the option of either deleting emails (that do not qualify as official records) or moving them out of the active email box (inbox, sent items, deleted items) to an archive. In addition, if an email qualifies as an official record, per IRS policy, the email must be printed and placed in the appropriate file by the employee.5 Archived email is moved off the IRS email server and onto the employee’s hard drive on the employee’s individual computer. As a result, these IRS employees’ emails no longer exist in the active email box of the employee and are not backed-up as part of the daily backup of the email servers. Email moved to a personal archive of an employee exists only on the individual employee’s hard drive. An electronic version of the archived email would not be retained if an employee’s hard drive is recycled or if the hard drive crashes and cannot be recovered.

—-Footnotes—-
2 This practice of retaining rather than recycling tapes is estimated to cost approximately $200,000 annually.
3 Litigation Hold, Attachment A.
4 The approximately 2,000 IRS Counsel employees (as opposed to IRS employees) have a system that allows archived email to be stored on a central drive.
5 An official “record” is any documentary material made or received by an agency under federal law or in connection with the transaction of public business and appropriate for preservation (44 U.S.C. § 3301). Not all of the emails on IRS servers or backup tapes qualify as official records; accordingly, the agency’s email system does not retain all email indefinitely. Rather, individual employees are responsible for ensuring that any email in their possession that qualifies as a “record” is retained in accordance with the requirements in the Internal Revenue Manual (IRM) and Document 12990 (Record Control Schedules).

9 goddamnedfrank  Jun 26, 2014 7:19:05pm

Also, the reason intelligent enterprise IT departments don’t put too much faith in network backup of .pst files is that Microsoft says it’s a bad practice and can easily lead to loss of data.

The Microsoft Exchange Server 4.0 team created .pst files in order to let users maintain a copy of their messages on their local computers. The .pst files also serve as a message store for users who do not have access to a Microsoft Exchange Server computer (for example, POP3 or IMAP email users).

However, .pst files are not intended as an enterprise network solution. Although it is possible to specify a network directory or a Universal Naming Convention (UNC) path as a storage location for a .pst file, network usage is not meant to be a long-term, continuous-use method of storing messages in an enterprise environment.

A .pst file is a file-access-driven method of message storage. File-access-driven means that the computer uses special file access commands that the operating system provides to read and write data to the file.

This is not efficient on WAN or LAN links because WAN and LAN links use network-access-driven methods. These are commands that the operating system provides to send data to, or receive from, another networked computer. If there is a remote .pst file (over a network link), Outlook tries to use the file commands to read from the file or write to the file. However, the operating system must then send those commands over the network because the file is not located on the local computer. This creates lots of overhead and increases the time that is required to read and write to the file. Additionally, the use of a .pst file over a network connection may result in a corrupted .pst file if the connection degrades or fails.

So prior to 2013 the only reliable backups were on those tapes that were being routinely recycled after two years. Backing up the personal computer .pst files was a shit option that Microsoft specifically went out of its way to discourage. The IRS was generating a fairly retarded amount of email traffic and needed some way of managing its archives, so it did what it could and had employees store email archives locally on their own work machines.

10 goddamnedfrank  Jun 26, 2014 7:26:29pm

re: #6 lostlakehiker

At my school, emails are remotely backed up and kept in perpetuity. This has been the case for decades. All in accordance with law.

You realize this is pure raw assertion, don’t you, which even if correct has no real bearing on the IRS situation. It’s highly doubtful your school has 90K employees steadily maintaining 170 Terabytes of email traffic in their inboxes alone. If it’s true as you say that your school’s IT department has maintained every email sent for decades (which would mean since the very invention of email,) in perpetuity and according to law then it shouldn’t be too hard for you to cite the law or point us to your IT department’s contact information so as to verify this interesting, absolute and categorical claim.

11 goddamnedfrank  Jun 26, 2014 7:32:39pm
The IRS is not the only federal agency to lose e-mails over the past few years. In fact, despite efforts at many agencies to standardize and improve e-mail by moving to services like Google Apps for Government and Microsoft Office 365 Government, many agencies still run their e-mail like it’s 1999. It’s not just a technology issue—it’s an IT policy issue, a staffing issue, and a cultural issue within government, one that the federal government shares with many private corporations.
12 Randall Gross  Jun 26, 2014 7:33:37pm

Most companies in the US today have retention policies that call for destruction of email and records once past the legal required date. Many specifically limit sizes on personal mail and set auto expire to be pretty aggressive so they don’t have massive oceans of email to go on fishing expeditions in.

13 Randall Gross  Jun 26, 2014 7:35:32pm

The other thing - if the “7 drive” thing is factual, several of them could have been in a single failed raid array (and yes, even RAID can fail.)

14 ObserverArt  Jun 26, 2014 7:48:37pm

Sad. This has some of the same tech issues the Snowden NSA revelations had. That is a bunch of folks that do not completely understand the tech issues, but by God they can make a stink of it all in spite of it.

And the opposition and media make it as scary as possible. And like a lot of human reactions…it is easier to be scared of what you don’t understand.

And it all circles out of control.

But it sells!

15 goddamnedfrank  Jun 26, 2014 8:05:04pm

deleted. wrong thread.

16 Zamb  Jun 26, 2014 8:19:22pm

re: #14 ObserverArt

After looking at my parents phones/ipads to help them out with some things I’m convinced that many people think that emails and such should just exist is because they simply don’t read/delete anything. Seriously 1000+ is not a number your inbox should read, clean that shit up.

17 ffakr  Jun 27, 2014 1:03:12am

Since I manage an IT Operation at a Research University, I can offer some insight about a few of the claims thrown around..
Heck, I’ve worked in IT at two different research Universities and that totals very nearly two “decades”.

At my school, emails are remotely backed up and kept in perpetuity. This has been the case for decades.

I doubt it. If they’ve actually been told this, their IT Staff is either lying.. or their school doesn’t generate much mail.

This doesn’t pass the smell test for several reasons.
Decades?
That takes you back to 1994. I was reading email from a MVS Mainframe in 1994, where my allotted storage was measured in “cylinders”. If we’re to believe that your school has email archived from at least 1994, that would mean they have been migrating archived messages from one Computer system to another one over and over,.. from one Mail system to another over and over,.. from one storage media to another, over and over.. without data loss. You can’t even write out 1994’s email to media and expect to retrieve it 20 years later without going back regularly to migrate it to new media. Not only does media go bad.. it goes obsolete. Those 20MB backup tapes aren’t so easy to read anymore in 2014, and not just because the magnetic film has depolarized the adjacent layers years ago.. because your last tape drive broke 10 years ago [old tape drives.. too many moving parts..]

If your school’s 20 year archive is bigger than one contemporary disk or tape, keeping it all would be pretty miraculous a feat.

At the Universities I’ve been associated with, there were many mail systems.
The State school was better about having people use central mail systems, but in an open educational environment, its near impossible to keep someone from spinning up a sendmail box for themselves, their lab, or even their entire Department or Division.
At the more prestigious Private school, everyone has their own mail systems. Some of the better funded Divisions go out of their way to get their people to avoid using the central mail system. Heck, I’ve found production mail servers that were set up by and serving one small research group.. running on an old Dell desktop with zero backups. I’m not sure the last person who knew the root password was still at the University when they called for help with it.

Other thoughts..
I’m in IT.. I’ve personally lost large chunks of email at several points during my professional life. It happens, and it doesn’t require a hard drive crash.
A misbehaving Mail reader can tell your mail server to delete much or all of your email.
In some cases, you’re not going to get that mail back even if you go to the email admins and request a restore. Various mail systems have had serious limitations on backup and restoration over the years. At one point in MS Exchange, it was pretty much impossible to restore individual emails.. even individual mail boxes. Our central IT Group wasn’t able to do individual mailbox restores with our previous proprietary Mail appliance, or with Exchange until they migrated to Exchange 2010.

I’m not at all surprised by the other information posted about how the IRS actually handles email. The size limitations.. the bad policy of telling users to “archive” their old mail by copying it to local mail folders. It’s what we’re occasionally forced to do because there isn’t enough funding to give people as much mail storage as they’d need to keep it all on the server.
.pst files? They get corrupted .. a lot. Well, used to be worse but it still happens often enough.

Keeping track of mail isn’t easy. Between the mail servers.. the storage.. the backup systems working reliably.. and the users all running software that’s designed to delete email from the local computers and the servers.. there’s endless ways to loose emails.

Oh yea.. almost forgot.. Enterprise Storage RAIDs really do fail too. We spent over 30 hours straight at work a few years back when we had cascading failures of Seagate Constellation drives in and expensive Enterprise storage box.. and then found out that our backup software had a bad index of the tape data. The Storage vendor got the array limping along right before we finished spinning all the tapes end to end to re-index their contents for a restore. RAID 50 with hot-spares.. and we lost drives faster than the spares could rebuild.. KABLOOIE!!

18 ffakr  Jun 27, 2014 1:13:45am

.. almost forgot..
Being an IT Guy.. I’m beta testing OS X 10.10. What can I say, I’m a glutton for punishment.
Guess who can’t use their preferred Email application to read their mail right now because it crashes on launch?
This guy. :-P

Sure, sane people don’t install beta OSes on their primary computer.. but you’d be insane to think it’s not easy to loose email.

..Email lost by IRS Staff with no particular technical expertise? UnFathomable!

19 wrenchwench  Jun 27, 2014 8:53:20am

re: #18 ffakr

Welcome, hatchling.

20 Slap  Jun 27, 2014 8:58:43am

re: #19 wrenchwench

Welcome, hatchling.

Seconded. And a wonderful first post!

21 Fairly Sure I'm Still Obdicut  Jun 27, 2014 10:08:50am

Lostlake, are you going to name the law that requires your school to keep your emails in perpetuity, or did you just make it up?

22 Amory Blaine  Jun 27, 2014 10:47:01am

End times. 7 seals.


This page has been archived.
Comments are closed.

Jump to top

Create a PageThis is the LGF Pages posting bookmarklet. To use it, drag this button to your browser's bookmark bar, and title it 'LGF Pages' (or whatever you like). Then browse to a site you want to post, select some text on the page to use for a quote, click the bookmarklet, and the Pages posting window will appear with the title, text, and any embedded video or audio files already filled in, ready to go.
Or... you can just click this button to open the Pages posting window right away.
Last updated: 2023-04-04 11:11 am PDT
LGF User's Guide RSS Feeds

Help support Little Green Footballs!

Subscribe now for ad-free access!Register and sign in to a free LGF account before subscribing, and your ad-free access will be automatically enabled.

Donate with
PayPal
Cash.app
Recent PagesClick to refresh
The Pandemic Cost 7 Million Lives, but Talks to Prevent a Repeat Stall In late 2021, as the world reeled from the arrival of the highly contagious omicron variant of the coronavirus, representatives of almost 200 countries met - some online, some in-person in Geneva - hoping to forestall a future worldwide ...
Cheechako
2 days ago
Views: 95 • Comments: 0 • Rating: 1
Texas County at Center of Border Fight Is Overwhelmed by Migrant Deaths EAGLE PASS, Tex. - The undertaker lighted a cigarette and held it between his latex-gloved fingers as he stood over the bloated body bag lying in the bed of his battered pickup truck. The woman had been fished out ...
Cheechako
2 weeks ago
Views: 261 • Comments: 0 • Rating: 1