It looks like Folio Photonics is finally rolling out actual products (Savartus)

[u/ernest314]

https://savartus.com/solutions/enterprise-laser-storage/

[u/dlarge6510]

Excellent 

Exactly what I need for where I work. 

I'm the data archivist at work and must migrate the following:

• ~100 DDS tapes, all versions from DDS up to DAT72. The oldest tapes were written in the mid 90's. 

• ~ 300 LTO tapes of all generations from 1 to 6

• Many CD-Rs and a few 15-20 year old USB HDDs are knocking about. 

All stored in the server rooms so 18 Deg C for decades.

I'm frequently asked to retrieve data. This is engineering data and we have projects going back to the 80's still in use and occasionally someone will ask for data from 1983. Unfortunately I didn't have that particular data, nobody seemed to have put it on tape or it's on an uncatalogued tape I haven't found yet.

The old tape catalogue is gone. I must rebuild it from scratch. The only catalogue that survives is a bunch of hand written notebooks that only say what system was backed up (those who know what these systems were used for are long gone). I hear rumours of legendary programmers who wrote the original code and the source code for the machine still in use 30 years later is probably on their old PC dump somewhere on a tape or CD perhaps. Source code that is practically looking like the holy grail.

I literally am like Indiana Jones or Relic Hunter lol, only the server room doesn't throw a boulder at me when I uncover a tape that hasn't been out of a drawer since 2002. But there are plenty of really scary H&S issues with the old building that have been uncovered so there is that.

The industry I work in makes most companies that talk to us choke when I tell them that I need what they are offering to retain data on a shelf in cold storage for at least 30 years at a bare minimum. We have paper and microfiche records too, and it's so old we know there are museums and historical associations that would write blank cheques for some of it. 

We even had 9 track reel to reel data tapes. STACKS OF THEM! As used in our first computer which was one of THE first computers, possibly a Ferranti back in the 50's.

Unfortunately I had no way to read those so we had to shred them :(

None of the paper tape survived either. Apart from some of the tools they used back then to punch holes in it manually.

Unfortunately I have a tough job or trying to shoot down IT storage companies as they suggest using a HDD or flash based system that won't last a decade. Heck they make me laugh out loud once when they asked how long we need to archive data for, as they thought it was 7 years 😂 

You should have seen their faces. I specifically asked them if they had a tape solution that could hook into their HDD/fash arrays. Firstly they were all like giggling and saying " most companies use the magical cloud" and I was all serious and like "we can't use the cloud, practically any cloud for contractual reasons it must be held on site and in this country at the least. Do you have a UK sovereign cloud then?". Of course they said no and said that most companies happily export all their data to all countries in the world randomly with no oversight or control of who has access and what nationality they are. 

Which is extremely important for many customers as worldwide customers sometimes stipulate that their commercial data is not to be seen by eyes that are not of an inappropriate nationality, because the world is complicated and they don't want the politics of foreign competitors would you believe 😲

So they ask how long do you really need to keep archive data?

I replied: Um, around 30-50 years.

When you are in engineering much of what is engineered for mega money tends to have a lifetime of d e c a d e s and sometimes someone will make a query about the design or the testing that was performed all the way back at the start. In the 80's.

And I'm the poor soul who has to try and find it. On a archive that has been ignored by those I have worked with in IT who have no concept of data retention and archival. 

I have had to literally defend a box of tapes from being shredded by some management just because they think it's old and hard to deal with and they think it's not an issue.

So I must migrate this stuff to LTO8 which is what we selected. Only I barely get time 😔 too many other things.

So after rattling on too much. This would be right up my street however I'll not jump right in as firstly I need that first LTO8 archive upgrade I'm doing to complete then if this is established enough I might let management know about it.

The tapes have time and time again shown me they are far superior to HDDs (I found three HDDs had died killing a server running of a raid 6 just yesterday with no warning) but as you can imagine they are starting to age badly and the older tapes dirty the heads of the drives as I read them. 

And I'm a f*cking optical fanatic as they last longer than the tapes and don't need that much baby sitting.

Whether this gets to the consumer I don't know. I recall one of the new "mega optical" technology companies certainly did say they had a road map to a consumer unit.  Heck if a prosumer unit comes around and like we have LTO5 externals second hand affordable today I'll wait and get the old IT gear. 

I also like the fact they have Blu-ray support in their cartridges. Neat. So you don't need to jump in with the new discs and perhaps can mix them.

[u/jreykdal → u/dlarge6510]

I work in Television. We have to migrate our videotapes to digital formats and have been on spinning disks until now. We are commissioning a flash/hdd/lto hybrid system. Hopefully it will buy us some time as the sizes are astronomical and we have to keep the stuff "forever".

[u/trizephyr → u/jreykdal]

In Television here too. We have a room full of tapes that needs to be digitized but I’m not volunteering XD

[u/External-Log630 → u/trizephyr]

Could I pick your brain? That's the kind of work I'd like to do as a subcontractor for someone like you guys. Sent you a message..

[u/TheRealHarrypm → u/jreykdal]

I would highly consider also going to modern FM RF Archival the FLAC compression efficiency on preserving the source signal instead of going for baseband V210 etc save the massive amount of initial ingest space, especially for Type-C and Umatic the post processing benefits along with save thousands in legacy hardware, this has a drastic cost saving later down the line this might also apply to u/trizephyr too.

Especially since this has easy exporting of the vbi space for example throws out Bluefish cards instantly.

[u/jreykdal → u/TheRealHarrypm]

We have to balance the quality (which is dubious at times) with usability so we store everything in IMX50. It would be fun to tap the rf directly but the complexity is way too high.

Also we are looking at many tens of thousands of hours. Been at it for 10 years already and still have a couple left.

60 years of TV and 95 years of radio. Of course not everything saved but still shed loads of stuff.

[u/TheRealHarrypm → u/jreykdal]

The complexity isn't too high at all it's 2-6 soldier joints, your RF-C line for single and multi-channel formats, and Y/C lines for dual channel formats like Betacam, and a couple cables for 12-36v power tap.

It goes to a fixed amplifier that is not complex at all to anyone familiar with an oscilloscope and minimum soldering skills hell you can just visually adjust it without test equipment even, in fact it's been so turnkey for the last 3 years, it was only the real time decoding and the unified metadata aspects that have been long since solved that held it back from being a complete drop in for legacy SDI or PCIe card workflows.

Especially with now MISRC GUI standardizing the ingest workflow for virtually any ADC platform deployable for this, decoding is just tell it the format and hit run and even that can be fully automated nowadays if 30 seconds of effort is made on the initial ingest to note the format and speed type, and we have multiple real-time forks of the original Python projects.

Literally all you're doing is just the same motions as any legacy ingest software.

And now we fully replaced the signal analysis hardware stack inside of tbc-tools there is not really much practical point investing in keeping legacy workflows running outside of historical purposes or representational training.

IMX50 is still MPEG2 we have it as an export option but it was completely made legacy by FFV1 lossless compressed at the same bit rates for 8-bit 4:2:2 little bit higher for 10-bit but most formats don't need it if kept within 8-bit level range, which is all automatic with software decoding instead of relying on a hardware tbc to be accurate or fixed configurations it can automatically adjust that on every single field and frame, again in real-time today.

[u/somersetyellow → u/dlarge6510]

Sony had Optical Disc Archive for a time but it never gained much ground against LTO. When it comes to long term archival pretty much everything is solidifying around LTO.

Long term archival is also about being able to read it back in the future with what you can cobble together, and enough LTO is out there you'll probably be able to find a drive out there somewhere in 30+ years.

Obscurity unfortunately nails a lot of these things.

Optical has the advantage of not being magnetic and having a much simpler mechanism though. Really hoping folio figures out a path because it hasn't looked good for them for a while.

[u/yohomatey → u/somersetyellow]

I've used ODA. It was slower and with far inferior capacity to even a somewhat modern Gen LTO. 1tb per puck, or thereabouts. At the time we were getting like 3tb on an LTO.

[u/somersetyellow → u/yohomatey]

That's what I heard too. More expensive too right? It was motivated after a lot of the flooding they had in Japan in the 2011 earthquake I read.

But yes, slower, new format nobody used, and LTO is just absolutely everywhere at this point when it comes to this niche market.

It was interesting in that they used ArchivalDisc which to date is the only commercially launched 12cm disc that went beyond BDXL and reached 250 gigabytes per side with a coming roadmap of 1TB per disc. But that was all we got of that format.

[u/yohomatey → u/somersetyellow]

I don't know the expense, I wasn't the ones writing the checks haha. But I do get the sense it was that way. It also had a pretty high failure rate, but I think that was the software?

[u/somersetyellow → u/yohomatey]

It involved treating stacks of BDXLs (gen 1) or ArchiveDiscs (Gen 2 and 3) like hard drive platters. Multiple optical read arms would swing in and read them all at once. And from what I understood they'd all write to them at once.

I've dealt with BDXL 128gb discs in the past and those write at 18MB/s with read back speeds of 32MB/s (or 54MB/s on faster drives). Not sure what AD got to, but I'd assume it was still slow. Especially considering LTO is well past 400MB/s read write nowadays.

[u/TheRealHarrypm → u/somersetyellow]

The biggest failure was selling the readers at a high margin of 15k instead of at a couple hundred bucks or less and then making back the money over indefinite production of media...

[u/somersetyellow → u/TheRealHarrypm]

Considering that even consumer hard drive sales are so small that it could be a single digit percentage rounding error to the companies making them nowadays I'm not super surprised unfortunately

[u/bigvalen → u/dlarge6510]

It's fun. In the storage world, you assume you only have data backed up if you can read it back. In reality, just because you could read it back last year, does not mean you could read it back this year.

When I worked in Google, they used a parity system where any data was streamed to (I think) ten tapes; two were for parity. So, when you read it back, you needed eight of the ten tapes to be working. More importantly, all data needed to be read & re-written every year. That was how they did the gradual LTO-3->4 and onto 5, 6 move. There were incidents where we lost three out of ten tapes, and I had photos of techs on the DC floor trying to sellotape snapped LTO tape back together. (Holy shit, it worked sometimes)

It meant that storying data was expensive, so it encouraged people to put a lifespan on it, but once you did, you had a decent chance to save it. This ethos came out of an unfortunate incident where I deleted the only copy of the original 1998 search index. To be fair, there was only one copy - in a single ancient storage cell, with the filename "search98.tgz" in the days when Google storage didn't have owners/groups, never mind file permissions. No one knew what it was, or missed it when I turned the cell off for a month before wiping everything.

[u/eliwood98 → u/dlarge6510]

This sounds like a cool job that is probably the kind that doesn't ever have enough people, so it has security and a decent pay. What kind of qualifications does a person need, and how does one break into this field?

[u/dlarge6510 → u/eliwood98]

Yes it is cool. 

I have a degree in Computer Science and experience in IT support, Software Quality Assurance (software testing), customer service, system design and implementation and maintenance. All comes with an 6 year job in Software Testing and 11 or so in general IT support across a broad range of technology.

My last job was working for a brewery as the guy literally bridging the gap between the service desk (1st-3rd line support) and my brother in law who was (still is) the system administrator. I did everything from figuring out why someone's print spooler wasn't working to actually building, installing and testing servers. As employee numbers dwindled due to the effects of COVID (6 months of furlough makes some look for new jobs) I was doing everything, all lines of IT support, infrastructure and visiting pubs on site to fix, diagnose and install. Then they made me and the one other guy still left redundant, he literally walked out, didn't even empty his drawers so I was left running it all with my brother in law. 

My current job employed me for very specific reasons:

• I had used the ticketing system and endpoint management system they also used in my last placement so knew it well.
• My software testing experience and my hobbies as a kid in photography, astronomy, computing and electronics gave me a very focused attention to detail in diagnosis of issues and root cause analysis. My brain is literally geared to it, my knowledge and experience of software testing and development and electronics and logic knowledge whilst making a list of actual test points to perform in reality helps to locate and identify the issue and build a solution. 
• I'm a retro computer nut. I like old methods and hardware. I possess knowledge about everything from a Motorola 6502 which I would read the datasheet for when I was 11 for fun, to PLAs, retro system maintenance and repair (I watch such stuff on YouTube to start my day). I collect and use vintage systems, they are not old, they just need something to do.

They needed an engineer IT guy and that's what I was which is why I was made redundant from my last placement as they wanted younger  fresher ones (who I had to train) who didn't even know what a dot matrix printer was and thus what you'll do to diagnose an issue with it not printing. Yes, dot matrix printers in 2022. Turns out thermal printers don't work well in hot pub kitchens!

Anyway, the very first thing I had to do in my current place was to restore data from tape and they had lost the expertise in using the tapes. I was practically a tape virgin myself and more so for SCSI. Both I had touched on, as I had started using LTO4 at home to backup my optical archive and in my last placement I operated the LTO3 backup and LTO5 backup system so I was able to hit the ground running. I cobbled together a working system within a week and recovered the 1993 data a department needed off a DDS1 tape. I then managed to identify they had a mix of tar, Backup exec and bacula tapes using dd and a logical approach of figuring out what format the 64KiB block the tape drive kept giving me was. 

They also wanted me because of my extensive Linux experience. Having used it as a main operating system since the late 90's, from back in the day when to get X Windows working I had to write my X11 config file by hand!.

Which meant when I was lazy and hadn't done that I just used the command line exclusively for a week or two, used to watch TV on the command line with my TV card, watching Buffy the Vampire Slayer with fbtv in the graphics cards framebuffer on the command line was typical. It was good times 👍 

I live on the command line even today, a GUI is just a way to run Firefox and manage terminal windows. I would spend Saturday mornings compiling a new kernel because it was fun and building it into an RPM package as if it were nothing (well you only need a spec file, it's just that really). And they have plenty of Linux there including very old systems from back in my day that have never touched the stench of systemd (yes I'm an "init freedom guy").

How to get in? Well you need to live and breathe computers. 

You need that deeper understanding so you can find paths to explore. You need to be able to apply that to older stuff, the new stuff auto probes itself and frankly I find new tech particularly boring. Like a drone it mostly flies itself, what I have developed is an understanding of how it all works and a potentially unhealthy dependency on letting the automated systems give me less to do 😂 

I have seen my sysadmin brother in law practically shit his pants when a support guy said we had to recompile a kernel on one of the few Linux systems they had, to install a third party module needed for a service we had to upgrade. He is younger than me and knows Linux from the easy days of 2004+ when you'd install a distro and login to KDE without doing much. I was like, "K, whatever, give me an hour to configure it as I like exploring all the config options".

I took two hours as I found new stuff to read about in the newer kernel!

As for security, cyber security is a massive issue with today. I have gained much of my knowledge there from simply listening to podcasts, specifically Security Now on the TWIT (This Week In Tech) network (always makes me laugh as I'm British and twit is something else). Steve Gibson's podcast there looks not only at security but also technology deep dives, you want to know how the internet works? He did a multi part series on it, in depth. Sure it's from about 20 years ago but get this, nothing has changed. Ok, there is more IPv6 but plenty of follow up podcast episodes to cover that. 

Security Now, ALL of it, is available on Steve's site at www.grc.com. He has a multitude of free programs and utilities there too, programs everything in assembly! Doesn't do C or C++, he codes direct for the x86/x64 CPU in the CPUs language directly.

Every episode of Security Now is typed up too, you can search the text with the search engine. He goes into everything, in a very accessible way. You'll learn what RowHammer is and why it's important and why DDR4 was important and yet why it was found to be not effective enough and then you'll nod your head when someone comes on here saying ECC RAM is essential and not having any when even doing a file copy is like driving the car without a windscreen on and thinking the guy who has a windscreen on every car he ever owned is a bit odd. 

Read, play and listen. If this isn't already your hobby like it was for me when I got my first computer in 1990 and built my own upgrade to a 486 from parts off my own back and pocket money then you'll need to look into courses and qualifications. 

In that case a general overview, such as a degree (I'm in the UK so don't know what you'll call it whenever you are). In my degree in 2002 we looked at everything from programming (Object Oriented, Functional), hardware (we looked at the CPU architectures particularly), GPU pipelining and graphics rendering, compression (Huffman codes, Lempel Ziv, particularly it's implementation in GIF), web technology (CSS, JavaScript, Active Server Pages and CGI scripts), Human Computer Interaction ( its why I SCREAMED in fear in my head when I saw Windows 8, also Gnome 3 and Unity, can't stand them), and many more subjects. 

At the end of my degree we were looking at Haskell and functional programming (Haskell makes you feel like a GOD but you also have a melted brain) as well as touching on the design of compilers and genetic algorithms. 

That would give a start, you could go further into a masters degree but I was fine going forward into the industry with Software Testing. Which I found boring and did it for 6 years but your first foray in the real world shapes you. I hated software testing but I cut my teeth on it and so I still have all that instinctively up there in my head. It's always chattering away as I diagnose this or solve that, it brings forward everything from me playing with transistors as a kid to compiling a kernel and yes even C64 BASIC pops up 😂. It's even made me a cynical complaining 45 year old IT guy as I mumble and cry over the quite obvious lack of software testing of any sort that directly affects me and the users. 

Oh and read. Chip datasheets. Service manuals. RFCs that defined SMTP! Play with Gopher even. You don't need to memorise it all, just the facts and an understanding of how it plugs together.

If you simply listen to Steve Gibson's podcasts on how the internet works, and his older one on how hard drives work, you'll be light years ahead of most.

[u/JT99-FirstBallot → u/dlarge6510]

This was a great post. I thoroughly enjoyed reading it. I thought I knew a lot as a lifelong IT worker and hobbyist. (Although, I finally left IT for telecom. I enjoy it more. I feel more like an engineer than a gopher.) But your knowledge seems awesome and far surpasses mine. I could pick your brain apart for an entire evening and be interested. Hah. Thanks for the write up

[u/tthatfreak → u/dlarge6510]

My bread and butter was tape. Identifying the tape format was the easy part. Telling people that I didnt have an ancient copy of the Mac version of Retrospect sitting around was the hard part (because heaven forbid it be compatible with Windows version of Retrospect).

NTBACKUP is the other fun one. Kept around a windows XP setup and loud, old decks running for that specific purpose.

[u/dlarge6510 → u/tthatfreak]

Luckily NTBackup and BackupExec are exactly the same format. 

I installed XP SP3 on a Dell Xeon 4 core PCIe machine as a BackupExec tape was an old DAT version I hadn't managed to get working on the BackupExec server running Windows server 2022 as they dropped their own drivers in favour of the default windows one which turned out to have a bug where it doesn't support very old 4mm DDS drives for some reason. 

[u/tthatfreak → u/dlarge6510]

I got dropped into the business the same way. "You see these tapes? They're your problem now. Here are the wrong commands to try and if it doesn't work you reject the work."

I couldn't accept not giving people a reason why. I even used to blame solar flares on unreadable tapes because no one drops tapes....

[u/crumblenaut → u/dlarge6510]

I've been given a tote full of Apple II 5.25" floppies and early Mac 3.5" disks to recover for archival purposes. They belonged to one of the most important figures in biochemistry and may contain unique research and other information not contained elsewhere.

So far my plan is to Greaseweasel every disk for a full flux read and emulator-capable disk image and then use a working Apple IIc I happened to have on hand with a rigged up DIN to RS232 null modem cable to try to get at the actual filesystem data off of the disks.

If this lights up anything for you or you have any insight or interest in nerding out or being a reference check for me on this project, let me know! I'm doing the work on a volunteer basis because the particular individual in question and the nonprofit I'm working with are extremely worthy of the donation of time and expertise.

Happy to share more if you care to hear it!

[u/wonlove → u/dlarge6510]

are there still gopher servers?

[u/dlarge6510 → u/wonlove]

Floodgap is a typical jumping off point with access to the Veronica 2 search engine 

https://gopher.floodgap.com/gopher/

[u/One-Reflection-4826 → u/dlarge6510]

great read, thank you!

[u/za419 → u/dlarge6510]

It's amazing how quickly data degrades when it's not in use. At my current job there were a few occasions where I had to modify source code written in 1991, and while intimidating to think that I was changing stuff that had been the same since before I was born, it was no harder than any other file, because it was being continually built and so lived (and lives, as far as I know) on the same servers as everything else.

But if that wasn't the case, and that stuff had to get tracked down and fetched from original archives? Suddenly it's a herculean task.

[u/bananajr6000 → u/dlarge6510]

I understood some of those words!

The fact that I recognize most of that and your plight is pretty disturbing to me

[u/Merusk → u/dlarge6510]

Heck they make me laugh out loud once when they asked how long we need to archive data for, as they thought it was 7 years 😂 

This timeframe is accurate for most business records. Only place longer is Architectural records in the state of PA that must be maintained for 15 years.

Most businesses destroy after that timeframe, so I can see why they had that number in their head. They didn't realize you were talking about something completely different.

My wife worked in this sector and authored on some papers around it between 2010 and 2017 before leaving academia. I'm going to have to share this with her. :D

[u/MasterK999 → u/dlarge6510]

Good luck with the CD-R's. I had a bunch of them that were going back to the early days of that tech and none was readable. A few later archival quality gold disks had low enough error counts to get some data but anything on cheaper media were toast.

[u/dlarge6510 → u/MasterK999]

Oh they'll be fine. Most are in the air conditioned room with the tapes. I was given a stack of about 20 a department kept in a drawer in an unconditioned room built in the 1950's. Two of them had issues reading but all that disappeared when I bypassed windows and read them with error correction and the correct tools like ddrescue.

They are however Imation discs and are Phthalocyanine types so newer that what I assume you had which were cyanine?

I'm way more concerned about the usb HDDs I find here and there. 

[u/decavolt → u/dlarge6510]

Super interesting. I've worked in software for 20+ years, and have always been interested in archival work. Your gig sounds fascinating.

I know it's not 30-50 year stable, but how do you feel about a RAID5 setup for a home server? I have one I built and it's fine, but I still worry a bit that SSDs, even in a RAID array, aren't as... durable as some other option.