LizardFS and DIAWAY Announce Partnership

LizardFS and DIAWAY Announce Partnership To Market a Joint Storage Solution that dramatically lowers total cost of ownership

Warsaw, Poland– April 16, 2020 – DIAWAY, the big-data storage, and networking integrator, and LizardFS, the Software-Defined Storage provider, today announced a new strategic partnership to integrate essential DIAWAY hardware and equipment solutions for endpoint and container workloads within LizardFS. Under this new partnership, DIAWAY tightly integrates its hardware and networking solutions for LizardFS software-defined storage.

Today many customers face unprecedented exponential data growth. Ensuring cost-efficiency and uptime for the growing data is critical.

With this new joint solution, customers can deploy DIAWAY’s complete offering for various use cases with LizardFS software-defined storage. DIAWAY’s Techonomics™ approach uses advanced hardware and financial analytics to help optimize, maintain, and scale workloads deployed on LizardFS.

LizardFS makes it possible to build storage solutions on multiple servers, scaling vertically and horizontally by a single drive or a single node. It allows customers to have up to exabytes of data. This type of solution ensures readiness for an exponential increase in storage while using the DIAWAY storage solution. Organizations can also leverage DIAWAY-powered LizardFS File Storage Solution for Elasticsearch Storage, Genome Research, Geological and Geophysical data, and General-Purpose File Storage. Using LizardFS for Docker volumes is one of the leading container usage cases. In environments with many docker instances, the DevOps teams are struggling with efficient and consistent backups. Usually, such consistent backups require a stop (put down) docker image for all the time needed by backup software to synchronize data between current status and backup. With LizardFS, no stop operation is needed, providing a non-interruptible workflow.

DIAWAY helps LizardFS customers optimize their infrastructure CapEx and OpEx by advising them on selecting the best hardware for the task and supplying it as a complete solution. Thus, significantly decreasing overall data storage costs and removing the complexity of designing, building, supplying, and supporting the storage infrastructure.

“Demand for different storage solutions, optimal for specific tasks, is on the rise and strong,” said Aleksandr Ragel, managing partner, and co-founder at DIAWAY. “A justifiable portion of this demand may leverage LizardFS capabilities, and we are proud to integrate our signature storage hardware and design in LizardFS software-defined storage infrastructure. This partnership ensures that LizardFS customers can benefit from proven cost-efficiency, reliability, and performance.”

“We’re excited to partner with DIAWAY to bring their proven, trusted multi-petabyte capabilities in building storage infrastructures to our customers,” said Piotr Modrzyk, Product Development Lead at LizardFS. “Integrating DIAWAY’s solutions with LizardFS means customers now have even more tools to ensure the highest levels of performance and resilience as they evolve with existing installations or start anew.”

Additional Resources

  1. LizardFS File Storage Solution powered by DIAWAY
  2. LizardFS Infrastructure online TCO tool by DIAWAY

About LizardFS
LizardFS Software Defined Storage is a free open-source distributed, parallel, scalable, fault-tolerant, Geo-Redundant, and highly available file system. It allows users to combine disk space located on several servers into a single name space, which is visible on Unix-like and Windows systems in the same way as other file systems. LizardFS makes files secure by keeping all the data in multiple replicas with erasure coding, spread over the available servers. It can also be used to build space-efficient storage because it is designed to run on commodity hardware. For more information, please visit: https://lizardfs.com

About DIAWAY
DIAWAY is the leading big-data storage and networking integrator for data-intensive global organizations. The company designs, deploys and optimizes systems, software, and storage solutions that enable enterprises, cloud providers, and software companies to generate more value from their data. Headquartered in Tallinn, Europe’s tech capital, DIAWAY develops its partnerships and solutions across the globe with infrastructure footprints exceeding hundreds of petabytes per project. DIAWAY is an official partner to Western Digital® and AMD EPYC™ Elite Partner. For more information, please visit: https://diaway.eu

 

LizardFS

 

Media contact:

contact@lizardfs.com

Community Article #8

 

Hi everyone, my name is Wessam Aly. I’m from Egypt. I speak English, French, and Arabic. I’m also pretty fluent in C, and I do some coding in Python, Perl and Go as well.

I have personally implemented and tested quite a few distributed file systems already, namely Ceph, OpenIO SDS, Minio, GlusterFS and obviously LizardFS.

Professionally, I have used, implemented and tested Cohesity, Rubrik, Hedvig and Veritas CFS.

 

I also have experience with other storage systems:

  • IBM (Flash system, Elastic Storage System, Elastic Cloud, SAN Volume Controller, etc.),
  • EMC (Isilon, DataDomain, Unity, Clariion, DMX, etc.),
  • HPE (3PAR),
  • Hitachi (XP)

So, the most important part is, why did I choose LizardFS?

I have heard good things about its speed and reliability at scale (all out of the box, with no special tuning).

I only did a bit of testing, but so far I have discovered that LizardFS is one of the easiest and most direct implementations of an SDS solution among all products I have experience with.

I really like a lot of things about LizardFS, low system requirements and straightforward architecture are a huge plus, so are the clients available for Windows. Geo-replication is really useful too.

 

There are some bad things about the project too, unfortunately. I dislike master servers and metadata servers. The release cycle could be shorter too.

I would prefer for the metadata to be distributed among chunk servers the same way data is. If a clear changelog could be added together with a near-fixed release cycle that would mean a world to me.

 

Community Article #7

Hi there, I’m just back from the Tet holiday, the Lunar New year in Asia and it’s nice to write something about LizardFS, which in my opinion was the best distributed software storage when I was looking for a solution to our storage issues.

My name is Nhan Pham T. I live in Ho Chi Minh City in Vietnam, which is UTC + 07:00. I’m a dad, a blogger and a dreamer 🙂

I speak Vietnamese and English. For coding I use Python, Bash, and some PHP for daily tasks or small pet projects.

 

I started my professional career as a Python developer and have spent more than 6 years using Linux and open source software. I don’t actually have a personal laptop or macbook just my work computer (strange I know). I started doing research on distributed file systems when I tried to learn Docker in early 2013. I built a small lab with Ceph and used it for containers. A few years later, I started using LizardFS for our projects in the biggest Vietnamese technology company, founded in 2004, specializing in digital content and online entertainment, social networking, and e-commerce.

 

LizardFS was chosen as our solution because it’s flexible, efficient and easy to set up. We have been using it since April 2019 as a distributed file system for our product. Before that, we used a physical disk for all storage with RAID 5, 6 for files and objects. We tried GlusterFS, Ceph and some solutions from HP and Dell in our lab, but lizardfs has URAFT for high metadata availability, something like MooseFS Pro, but with more features and functionalities available on open source.

 

Lizardfs is easy to learn but its documentation is not up-to-date, and a lot of issues on github that are open and seem not to be closed or fixed.

What I dislike about distributed file systems is that they are generally a nightmare for system engineers like me. Lots of issues and some incidents, not so much with LizardFS though.

 

Thinking ahead I would like to see multithreading implemented in the next release. (https://github.com/lizardfs/lizardfs/issues/476) and a webmin to administration (https://github.com/lizardfs/lizardfs/issues/546)

Also reducing memory usage of the master server would be nice. In my case, I have 256GB RAM for the cluster but I don’t think it is enough to lead many chunkservers to exabyte levels.

 

In the end, I would like to thank Mark Mulrainey for starting this exciting movement of sharing the community experiences with LizardFS.

Community Article #6

Lizard server

My name is Carlo Daffara, from Udine, Italy. A very small town 100KM north-east of Venice.

I speak Italian and English; as for coding- I started many, many years ago with 6502 assembly and Pascal. Now, mostly Go, Bash and some python.

I have worked in the field of distributed systems for a long time; mostly in parallel file systems for HPC.

LizardFS was chosen as our solution as it’s flexible, efficient and allows for lots of customization. We have been using it since 2014 as the distributed file system within our product (NodeWeaver).

What I like most about LizardFS is it’s flexible and lightweight.

Due to some recent instabilities we had to create our own tree… we hope that with the management change and new development team this can be resolved.

I would love to see more efficient handling of replication; multiple labels per chunkserver in upcoming versions of LizardFS.

Community Article #5

Logo LizardFS

Hi my name is Adri Istwan Mansor, I am from Kuala Lumpur in Malaysia (GMT+8). 

Actually it’s south east Asia, has great cheap food and very friendly mixed Malay-Chinese-Indian-Aborigines people.

We went from an insanely slow ADSL to fiber throughout the whole nation of 20 million people in a few years. So after fiberizing the whole nation, we use the interweb mostly for Facebook (nation with the longest amount of time on FB lol), trolling others and doxxing “bad”people. Nobody seems to care though.

We tend to try and relieve daily issues in our lives by creating and using tech startups. Bad taxi reputation – use Grab, moving house – Use thelorry, slow bank – use e-wallets like Grabpay, unavailable products – create cheap local e-commerce sites, slow/poor (modern) education syllabus – use AI. Too difficult/cheapskate to create a site (takes a week for the government to validate a .my domain application) – use Facebook & Instagram. The government encourages the startup industry, slow moving rural areas – the government keeps throwing free grant money to create startups, fiber is too expensive – the government cuts fees in half and doubles the speed.

I predominantly speak English and Malay although due to our mixed culture we tend to mix this with some Cantonese and Tamil, it’s similar to Indonesian. As for Computer languages, a bit of python, javascript and ye olde php4. I’m currently brushing up on my javascript as I tend to do lots of tensorflow.js.

Before discovering LizardFS I used BTRFS, NTFS and ZFS.

The reasons I decided to use LizardFS are that ZFS is not flexible, NTFS is/was good, but not many modern/current features, BTRFS is fast and simple (ish). SnapRaid is… dangerous. My current storage is mostly BTRFS, NTFS and some XFS, Synology RAID is limited to 26 drives, GlusterFS and Ceph are not flexible enough, MooseFS (free) does not have EC and redundancy AFAIK and any others are not that well known enough for me.

LizardFS has EC, redundancy and best of all, flexibility.

Currently I have been using it for around 6 months, still really in the testing phase, been trying out various distributed file systems for a year now. But after trying out the others, I believe LizardFS has a huge potential for a small-medium enterprises (SMEs), at least from what I see in my country

I intend to use LizardFS in my two tv production companies.

Previously, all raw media would be stored in external NTFS drives. I stopped counting after 30 drives, so lets say lots. We need to access footage every now and then, so cataloging and referring footage in excel sheets are a burden. Standard practice in most if not all of tv production houses in the country. That prompted me to find a way to put them all inside a rack mount server or two. Problem #1 is, there are literally zero market for 4U racks over here. Even 2U servers are unheard of in the secondhand market. Problem #2, the crazy price of hard drives compared to the ones in the States. Even when they’re (used to be) made (Western Digital) just 20 minutes from KL or 5 hours from KL (Seagate). I’m shocked. 

Then I saw these two images on reddit and freaked out like a kid getting a PlayStation for Christmas

That was a Eureka moment for me. So I researched more on Distributed File Systems, and after a while of research and experimenting I settled with LizardFS to start testing. Barring a few setbacks (master server seems to go down every now and then), seems like LizardFS is the way to go. I’m still in the testing phase though.

Another use I have for LizardFS is Malaysian Coal & Minerals Corporation – my mining & trading company it will come in useful there too.

The things I like most about LizardFS are the flexibility with mixed-capacity drives, and the underlying filesystem – I don’t have to reformat the external drives!

EC & Redundancy

(Now) docker-based installation OH MY GOD THIS HELPS SO MUCH

Per-folder goals OMG

Long-winded documentation with explanations for a boss/sysadmin like me

On the other side things that I dislike about it are the long-winded documentation (only). I need a (very) quick guide on setting up a quick LFS setup, both from official and not-so-official sources. Or better yet, have a scenario-based guide. I’m still wrapping my head around understanding and implementing goals. In my case, I need LFS to consolidate all NTFS drives’ folders into just a few main folders: 

  • raw footage 
  • stock media
  • artist info
  • admin/management files
  • casting
  • projects
  • syndicated projects
  • location info

So since each folder has different importance and file types (large single files, many small files, lots of sub-folders, etc) I’d like it if there’s an example of how the goals should be set. I’m still unsure if I can just pop the drives online, and turn existing data and drives into chunkservers. I’ve yet to find out how.

– No webui-based right click folder>Set goals. This is something HUGE for someone like me.

– Synology/Odroid-HC2/RaspberryPi/armhf support and speed could be improved (based on what I’ve read on github).

What I would love to see in the future, a proper webui for all of the steps in the official guide. At least to me, this greatly speeds up testing and I believe, adoption. But biggest wish list would be a focus on SBCs like odroid HC2. I really believe the future in LizardFS lies in rock solid support on SBC hardware as well as cheap, low-powered x64 PCs.