Release Candidate of LizardFS 3.13 with built in HA available!

Dear Users,

3.13.0-rc1 (release candidate) is officially out!

Featuring:

  • uRaft HA
  • fixes to EC handling
  • nfs-ganesha plugin changed to use only C code
  • reduced number of secondary group retrievals
  • new fuse3 client
  • many fixes

Detailed info:

  • uRaft HA *

uRaft is HA solution designed for use with LizardFS. It allows for seamless switching of  the master server in case of hardware failure. More information about uRaft is available in LizardfFS Handbook.

  • fixes to EC handling *

After extensive tests, we decided to improve the mechanism of calculating parities greater than 4 e.g EC( 6,5). After this upgrade, the system will show chunks of parities as endangered until the system automatically recalculates.

  • nfs-ganesha plugin changed to use only C code *

In preparation for moving LizardFS nfs-ganesha plugin to official nfs-ganesha repository, we had to remove all occurrences of C++ code and replace it with plain C.

  • reduced number of secondary group retrievals *

In LizardFS we introduced the handling of secondary groups. Unfortunately, the function to retrieve secondary groups in FUSE library used a lot of CPU resources. Thanks to removing unnecessary calls to this function, mount performance increased significantly.

  • added fuse3 client *

LizardFS now includes mount3 client which uses FUSE3 library. Thanks to new features in FUSE3, now the mount performs much better in many scenarios. Here are the most important changes visible to LizardFS users:

  • big_writes option is now enabled by default (not recognized as a parameter anymore).
  • added writeback_cache option which with kernel 3.14 and later improves performance significantly
  • increased read/write performance (especially for small operations)

Because most of the Linux distributions don’t include FUSE3 library, we have build FUSE3 packages and made them available on the LizardFS website.

 

Looking for Partners

Storage for OpenNebula Cloud

LizardFS TM drivers for OpenNebula

Based on the original OpenNebula code, changes by Carlo Daffara (NodeWeaver Srl) 2017-2018

For information and requests: info@nodeweaver.eu

To contribute bug patches or new features, you can use the GitHub Pull Request model.

Code and documentation are under the Apache License 2.0 like OpenNebula.

This is the same set of drivers used within our NodeWeaver platform, compatible with Lizardfs 3.x and OpenNebula 5.4.x

Prerequisites:

  • lizardfs command executable by the user that launch the OpenNebula probes (usually oneadmin) and in the default path
  • the lizardfs datastore must be mounted and reachable from all the nodes where the TM drivers are in use, and with the same path

The TM drivers are derived from the latest 5.4.3 “shared” TM drivers, with all the copies and other operations modified to use the live snapshot feature of LizardFS.

To install in OpenNebula

Copy the drivers in /var/lib/one/remotes/tm/lizardfs

# fix ownership

# is you have changed the default user and group of OpenNebula, substitute  oneadmin.onedadmin with <installationuser>.<installationgroup>

chown -R oneadmin.oneadmin /var/lib/one/remotes/tm/lizardfs

Visit GitHub

Hostersi implemented LizardFS as a storage to simplify administration and increase performance of Webankieta.pl platform

Webankieta is a system for creating questionnaires. It serves such prestigious clients as ING Bank Slaski, Deutsche Bank, BZWBK, Itaka, Medicover, PKP, PZU, Danone, Jysk, Polska Press Grupa.

Implementation

Hostersi first configured their new infrastructure and migrated the systems to a new Data Center in order to eliminate some of the bottlenecks and challenges it was experiencing.

The changes implemented by Hostersi resulted in full redundancy and High Availability. That means that even if a substantial part of the Customers infrastructure should fail, users would not even notice it.

In order to increase the performance of the applications, Hostersi implemented HTTP2 protocol. Another part of the project was to provide fast access to the platform from anywhere by implementing CDN (Content Delivery Network). The level of security was increased by building in DDoS Prevention mechanisms. Upgrading the overall infrastructure eliminating major security gaps related to meltdown.

Last (but not least) part of the project was the implementation of the log management tools set ELK Stack that consists of Elasticsearch (text search engine), Logstash (log aggregation) and Kibana (visualization).

The parallel Distributed Geo-Redundant File System LizardFS was used for storing and aggregating data from many applications before it gets to ELK.

Thanks to that configuration (Elasticsearch and LizardFS) Hostersi managed to build a storage with searching capabilities that are compliant with GDPR.

Data aggregation due to security precautions is done from flat files. These files are later processed by parsers in Logstash. The whole process is done this way so data can return to the canonical state of a log.

Thanks to LizardFS Hostersi can scale both up and down simply by adding a drive or a node – the system will automatically balance itself.

LizardFS being totally hardware agnosticism enables Hostersi to use their existing infrastructure and gives them the possibility to exit vendor lock-in if they are buying any new commodity components.

LizardFS and Elasticsearch are open source products, you can install them on existing infrastructure. They enable you to create a platform for content management and storage in a cost-effective way. High Availability and extreme scalability are just a few added value features provided by LizardFS.

 

LizardFS multilingual – manual in Russian

We are so grateful to the people that contribute to our project. Recently thanks to Andrew Mind there is a LizardFS manual available in Russian.
Next contributions are coming up shortly so stay tuned!

https://habr.com/post/343326/

Open Source changing media sector.

 

 

 

 

 

 

 

 

It is a general trend which is getting stronger and stronger. Moving away from closed-source, proprietary solutions to COTS, open infrastructure, solutions that enable people to use commodity hardware.

You could see that in a couple of talks at #NABshow. People are not only working on them, some of them are already using such solutions.

Embrace the shift from #fiberchannel to #ethernet Find out how you can benefit from #opensource hardware agnostic #scaleout high-performance #distributed #filesystem Visit #lizardfs booth on #NAB2018 N2936sul-b (North Hall).

Hadoop plugin for LizardFS is here!

As everything that we do Hadoop Plugin for LizardFS is as simple as we could make it.

This is a java based solution allowing Hadoop to use LizardFS storage, implementing an HDFS interface to LizardFS. It functions as kind of a File System Abstraction Layer. It enables you to use Hadoop jobs to directly access the data on a LizardFS cluster. The plugin translates LizardFS protocol and makes the metadata readable for Yarn and Map Reduce. For performance, Hadoop nodes should run on the same machines as LizardFS chunk servers.

LizardFS mount gives direct access to stored files, from the OS level. This allows you to use it as a shared storage in your company and a computation storage for HADOOP at the same time. It is not required to use HADOOP tools to put/get files from your storage in comparison to HDFS. We can also take advantage of Erasure Coding and save a lot of disk space (HDFS recommends to store 3 copies).

The function:

public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len)

Returns information where data blocks are held in your LizardFS installation. If Hadoop is run on the same machines, it can take advantage of data locality.

To install Hadoop with LizardFS:

1) Install and setup LizardFS cluster

2) Install HADOOP – but don’t start

3) Install LizardFS-HADOOP plugin on all HADOOP nodes

4) Configure LizardFS-Plugin in HADOOP (alongside HDFS or replace it)

5) Start HADOOP

Let us know what you think of it.

Enjoy!

LizardFS @ NABSHOW 2018

 

Come visit our stand at NABSHOW in Las Vegas.

North Hall Central Lobby in the Startup Loft, Booth number: N2936SUL-B

LizardFS@NABSHOW (how to find us)